Error when using syslog-ng graylog2() with Graylog Central

Graylog Central (peer support)
1

I’m using syslog-ng graylog2() with Graylog GELF TCP input.
The conf of syslog-ng is as below.

destination redbull_graylog_test {
graylog2( host("xxx.xxx.xxx.xxx") port(9002) persist-name("redbull_graylog_test"));
};

But I got the error log on graylog server as below.

2022-10-21T09:27:22.477+08:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=8309cca0-50df-11ed-8dff-0242c2d85535, messageQueueId=2068049, codec=gelf, payloadSize=171, timestamp=2022-10-21T01:27:22.474Z, remoteAddress=/119.45.27.124:41519} on input <6351ef6dfb510b3ff8b947ad>.
2022-10-21T09:27:22.477+08:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=8309cca0-50df-11ed-8dff-0242c2d85535, messageQueueId=2068049, codec=gelf, payloadSize=171, timestamp=2022-10-21T01:27:22.474Z, remoteAddress=/119.45.27.124:41519}
com.fasterxml.jackson.core.JsonParseException: Unexpected character (',' (code 44)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (String)"{"version":"1.1","timestamp":1666315642,"short_message":"java.lang.NullPointerException","level":5,"host":"localhost","_program":"","_pid":,"_facility":"user","_class":""}"; line: 1, column: 141]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:591) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1902) ~[graylog.jar:?]
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:968) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:247) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4057) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2552) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

It seems the “” empty string is missing after “_pid”:.

Could anyone tell me what is wrong about my issue? Thank you very much.

2

Hello @daemondong

From what I see in the logs

Unable to decode raw message RawMessage

And this

 codec=gelf, payloadSize=171

So when you get this…

[Source: (String)"{"version":"1.1","timestamp":1666315642,"short_message":"java.lang.NullPointerException"
&&
org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:130)

The NullPointerException occurs due to a situation in application code where an uninitialized object is attempted to be accessed or modified. Essentially, this means the object reference does not point anywhere and has a null value.

Try using a different input.

EDIT: I looked into this a little more. Unfortunately, I don’t use syslog-ng but during my Googling for this issue

com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693)

After looking at this…

destination redbull_graylog_test {graylog2( host(“xxx.xxx.xxx.xxx”) port(9002) persist-name(“redbull_graylog_test”));};

I came across a statement about double quotes might be the issue, if so, then preprocess the string with one of the following before passing it to Jackson:

Remove the double quoting: newValue = initialValue.replaceAll("\"\"", "\"");

1 Like
3

I think the problem is when the Graylog analysis the string "

{"version":"1.1","timestamp":1666315642,"short_message":"java.lang.NullPointerException","level":5,"host":"localhost","_program":"","_pid":,"_facility":"user","_class":""}" it found there is NO "" after "_pid":, so it showed "Unexpected character (',' (code 44))". You can find there is "" after "_program":

I couldn’t find the way to add the _pid parameter is the source conf as below.

source redbull_log {
       file("/var/jxzc/enjoyment/logs/starter.log" follow_freq(1) multi-line-mode(regexp) multi-line-prefix("(ERROR|java)") flags(no-parse));
};

Maybe if the Graylog think the [:,] is the same as [:“”,] the problem will be fixed.
Thanks for your response.

4

I think you are correct, most, if not all the time "," need to escape those.

5

Hi gsmith

What should I do about this issue for now? Will the Graylog fix this issue on next version ?

6

Not sure, if you think its a bug you can post your issue here

As stated before, I have not used Syslog-ng, but after googling that error from above either the syslog-ng needs to different configuration or perhaps try a different input on graylog. Either way Graylog does not like it. Maybe someone else here uses Syslog-ng,

I was looking at this.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.