Unable to decode raw message RawMessage cause 100% ProcessBuffer [Solved Partial]

Graylog Central (peer support)
1

Hi,

Sporadically, the Graylog server receives some message and can not decode, the following message appears in the log: “Unable to decode raw message RawMessage”. However, it stops processing messages and the Processing Buffer is 100%, queuing in the Journal, in some cases it takes about 10 minutes and it resumes processing the messages, however most of the time I have to delete the Journal and restart the service Graylog.

The full log.

Unable to decode raw message RawMessage{id=0ee0fc70-2f59-11e7-b7ab-0050568009f6, journalOffset=9589000, codec=syslog, payloadSize=786, timestamp=2017-05-02T17:02:00.631Z, remoteAddress=/192.168.0.200:44149} on input <58e28e061f6e962c95d7ecf1>.
2017-05-02T14:02:00.634-03:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=0ee0fc70-2f59-11e7-b7ab-0050568009f6, journalOffset=9589000, codec=syslog, payloadSize=786, timestamp=2017-05-02T17:02:00.631Z, remoteAddress=/192.168.0.200:44149}
java.time.format.DateTimeParseException: Text '161.98&iv-event=2&event=iv&v=WON12GaKpS4&a-id=66d9155f-7cb9-4146...' could not be parsed at index 2
        at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) ~[?:1.8.0_131]
        at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) ~[?:1.8.0_131]
        at java.time.LocalTime.parse(LocalTime.java:441) ~[?:1.8.0_131]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parseDate(FortiGateSyslogEvent.java:90) ~[graylog.jar:?]

I believe that in the message (which is from a fortigate) the sent URL causes the problem:

java.time.format.DateTimeParseException: **Text '161.98&iv-event=2&event=iv&v=WON12GaKpS4&a-id=66d9155f-7cb9-4146...'** could not be parsed at index 2
2

Try using a Raw/Plaintext input for processing Fortigate syslog messages and extract the required information with Extractors or the Processing Pipeline:

1 Like
3

3 hours and no more errors. After changing to RAW I think have corrected it. Thanks!

4

@jochen.

The raw messages solve the error in decode. But the process continues stop:

5

have you checked that you have enough memory in the Graylog nodes? Make sure you don’t allocate too much memory for Graylog JVM, as other stuff needs also memory. For my 24G memory Graylog nodes I found that allocating 12G for Graylog JVM caused out of memory errors, while with 8G memory for Graylog JVM does not.

Does the mongod run properly in the node that stops?