How-to troubleshoot with loadbalancer

Graylog Central (peer support)
1

We have a NetScaler loadbalancer in front of 3 graylog servers (they are all-in-one) because we didnt have the resources at the time to setup the cluster with seperate elasticsearch nodes.

We have Fortigate firewalls syslogging to the IP adress of the loadbalancer on UDP 30000. Works pretty fine. Every now and then we get this error message in the logs:

2020-08-26T16:54:26.257+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=092d3410-e7ac-11ea-9380-00155d85450a, journalOffset=729253992, c
odec=syslog, payloadSize=652, timestamp=2020-08-26T14:54:26.257Z, remoteAddress=/10.221.5.170:47533}
java.time.format.DateTimeParseException: Text '1598453672&type=accesspoint' could not be parsed at index 2
        at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) ~[?:1.8.0_265]
        at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) ~[?:1.8.0_265]
        at java.time.LocalTime.parse(LocalTime.java:441) ~[?:1.8.0_265]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parseDate(FortiGateSyslogEvent.java:90) ~[graylog.jar:?]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parse(FortiGateSyslogEvent.java:58) ~[graylog.jar:?]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.<init>(FortiGateSyslogEvent.java:44) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]

But i don’t know how to troubleshoot this. Can i find the message somewhere in an index? So i can see where it comes from? Because the source always seems to be the netscaler.

-edit-

I also see one now that says:

2020-08-27T10:31:33.173+02:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=b6908650-e83f-11ea-8468-00155d85450b, journalOffset=907025781, codec=syslog, payloadSize=708, timestamp=2020-08-27T08:31:33.173Z, remoteAddress=/10.221.5.170:22891} on input <5f43cf861035f340f3c687d9>.
2020-08-27T10:31:33.174+02:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=b6908650-e83f-11ea-8468-00155d85450b, journalOffset=907025781, codec=syslog, payloadSize=708, timestamp=2020-08-27T08:31:33.173Z, remoteAddress=/10.221.5.170:22891}
java.time.format.DateTimeParseException: Text '1598517093' could not be parsed at index 2
        at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949) ~[?:1.8.0_265]
        at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1851) ~[?:1.8.0_265]
        at java.time.LocalTime.parse(LocalTime.java:441) ~[?:1.8.0_265]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parseDate(FortiGateSyslogEvent.java:90) ~[graylog.jar:?]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.parse(FortiGateSyslogEvent.java:58) ~[graylog.jar:?]
        at org.graylog2.syslog4j.server.impl.event.FortiGateSyslogEvent.<init>(FortiGateSyslogEvent.java:44) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.parse(SyslogCodec.java:130) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.SyslogCodec.decode(SyslogCodec.java:96) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]

So this has something to do with the numbers?

If so, then it’s probably because of my Pipeline? Making an timestamp?

rule "Epoch Convert"
when
has_field("devname") && has_field("date") && has_field("time") && has_field("eventtime")
then
let epoch = parse_date("1970-01-01 00:00:00.000Z", "yyyy-MM-dd' 'HH:mm:ss.SSSZ");
let ts_seconds = seconds(to_long($message.eventtime));
set_field("epoch_timestamp", epoch + ts_seconds);
set_field("timestamp", epoch + ts_seconds);
End
2

No one? :slight_smile:

3

Probably your eventtime field is parsed incorrectly, so it contain non numeric value (for epoch). Check how do you parse field eventtime (extractor/pipeline rule/key value extractor/grok?).

Your pipeline rule is little weird, I would suggest to use rather function parse_unix_milliseconds:
https://docs.graylog.org/en/3.3/pages/pipelines/functions.html#parse-unix-milliseconds

let epoch = parse_unix_milliseconds(to_long($message.eventtime) * 1000);
set_field("timestamp", epoch);
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.