Messages Arrive 4h in the Future

I’ve been beating my head up against a wall with this one for the past four days. I’ve searched and tried a few items but I’m at whits end. Hoping that someone can offer some insight, or maybe a suggestion.
I have a pair of Graylog servers behind a load balancer. Messages are streaming in from a Syslog source on UDP/514 which I’m translating to UDP/1515 with the load balancer. Graylog is configured to take all of the messages on the UDP/1515 input and send them to a stream. I haven’t done any pipeline rules against the messages at this time.
The issue is that messages from this one source arrive four hours in the future. This leads the Graylog server to not process any alerts on the messages for four hours.
The sender, who is controlling the source device, states that their device is running UTC. My Graylog servers, and load balancer are configured for UTC.
When I use my laptop, configured as UTC, to generate random Syslog messages and point it at the UDP/1515 receiver the messages arrive at the correct time and I get alerts immediately. When I reconfigure and use the loadbalanced UDP/514 receiver the messages also arrive at the correct time and Graylog immediately alerts.
At this point I’ve confirmed UTC on all systems in the Graylog stack. I have confirmed UTC on the load balancers. I have configured the Syslog UDP input to both override the date and leave it as-is. Also of note, there are no other messages which are arriving in the future like this. I’ve redirected other syslog streams to the UDP/1515 input as a test from multiple other devices and they also arrive at the current time.
Again I know this forum is a best-effort so if anyone has any where I can look I’d be grateful.

do you have one example message?

Maybe no timestamp is provided in the message what can make crazy shit …

Good call. Interesting that you pointed out the timestamp, I just noticed that in the messages I see the following.

full_message: <24>Jun 17 12:08:12 pfsp: Host Detection alert ########, start 2019-06-17 12:08:11 UTC, duration 1, direction incoming, host #.#.#.#, signatures (TCP SYN), impact 9.68 Mbps/25.83 Kpps, importance 2, managed_objects ("##########################"), (parent managed object “nil”)
timestamp: 2019-06-17T16:08:12.000Z

full_message: <24>Jun 17 12:10:27 pfsp: Host Detection alert ########, start 2019-06-17 12:08:11 UTC, duration 135, stop 2019-06-17 12:10:26 UTC, importance 2, managed_objects ("#######################"), is now done, (parent managed object “nil”)
timestamp: 2019-06-17T16:10:27.000Z

So the timestamp on the full message is 4h before the timestamp that has been assigned to it by Graylog. Should I look into trying to overwrite the timestamp field with a pipeline process?

yes I would use the processing pipelines to correct that timestamp.

we have some examples here in the community for that.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.