Hello everyone, I’m writing this post due to an issue in log processing on Graylog.
1. Describe your incident:
The logs are not showing in real-time over Graylog and because of that we are not receiving Slack notifications. This occurs only (for now) with the AWS Cloudtrail logs, we have 3 accounts and 1 SQS for each account and the inputs in Graylog were created successfully (no problems in configuration reviewing that). We notice in SQS the messages accumulated are rounding 8000 and the “Message in flight” are a small amount when we refreshing many times.
In Graylog, we saw the messages incoming against an event in AWS are having a delay between 10 and 15 minutes (Example, I make a “DescribeSecurityGroup” and that message arrives on Graylog then 10 minutes, not in Real-time).
2. Describe your environment:
OS Information: (EC2 c5) Ubuntu 22.04 LTS, 4 cores, 8 GB RAM, 1 disk (20 GB) and 1 partition (49 GB).
Package Version: Graylog 4.3.3 Noir, Elasticsearch 7.10.2, MongoDB 6.0.0 (NOTE: the 3 are installed in the same server).
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
We were trying changing any configuration of Graylog and/or Elasticsearch about JVM (from 1 to 2 or 3 GB) but that’s not working. Then, we tried modifications on Índices options but I’m not sure what is the more optimized configuration for an installation in we use case.
NOTE: I’m not understanding clearly the documentation about Índices and Pipelines on Graylog (I think maybe the problem can be fixed implementing something about these), if anybody can give a little additional explanation about that I appreciate so much.
Only couple ideas for delay messages come to mind. Ensure Date/time is correct on both the server and Client. Check logs on the server (i.e., Graylog, Elasticsearch, etc…) maybe something in there might help. Check resources (CPU, RAM) sometime running out of resources can cause a glitch. Do you have Extractors or pipelines Configured?
I’m assuming this maybe a Virtual machine ?
Since this is only 10-15 delay I don’t think it’s a Time Zone issue, There should be two timestamps, 1 for Elasticsearch when it indexed those messages and 1 for the original message from that device, you can see the date/time with the message
Below is an example of a message sent at 17:25:31 ( 5:25 PM) and did not get indexed till 17:45:40 (5:45 PM)
This issue was a result of resources, elasticsearch could not keep up during busy hours. I would need to either separate ES from Graylog/MongoDb or add more resource to this single node.
Your timestamp could be the timestamp from the message coming in to graylog.
At one input we convert messages with a pipeline rule and put te timestamp of the incoming message into the timestamp field, Graylog can handle this in a very smart way and even handles timezones pretty well.
rule "Parse - messages"
when
// Use routing message to select messages
to_string($message.message_route) == "cbs_logging_route"
then
let pf = grok(
pattern: "%{TIMESTAMP_ISO8601:timestamp}
understand that this is a part of our entire message.
