Messages before 1PM (5AM UTC) china time never appeared in graylog. New messages start poping at 1:00PM.
All messages before 1:00 never arrived.
I have light setup, filebeat, sidecar, graylog, with just 2 log files parsed and pushed to the graylog.
Log files created at the morning ~8AM China and data starts (explicitly flushed) into it.
Filebeat start pushing data into graylog, but graylog does not display anything until 1PM.
After 1PM everything works as expected.
Both filebeat and graylog run on machine with same China timezone.
Pattern is consistent across the dates and all machines in here.
Any suggestion where to look?
Configuration is:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
Boba, Moving your message to Daily Challenges where it may get more community members’ eyes on it.
While we await responses, can you please include additional information, as seen in this similar message below?
Filebeat by default has an at least once delivery. So if there are some errors when sent, FB will retry. Do you see any errors elsewhere?
Another thought: If you don’t ingest anything else, you can use the http random generator input to generate messages in discrete intervals. That might give you some more datapoints where and when behaviour changes, things stop working or a dropped.
I have tried suggestion to generate logs to get datapoints. FYI: log appears from 1PM till 1AM).
I figured out the cause of the problem:
I have created extractor with “store as a field” name “timestamp” that parse timestamp from the log file.
Message has own “Timestamp”. 2 fields conflict with each other.