Filebeat messages dropping before 1 PM in China Zone

Messages before 1PM (5AM UTC) china time never appeared in graylog. New messages start poping at 1:00PM.
All messages before 1:00 never arrived.
I have light setup, filebeat, sidecar, graylog, with just 2 log files parsed and pushed to the graylog.
Log files created at the morning ~8AM China and data starts (explicitly flushed) into it.
Filebeat start pushing data into graylog, but graylog does not display anything until 1PM.
After 1PM everything works as expected.
Both filebeat and graylog run on machine with same China timezone.
Pattern is consistent across the dates and all machines in here.
Any suggestion where to look?

Configuration is:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
paths:
- /home/user1/log_*.log
type: log
output.logstash:
hosts: ["loghost.com:5044"]
path:
data: /var/lib/graylog-sidecar/collectors/filebeat/data
logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Beats configuration:
bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 4
override_source:
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file:
tls_client_auth: disabled
tls_client_auth_cert_file:
tls_enable: false
tls_key_file:
tls_key_password: ********

Boba, Moving your message to Daily Challenges where it may get more community members’ eyes on it.
While we await responses, can you please include additional information, as seen in this similar message below?

Filebeat by default has an at least once delivery. So if there are some errors when sent, FB will retry. Do you see any errors elsewhere?

1 Like

Thank you David, hope following helps:

  • There are no errors in filebeat log. It looks like filebeat pushes messages to graylog from system prospective.

  • There is no errors anywhere else

  • 1 PM china time is our Chicago office midnight, but I could find any consequences of it. Graylog server shows it runs under local time.

  • Log line converted to message looks like (timezone local) inside log :

2021-06-02 08:05:01.694364 [Ocean.cc:53,INFO] Ocean initiated.

  • time extractor successfully extract timestamp from messages we receive.

  • versions:
    Graylog 4.0.5
    filebeat 7.12.1

  • Operation system is Linux.

  • Log files generated by actual user

  • filebeat runs under root.

2 Likes

Awesome, Boba. Let’s get the community’s eyes on this!

When does the logging stop again? I have had situations with elasticsearch hovering around its disk watermark:

  1. New index created due to rotation, below watermark
  2. Data is ingested over time, until the disk usage rises above the watermark
  3. Elasticsearch marks the Index read only, data is not written anymore
  4. After some time Index rotation kicks in, cleans up older indices, disk usage drops below watermark
  5. Go back to 1

Besides the filebeats logs, have you also checked /var/log/graylog/server.log and /var/log/elasticsearch/ for any clues?

2 Likes

Another thought: If you don’t ingest anything else, you can use the http random generator input to generate messages in discrete intervals. That might give you some more datapoints where and when behaviour changes, things stop working or a dropped.

2 Likes

I have tried suggestion to generate logs to get datapoints. FYI: log appears from 1PM till 1AM).
I figured out the cause of the problem:
I have created extractor with “store as a field” name “timestamp” that parse timestamp from the log file.
Message has own “Timestamp”. 2 fields conflict with each other.

Timestamp 2021-06-08 13:00:04.238
timestamp 2021-06-08 17:00:04.238 +00:00

I rename timestamp to log_timestamp and it works now.

Ty @jrunu and @dscryber.

2 Likes

Happiness was preliminary. After I rename my timestamp extractor and restarted sidecar at 1AM, Messages keep showing but stopped at 1PM.

I have deleted extractor and it works. Issue resolved.
The cause was our timestamp with microseconds, while graylog understand only millis.

1 Like