Filebeat stops working after OS reboot

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

After running the client that Graylog collects logs from for many months, I had to perform a reboot. After the reboot, I noticed that filebeat is no longer working under graylog-sidecar.

2. Describe your environment:

  • OS Information:
    Rocky Linux 8.8
  • Package Version:
  • Service logs, configurations, and environment variables:

Default configs as per Graylog 4.x instruction are used for the sidecar.

Upon reboot, graylog-sidecar stopped filebeat. Then when the OS came up again, filebeat stopped working and no longer ships logs to Graylog.

2023-08-24T22:15:38.254+0200    INFO    beater/filebeat.go:515  Stopping filebeat
2023-08-24T22:15:38.254+0200    INFO    beater/crawler.go:148   Stopping Crawler
2023-08-24T22:15:38.255+0200    INFO    beater/crawler.go:158   Stopping 1 inputs
2023-08-24T22:15:38.255+0200    INFO    [crawler]       beater/crawler.go:163   Stopping input: 3121193877724421705
2023-08-24T22:15:38.255+0200    INFO    input/input.go:136      input ticker stopped
2023-08-24T22:15:38.255+0200    INFO    beater/crawler.go:178   Crawler stopped
2023-08-24T22:15:38.255+0200    INFO    [registrar]     registrar/registrar.go:132      Stopping Registrar
2023-08-24T22:15:38.255+0200    INFO    [registrar]     registrar/registrar.go:166      Ending Registrar
2023-08-24T22:15:38.255+0200    INFO    [registrar]     registrar/registrar.go:137      Registrar stopped
2023-08-24T22:15:38.277+0200    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"graylog-sidecar.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":362156032}}}},"cpu":{"system":{"ticks":1403790,"time":{"ms":1403790}},"total":{"ticks":2933610,"time":{"ms":2933618},"value":2933610},"user":{"ticks":1529820,"time":{"ms":1529828}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":12},"info":{"ephemeral_id":"dae0c2be-8a7e-4e79-8fa2-d0cf81954ce5","uptime":{"ms":14074924324}},"memstats":{"gc_next":17803648,"memory_alloc":10381496,"memory_total":185118979296,"rss":87212032},"runtime":{"goroutines":11}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":2},"load":{"1":0.45,"15":0.4,"5":0.77,"norm":{"1":0.225,"15":0.2,"5":0.385}}}}}}
2023-08-24T22:15:38.278+0200    INFO    [monitoring]    log/log.go:154  Uptime: 3909h42m4.325894382s
2023-08-24T22:15:38.278+0200    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2023-08-24T22:15:38.278+0200    INFO    instance/beat.go:461    filebeat stopped.

3. What steps have you already taken to try and solve the problem?
So far I have reinstalled filebeat and graylog-sidecar but I am still stuck.

4. How can the community help?
Has anyone experienced this mystery?
What is the best way to debug this filebeat issue on the client? Is there a way to activate debug mode for graylog-sidecar?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Further to this, I see graylog-sidecar is failing to start filebeat. Only PID = 4014017 is started. So I then need to see why graylog-sidecar is not starting filebeat.
There was no config change on graylog-sidecar when rebooting. Maybe hostname changed.

● graylog-sidecar.service - Wrapper service for Graylog controlled collector
   Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-08-25 20:04:56 SAST; 2min 13s ago
 Main PID: 4014017 (graylog-sidecar)
    Tasks: 9 (limit: 102344)
   Memory: 10.3M
   CGroup: /system.slice/graylog-sidecar.service
           └─4014017 /usr/bin/graylog-sidecar

Eventually I deleted the token from the server. Created a new one and attacked it back to the “missing” sidecars. Then the sidecars started showing in the server sidecar dashboard.
Then I could could slap on my config and all good. I also noticed that the server listener was in a different subnet to my sending clients. I changed that. So I am not sure what the real solution is.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.