Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
After running the client that Graylog collects logs from for many months, I had to perform a reboot. After the reboot, I noticed that filebeat is no longer working under graylog-sidecar.
2. Describe your environment:
- OS Information:
Rocky Linux 8.8 - Package Version:
filebeat-7.10.0-x86_64.rpm
graylog-sidecar-1.2.0-1.x86_64.rpm - Service logs, configurations, and environment variables:
Default configs as per Graylog 4.x instruction are used for the sidecar.
Upon reboot, graylog-sidecar stopped filebeat. Then when the OS came up again, filebeat stopped working and no longer ships logs to Graylog.
2023-08-24T22:15:38.254+0200 INFO beater/filebeat.go:515 Stopping filebeat
2023-08-24T22:15:38.254+0200 INFO beater/crawler.go:148 Stopping Crawler
2023-08-24T22:15:38.255+0200 INFO beater/crawler.go:158 Stopping 1 inputs
2023-08-24T22:15:38.255+0200 INFO [crawler] beater/crawler.go:163 Stopping input: 3121193877724421705
2023-08-24T22:15:38.255+0200 INFO input/input.go:136 input ticker stopped
2023-08-24T22:15:38.255+0200 INFO beater/crawler.go:178 Crawler stopped
2023-08-24T22:15:38.255+0200 INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2023-08-24T22:15:38.255+0200 INFO [registrar] registrar/registrar.go:166 Ending Registrar
2023-08-24T22:15:38.255+0200 INFO [registrar] registrar/registrar.go:137 Registrar stopped
2023-08-24T22:15:38.277+0200 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cgroup":{"memory":{"id":"graylog-sidecar.service","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":362156032}}}},"cpu":{"system":{"ticks":1403790,"time":{"ms":1403790}},"total":{"ticks":2933610,"time":{"ms":2933618},"value":2933610},"user":{"ticks":1529820,"time":{"ms":1529828}}},"handles":{"limit":{"hard":262144,"soft":1024},"open":12},"info":{"ephemeral_id":"dae0c2be-8a7e-4e79-8fa2-d0cf81954ce5","uptime":{"ms":14074924324}},"memstats":{"gc_next":17803648,"memory_alloc":10381496,"memory_total":185118979296,"rss":87212032},"runtime":{"goroutines":11}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":2},"load":{"1":0.45,"15":0.4,"5":0.77,"norm":{"1":0.225,"15":0.2,"5":0.385}}}}}}
2023-08-24T22:15:38.278+0200 INFO [monitoring] log/log.go:154 Uptime: 3909h42m4.325894382s
2023-08-24T22:15:38.278+0200 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2023-08-24T22:15:38.278+0200 INFO instance/beat.go:461 filebeat stopped.
3. What steps have you already taken to try and solve the problem?
So far I have reinstalled filebeat and graylog-sidecar but I am still stuck.
4. How can the community help?
Has anyone experienced this mystery?
What is the best way to debug this filebeat issue on the client? Is there a way to activate debug mode for graylog-sidecar?
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]