Hi,
First of all, thanks for the help to setup Graylog with Filebeat. Currently, the configuration allows to ship logs from Filebeat to Graylog by configuring Beats input in Graylog using 5044 port (same as logstash). I have been receiving successfully the logs in Graylog. However, if a log file does not change after 3 retries, filebeat dies and I have to start manually to continue with harvesting.
This is the log in Filebeat before it dies (this fragment appears three times in total):
2019-04-18T17:09:54.324-0700 DEBUG [input] input/input.go:152 Run input
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:174 Start next scan
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/A/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/A/api.log, offset: 2922198
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/A/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/B/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/B/api.log, offset: 28149
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/B/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/C/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/C/api.log, offset: 37930655
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:546 Harvester for file is still running: /app/logs/C/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:404 Check file for harvesting: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:494 Update existing file for harvesting: /app/logs/D/api.log, offset: 173342
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:548 File didn't change: /app/logs/D/api.log
2019-04-18T17:09:54.324-0700 DEBUG [input] log/input.go:195 input states cleaned up. Before: 4, After: 4, Pending: 0
In filebeat.yml this is the configuration for process in port 5044. I changed in filebeat.yml max_retries:3 to -1 (infinite) as follows:
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["log1.cgi-dev.ca:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities:
# Certificate for SSL client authentication
#ssl.certificate:
# Client Certificate Key
#ssl.key:
# The number of times to retry publishing an event after a publishing failure.
# After the specified number of retries, the events are typically dropped.
# Some Beats, such as Filebeat and Winlogbeat, ignore the max_retries setting
# and retry until all events are published. Set max_retries to a value less
# than 0 to retry until all events are published. The default is 3.
max_retries: -1
As I mentioned, I get the logs from filebeat successfully in Graylog, but when the log file does not change for a time, the filebeat process dies. What would be the culprit to cause Filebeat to die?Should I create a daemon to ensure the filebeat process in up and start it if the process goes down?
Thanks