Filebeat log rotation question

Graylog community 4.1, filebeat 7.10.2

When I set the sidecar paths for logs files what is best practice here?
All logs will log to the initial “first” log → somelog.log and these will subsequently get rolled over to somelog.log.1, somelog.log.2 etc

Should I be harvesting




Any gotchas?


If a file rolls over I personally would use /opt/appname/somelog.log
For instance boot.log will be the newest logs. This also depends on how the logs are named after it rolls over


Hope that helps

mmm my reading is that the key difference is that when I first start up and have no registry, then having


will JUST harvest the current log. And that in theory is all I need. The issue I have is even setting some throttling params, if I use


Then I managed to bring down a prod system at startup, out of memory as it tried to harvest all 50 of the log4j rolling file appender log set. Just wondering if there are any subtle gotchas…

From my understand when you use /opt/appname/somelog* that is every log in that directory that starts with somelog so FileBeat is trying to get some.log, some.-20120809 log etc…
EDIT: really depends on what you want. the gotchas would be using a wild card

Clearly that is the case BUT if I am not interested in the history specifying the one main log file specifically should capture everything going forward. My only concern is how filebeat handles rollover.


My apologies, I misunderstood your question. Maybe this might help.

When an input log file is moved or renamed during log rotation, Filebeat is able to recognize that the file has already been read. After the file is rotated, a new log file is created, and the application continues logging. Filebeat picks up the new file during the next scan. Because the file has a new inode and device name, Filebeat starts reading it from the beginning.

To avoid missing events from a rotated file, configure the input to read from the log file and all the rotated files. For examples, see Example configurations.

Hope that helps

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.