Graylog community 4.1, filebeat 7.10.2
When I set the sidecar paths for logs files what is best practice here?
All logs will log to the initial “first” log → somelog.log and these will subsequently get rolled over to somelog.log.1, somelog.log.2 etc
Should I be harvesting
/opt/appname/somelog.log
OR
/opt/appname/somelog*
Any gotchas?
Hello,
If a file rolls over I personally would use /opt/appname/somelog.log
For instance boot.log will be the newest logs. This also depends on how the logs are named after it rolls over
boot.log
boot.log-20210401
boot.log-20210408
boot.log-20210607
boot.log-20210715
boot.log-20210803
boot.log-20210911
boot.log-20210924
Hope that helps
mmm my reading is that the key difference is that when I first start up and have no registry, then having
/opt/appname/somelog.log
will JUST harvest the current log. And that in theory is all I need. The issue I have is even setting some throttling params, if I use
/opt/appname/somelog*
Then I managed to bring down a prod system at startup, out of memory as it tried to harvest all 50 of the log4j rolling file appender log set. Just wondering if there are any subtle gotchas…
From my understand when you use /opt/appname/somelog* that is every log in that directory that starts with somelog so FileBeat is trying to get some.log, some.-20120809 log etc…
EDIT: really depends on what you want. the gotchas would be using a wild card
Clearly that is the case BUT if I am not interested in the history specifying the one main log file specifically should capture everything going forward. My only concern is how filebeat handles rollover.
Hello
My apologies, I misunderstood your question. Maybe this might help.
When an input log file is moved or renamed during log rotation, Filebeat is able to recognize that the file has already been read. After the file is rotated, a new log file is created, and the application continues logging. Filebeat picks up the new file during the next scan. Because the file has a new inode and device name, Filebeat starts reading it from the beginning.
To avoid missing events from a rotated file, configure the input to read from the log file and all the rotated files. For examples, see Example configurations.
Hope that helps
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.