Filebeat configuration to forward the content of log files to graylog to create one entry per file

[phil]

Hello,

I have a question regarding filebeat and graylog configuration. I have configured a windows machine that is recording siebel DB transactions in log files to send those log files from a specific path to logstash, and then logstash forwards them to graylog. This is working.
The problem is that log files on the source machine have the same names (e.g. fileA has new content appended by the siebel) name until a period of time (probably until a server restart) while the log files updated (new content appended) every time a new activity occurs by the user. This means, filebeat sends each time the new appended content to graylog. That behaviour is not the preferred one because for 1 file I may end to have more than 10 entries in my graylog which may not contain useful information for auditing.

Is there any way to force filebeat send me the files once per day (at least to configure a specific time) or append the new appended content that arrives for the same log filename to the entry that corresponds to that log file? I want to end up with one entry per file and the “message” field of graylog should contain the whole content of the file.
I know that after server restart, the log files from the source machine are archived and one solution should be to retrieve those files to graylog but this is not recommended because server restart is not scheduled.

Thank you!

[tmacgbay]

Its not exactly what you are asking for but you could exclude certain lines from being transmitted in the beat configuration - here is an example from one of mine:

    - type: log
      enabled: true
      include_lines: ['smoot.length']
      exclude_lines: ['header','^#','^rail','spraypaint','10.99.[0-9]+.[0-9]+']
      fields:
        unique_log_tag: bridgelet
      ignore_older: 72h

You may find more this stuff and of what you are looking for here:

Log input | Filebeat Reference [7.13] | Elastic

[phil]

Thanks for your idea. Unfortunately it is not clear for me how to use your solution to fit my problem.

I notice that in siebel DB logging the header of each file contains “creation timestamp” and “completion timestamp”. If a file is still active, the “completion timestamp” has all zero values. Is there any way to check the file every X time for the “completion timestamp” header field and if it changes to NOT zero to start harvesting and reading it to send to graylog?

[tmacgbay]

Having the message contain the whole content of the file - there is a limit on size, which could be a problem (32kb I have seen in the forums)

I was thinking if you were only interested in particular information in the siebel logs, you could use the inlcude_lines and/or exclude_lines to pre parse what you want and ignore what you don’t. Filebeat would only send what you want. In the filebeat reference link, you may find a way to tell it to only send for the periodicity you want…