Logs from time before the collector sidecar starting time

Graylog Central (peer support)
1

I’m running a collector sidecar agent to collect logs and send them to my graylog platform. I would like to collect logs lines from date-time previous to the collector sidecar starting time. Is it possible?

2

Yes, that’s possible.

You can configure Filebeat, so that it will also read the existing content of a log file and send it to Graylog:

You’ll probably also need some kind of extractor or pipeline rule in Graylog to parse the timestamp from the log messages and set it as “timestamp” field in your messages.

3

I have set ignore_older to 0. I understand that this option would allow retrieving all messages in the log files “from the beginning” (without the need of any extractor or pipeline) but I do not see this behavior. What am I missing?

Thx!

4

Have you cleaned the state file for that particular log file?

5

I have been struggling with this issue for quite some time and now it turns to be a different one.

I’m trying to read logs from kubernetes and many of them are symbolic links, being the date of the symlinks no rewritable when a new event goes in the log…

I found that there is an option to enable reading from symlinks https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#_literal_symlinks_literal and I’m very interested in using it for the reasons stated there “The symlinks option can be useful if symlinks to the log files have additional metadata in the file name, and you want to process the metadata in Logstash. This is, for example, the case for Kubernetes log files.”.

I’m trying to do a snippet for my collector but I’m not sure how to properly write it. The lines I’m adding go directly to the end of the filebeat.yml and do not seem to have any effect.

Snippet:
filebeat:
prospectors:

  • document_type: log
    symlinks: true

And I was expecting they going into the prospectors already defined:

filebeat:
prospectors:

  • document_type: log
    encoding: plain
    fields:
    gl2_source_collector: xyz
    ignore_older: 0
    input_type: log
    paths:
    • /var/log//.log
    • /var/log///*.log
      scan_frequency: 10s
      tail_files: true
      symlinks: true

Any hint?

Thanks!

6

I’d recommend asking questions related to Filebeat in the Elastic Discussion Forum:

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.