Ingesting .gz log files with filebeat?

I’m new to GrayLog and trying to figure things out. I have a couple collectors working, but I’m not sure how or if the following is possible.

I have a Windows server that throws normal log files from various services to a folder. I’m grabbing those fine. However, there is another service on this server that throws logs in a compressed (.gz) format to a separate folder from the others. Is there any way to get these into GrayLog?

I’m guessing something like this is what I need: Filebeat - On-Demand Loading Compressed (.gz) Files from Stdin - Beats - Discuss the Elastic Stack
But, I’m not sure where to configure something like this with GrayLog’s sidecars. Is this just done within the Collector Configuration GUI?

Since these are on the same server, do I need a separate collector for the .gz files or can I append things to my current one that is collecting the normal log files from the same machine?

I’ve Googled back and forth, but just running into a wall.

Hello @aguisler

Use the same log shipper ( Filebeat)
You can have as many inputs as you want but you can only have one output, you will need to send your logs to a single Graylog INPUT.

Example: Something like this.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log  <------------------------NUMBER-1
  paths:
    - /var/log/nginx/error.log
    - /var/log/someother_logfile.log
  type: log
  
- input_type: stdin <------------------------NUMBER-2
  fields_under_root: true
  fields:
    redis-key: "stdin-nazev-aplikace"
    application.name: "nazev-aplikace"
    application.environment: "production"
    service.type: "application"
    service.name: "tomcat"
logging.level: warning
logging.to_files: true

output.logstash:
   hosts: ["8.8.8.8:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Thanks for the reply @gsmith

I added the new input type to my existing config so it now reads as:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: [${user.c1137Beats}]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\Program Files\Apache Software Foundation\Tomcat 8.5_Tomcat8.5.66\logs\*
- type: stdin
  fields_under_root: true
  fields:
    application.name: "idsmanager"
  logging.to_files: true
  enabled: true
  paths:
    - C:\Program Files\Apache Software Foundation\Tomcat 8.5_Tomcat8.5.66\idsmanager-logs\*

Now, when trying to restart the collector to load the new config, I get this error in the sidecar logs and filebeat will not start at all. Is there a way to edit the access for filebeat within the config or how do I get around this?
If I somehow give filebeat exclusive, won’t that block writing to the file while filebeat is accessing?

time="2022-06-14T10:01:39-05:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file." 
time="2022-06-14T10:01:39-05:00" level=error msg="[filebeat] Collector configuration file is not valid, waiting for the next update." 
time="2022-06-14T10:01:39-05:00" level=error msg="[filebeat] Validation command output: Exiting: stdin requires to be run in exclusive mode, configured inputs: log, stdin\n"

Assuming I’m missing that part at the bottom that essentially decompresses the .gz before filebeat looks at it. I guess I’d have to create a script or something equivalent since this is a Windows server. But then, if that is the case, is this just constantly decompressing whenever filebeat wants to look at it? Doesn’t that nullify the benefit of compressing it in the first place? I’m so confused lol.

I wish I could look at the old ELK servers we had to figure out how it was functioning, but they’re so out of date and nothing is in it’s correct spot.

zcat oracle-logs.2019-05-*.log.gz | filebeat -e -c /etc/filebeat/filebeat-stdin.yml

@aguisler

Sorry about your troubles, I have successfully in the past created two Input but they were not for compressed logs. My demo above was just for something you could try out.

If I was going to execute this, it would be just for compressed (.gz). and try to get it to work. Once successful then add another input into the file.
Couple things to look at if you haven’t already
Graylog-sidecar logs ( which I think you did), Graylog logs to find any clue on what is going on.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.