# Ingesting .gz log files with filebeat?

**URL:** <https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286>\
**Category:** Graylog Central (peer support)\
**Tags:** sidecar, filebeat-windows\
**Created:** [June 13, 2022, 8:21pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286 "2022-06-13T20:21:27Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![aguisler](https://avatars.discourse-cdn.com/v4/letter/a/ea666f/32.png) [@aguisler](https://community.graylog.org/u/aguisler)\
**Post date:** [June 13, 2022, 8:21pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/1 "2022-06-13T20:21:27Z")

</div>

I’m new to GrayLog and trying to figure things out. I have a couple collectors working, but I’m not sure how or if the following is possible.

I have a Windows server that throws normal log files from various services to a folder. I’m grabbing those fine. However, there is another service on this server that throws logs in a compressed (.gz) format to a separate folder from the others. Is there any way to get these into GrayLog?

I’m guessing something like this is what I need: [Filebeat - On-Demand Loading Compressed (.gz) Files from Stdin - Beats - Discuss the Elastic Stack](https://discuss.elastic.co/t/filebeat-on-demand-loading-compressed-gz-files-from-stdin/184159)  
But, I’m not sure where to configure something like this with GrayLog’s sidecars. Is this just done within the Collector Configuration GUI?

Since these are on the same server, do I need a separate collector for the .gz files or can I append things to my current one that is collecting the normal log files from the same machine?

I’ve Googled back and forth, but just running into a wall.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [June 14, 2022, 1:36am UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/2 "2022-06-14T01:36:08Z")

</div>

Hello @aguisler

> [@aguisler](#):
>
> or can I append things to my current one that is collecting the normal log files from the same machine?

Use the same log shipper ( Filebeat)  
You can have as many inputs as you want but you can only have one output, you will need to send your logs to a single Graylog INPUT.

Example: Something like this.

```auto
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log <------------------------NUMBER-1
  paths:
    - /var/log/nginx/error.log
    - /var/log/someother_logfile.log
  type: log
  
- input_type: stdin <------------------------NUMBER-2
  fields_under_root: true
  fields:
    redis-key: "stdin-nazev-aplikace"
    application.name: "nazev-aplikace"
    application.environment: "production"
    service.type: "application"
    service.name: "tomcat"
logging.level: warning
logging.to_files: true

output.logstash:
   hosts: ["8.8.8.8:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

```

---

<div class="post-metadata">

**Author:** ![aguisler](https://avatars.discourse-cdn.com/v4/letter/a/ea666f/32.png) [@aguisler](https://community.graylog.org/u/aguisler)\
**Post date:** [June 14, 2022, 3:19pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/3 "2022-06-14T15:19:29Z")

</div>

Thanks for the reply @gsmith

I added the new input type to my existing config so it now reads as:

```auto
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: [${user.c1137Beats}]
path:
  data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
  logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\Program Files\Apache Software Foundation\Tomcat 8.5_Tomcat8.5.66\logs\*
- type: stdin
  fields_under_root: true
  fields:
    application.name: "idsmanager"
  logging.to_files: true
  enabled: true
  paths:
    - C:\Program Files\Apache Software Foundation\Tomcat 8.5_Tomcat8.5.66\idsmanager-logs\*

```

Now, when trying to restart the collector to load the new config, I get this error in the sidecar logs and filebeat will not start at all. Is there a way to edit the access for filebeat within the config or how do I get around this?  
If I somehow give filebeat exclusive, won’t that block writing to the file while filebeat is accessing?

```auto
time="2022-06-14T10:01:39-05:00" level=info msg="[filebeat] Configuration change detected, rewriting configuration file." 
time="2022-06-14T10:01:39-05:00" level=error msg="[filebeat] Collector configuration file is not valid, waiting for the next update." 
time="2022-06-14T10:01:39-05:00" level=error msg="[filebeat] Validation command output: Exiting: stdin requires to be run in exclusive mode, configured inputs: log, stdin\n"

```

---

<div class="post-metadata">

**Author:** ![aguisler](https://avatars.discourse-cdn.com/v4/letter/a/ea666f/32.png) [@aguisler](https://community.graylog.org/u/aguisler)\
**Post date:** [June 14, 2022, 7:21pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/4 "2022-06-14T19:21:13Z")

</div>

Assuming I’m missing that part at the bottom that essentially decompresses the .gz before filebeat looks at it. I guess I’d have to create a script or something equivalent since this is a Windows server. But then, if that is the case, is this just constantly decompressing whenever filebeat wants to look at it? Doesn’t that nullify the benefit of compressing it in the first place? I’m so confused lol.

I wish I could look at the old ELK servers we had to figure out how it was functioning, but they’re so out of date and nothing is in it’s correct spot.

```auto
zcat oracle-logs.2019-05-*.log.gz | filebeat -e -c /etc/filebeat/filebeat-stdin.yml

```

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [June 14, 2022, 10:41pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/5 "2022-06-14T22:41:08Z")

</div>

@aguisler

Sorry about your troubles, I have successfully in the past created two Input but they were not for compressed logs. My demo above was just for something you could try out.

If I was going to execute this, it would be just for compressed (.gz). and try to get it to work. Once successful then add another input into the file.  
Couple things to look at if you haven’t already  
Graylog-sidecar logs ( which I think you did), Graylog logs to find any clue on what is going on.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [June 28, 2022, 10:41pm UTC](https://community.graylog.org/t/ingesting-gz-log-files-with-filebeat/24286/6 "2022-06-28T22:41:55Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.