Graylog-sidecar filebeat file name

[psargent]

I am trying to replace collectors(legacy) with graylog-sidecar. It is working but I can’t seem to figure out how to replace the metadata that was “just there” with the legacy collectors. In particular, the file field with the file name is missing in the sidecar configuration.

• OS Information: centos 7

• Package Version:
  graylog server 4.2.5
  graylog-sidecar 1.1.0
  filebeat 7.16.2

• Service logs, configurations, and environment variables:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: filestream
  paths:
    - '/var/log/httpd/*_log'
  fields:
    source: ${sidecar.nodeName}
    name: ${sidecar.nodeName}
    gl2_source_collector: ${sidecar.nodeId}
    application: interface_apache
    @source:
- type: filestream
  paths:
    - '/var/log/drupal/apps/*/*.log'
  fields:
    source: ${sidecar.nodeName}
    name: ${sidecar.nodeName}
    gl2_source_collector: ${sidecar.nodeId}
    application: interface_drupal
- type: filestream
  paths:
    - /var/local/newsbank/metrics/*.dat
  fields:
    source: ${sidecar.nodeName}
    name: ${sidecar.nodeName}
    gl2_source_collector: ${sidecar.nodeId}
- type: filestream
  paths:
    - '/local/IBMHTTPD/logs/*.log'
  fields:
    source: ${sidecar.nodeName}
    name: ${sidecar.nodeName}
    gl2_source_collector: ${sidecar.nodeId}
    application: platform_weblogs
- type: filestream
  paths:
    - '/var/log/samba/log.*'
  fields:
    source: ${sidecar.nodeName}
    name: ${sidecar.nodeName}
    gl2_source_collector: ${sidecar.nodeId}
    application: platform_weblogs
  include_lines: ['\sservice\s']
  exclude_files: ['/var/log/samba/log.0.0.0.0::UNKNOWN:UNKNOWN']
output.logstash:
   hosts: ["server01:5044","server02:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

3. What steps have you already taken to try and solve the problem?
In the fields section of the config I have tried source, @source, beat.source

4. How can the community help?
How can I get the graylog-sidecar to pass the log filename to graylog?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[tmacgbay]

(I edited the data using the </> forum tools feature)

In the filebeats that I am running (v7.11.2) it automatically creates a field called log_file_path that contains the path and file the message was pulled from.

Is that what you were looking for?

If you just wanted the filename you could create an extractor or pipeline rule that would clip out the path…

[psargent]

Thank you for your reply. I don’t seem to have a field called log_file_path. How do I get that into graylog? Does it have anything to do with changing the input type from log to filestream? Elastic says log input is deprecated. Update: I tried changing it back to type log and still don’t have log_file_path so that is not it.

[tmacgbay]

What version of Elasticsearch are you running?

[psargent]

elasticsearch 6.8.22

I ran filebeat with a -d “*” debug all and I do see filebeat has the filename but it’s not getting into a field on the graylog side

This is part of the debug output

	"@timestamp": "2022-01-27T22:01:04.360Z",
	"@metadata": {
		"beat": "filebeat",
		"type": "_doc",
		"version": "7.16.2"
	},
	"log": {
		"offset": 166742,
		"file": {
			"path": "/var/log/httpd/access_log"
		}
	},

[psargent]

Thank you for your help! I figured it out. I’ve been upgrading since graylog v 1. I was using a depreciated beats input. Once I set up a new beats input I am getting the metadata!

[tmacgbay]

Great! Don’t update Elasticsearch beyond 7.10.2, it is unsupported after that!