Thank you for this macgbay.
I unhashed the filebeat binary in sidecar.yml. The sidecar service starts but i see a warning
Sep 01 12:43:12 collector graylog-sidecar[9362]: time=“2021-09-01T12:43:12+01:00” level=warning msg="collector_binaries_whitelist is deprecated. Migrate your configuration to collector_binaries_accesslist."
Do you know where this accesslist is?
this is my sidecar.yml output
server_url: “http://192.168.1.48:9400/api/”
server_api_token: “”
node_id: “file:/etc/graylog/sidecar/node-id”
node_name: “collector”
update_interval: 5
tls_skip_verify: false
send_status: true
list_log_files:
As you can see from the screen capture log folder and log files can be seen in graylog just the input has no messages coming to it.
my log entry:
2021-09-01T13:17:07.805+0100 INFO instance/beat.go:473 filebeat start running.
2021-09-01T13:17:07.805+0100 INFO memlog/store.go:119 Loading data file of ‘/var/lib/graylog-sidecar/collectors/filebeat/data/registry/filebeat’ succeeded. Active transaction id=0
2021-09-01T13:17:07.810+0100 INFO memlog/store.go:124 Finished loading transaction log file for ‘/var/lib/graylog-sidecar/collectors/filebeat/data/registry/filebeat’. Active transaction id=159
2021-09-01T13:17:07.811+0100 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-09-01T13:17:07.811+0100 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 5
2021-09-01T13:17:07.811+0100 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2021-09-01T13:17:07.811+0100 WARN [cfgwarn] input/config.go:42 DEPRECATED: input_type input config is deprecated. Use type instead. Will be removed in version: 6.0.0
2021-09-01T13:17:07.812+0100 INFO [input] log/input.go:164 Configured paths: [/Volumes/assets/logs/dataiosync] {“input_id”: “7a9f0555-2a8a-4437-8a45-ed88dc4a361c”}
2021-09-01T13:17:07.812+0100 INFO [crawler] beater/crawler.go:141 Starting input (ID: 14655181980806047023)
2021-09-01T13:17:07.812+0100 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
collector_binaries_whitelist → collector_binaries_accesslist. My guess is it’s just a name change… I likely won’t hit that until I update my Linux sidecars.
your sidecar.yml output looks OK except you forgot to use the forum tools to make it readable… more specifically </> Your sidecar shows up in Graylog so it’s communicating.
What is the Collector Configuration you applied to the machine with sidecar installed? (use the </> tool!!)
sorry ill use the tool
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
filebeat.inputs:
- input_type: log
paths:
- /Volumes/assets/logs/dataiosync/
type: log
output.logstash:
hosts: ["${user.BeatsinputServer}"]
path:
data: /var/lib/graylog-sidecar/collectors/filebeat/data
logs: /var/lib/graylog-sidecar/collectors/filebeat/log or paste code here
Sorry where is the collector_binaries_whitelist → collector_binaries_accesslist ? As I do not see that
sorry re-read what you wrote. Understood.
It’s just a name change in the sidecar.yml config file… change the angsty word bit “whitelist” to the safe word bit of “accesslist”.
You may need your “paths:” to be: /Volumes/assets/logs/dataiosync/* using a * at the end so as to include all files in that directory… or /Volumes/assets/logs/dataiosync/*.log to only capture files ending in .log. Also, though it may be just a posting typo, remove the “or paste code here” from the last line in the config.
That was a typo in pasting.
So it looks like the path of my logs I have to edit in 3 places.
When I edit in 3) and add *.log at the end of the path the service for sidecar fails to start.
So its seems my sidecar is all configured correctly as its running and I can see that sidecar does see the logs in the log file. Its only my inputs is not picking any messages up.
This is a screen shot from sidecar collector status
The path of logs you want to capture need only be set/changed in the UI of the Graylog server in the collector configuration. Graylog will automatically manage updating all sidecars that have been assigned the configuration.
The do I remove that path in /etc/graylog/sidecar/sidecar.yml
and leave it in the filebeat.cfg file
Make sure that ${user.BeatsInput} points to the server and port that corresponds to your input on Graylog
Just to be clear all this time I had the sidecar server ip and port in user.BeatsInput. I have now changd that to graylog server ip and port I specified for the beats input.
Is that correct?
could it be because i have no extractors setup in my input?
The line in my Collector Configuration that is applied to my Linux servers that tells them where to send looks like so:
output.logstash:
hosts:
- ${user.BeatsInput}
which corresponds to:
You don’t have to use ${user.BeatsInput}, you could just put your FQDN and port in directly.