Graylog sidecar and filebeat

tor · August 27, 2021, 4:10pm

So I have been following this documentation.
https://docs.graylog.org/en/4.1/pages/sidecar.html#installation

I have installed filebeat and sidecar on the client system.
Configured sidecar on graylog server and the status is saying running and when i click on the name I see the folder where the logs are but the messages from the logs are not being sent to the global beats input that I created.

I use graylog to edit the filebeat config file. But when I go to the filebeat server the changes are not present in the .yml file.

The service for filebeat and sidecar are running on the client system successfully.

tor · August 27, 2021, 4:14pm

Here is the sidecar.yml
where I added the log file path to gather logs from to send to filebeat.

tor · August 27, 2021, 4:31pm

I noticed in the filebeat logs im getting

2021-08-27T17:00:14.154+0100 INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://192.168.14.17:5044)) with 6 reconnect attempt(s)

I did allow on our firewall to allow graylog server access to the ip and port via tcp. Or is this some other permissions it needs?

gsmith · August 27, 2021, 9:49pm

Hello,

The screenshot of your Graylog-Sidecar configuration does not look right. Might want to go back and check that out, plus that’s not the whole file so I’m not sure if its a configuration error or not.
What you might want to use is this command and then post your file here. When you do please use the editor in Markdown.

grep -v "^#\|^$" /etc/graylog/sidecar/sidecar.yml

Example GL Sidecar:

server_url: "http://8.8.8.8:9000/api/"
server_api_token: "let_me_in_string"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "keycloak"
tls_skip_verify: false
send_status: true
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10

You can also check your FileBeat

sudo systemctl status filebeat

aaronsachs · August 27, 2021, 10:05pm

The sidecar config is also not stored in filebeats default location. You should be able to find your config in /var/lib/graylog-sidecar/generated/filebeat.conf

tor · August 31, 2021, 12:16pm

When i run systemctl status filebeat
filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2021-08-27 17:37:07 BST; 3 days ago
Docs: Filebeat: Lightweight Log Analysis & Elasticsearch | Elastic
Main PID: 7771 (filebeat)
CGroup: /system.slice/filebeat.service
└─7771 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Aug 31 13:07:07 collector filebeat[7771]: 2021-08-31T13:07:07.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78220,“time”:{“ms”:6}},“total”:{“ticks”:157610,"time…
Aug 31 13:07:37 collector filebeat[7771]: 2021-08-31T13:07:37.939+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78230,“time”:{“ms”:4}},“total”:{“ticks”:157640,"time…
Aug 31 13:08:07 collector filebeat[7771]: 2021-08-31T13:08:07.939+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78240,“time”:{“ms”:10}},“total”:{“ticks”:157660,"tim…
Aug 31 13:08:37 collector filebeat[7771]: 2021-08-31T13:08:37.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78240,“time”:{“ms”:4}},“total”:{“ticks”:157660,"time…
Aug 31 13:09:07 collector filebeat[7771]: 2021-08-31T13:09:07.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78250,“time”:{“ms”:4}},“total”:{“ticks”:157680,"time…
Aug 31 13:09:37 collector filebeat[7771]: 2021-08-31T13:09:37.939+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78250,“time”:{“ms”:10}},“total”:{“ticks”:157690,"tim…
Aug 31 13:10:07 collector filebeat[7771]: 2021-08-31T13:10:07.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78260,“time”:{“ms”:6}},“total”:{“ticks”:157710,"time…
Aug 31 13:10:37 collector filebeat[7771]: 2021-08-31T13:10:37.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78270,“time”:{“ms”:8}},“total”:{“ticks”:157720,"time…
Aug 31 13:11:07 collector filebeat[7771]: 2021-08-31T13:11:07.939+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78270,“time”:{“ms”:6}},“total”:{“ticks”:157720,"time…
Aug 31 13:11:37 collector filebeat[7771]: 2021-08-31T13:11:37.940+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:78280,“time”:{“ms”:8}},“total”:{“ticks”:157750,"time…
Hint: Some lines were ellipsized, use -l to show in full.

tor · August 31, 2021, 12:16pm

why is it seeing usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

and not the sidecar filebeat?

tor · August 31, 2021, 12:17pm

Thanks for this location it was useful

tor · August 31, 2021, 12:19pm

my sidecar is running fine and seeing the filebeat.conf file
systemctl status graylog-sidecar
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-08-31 12:48:16 BST; 30min ago
Main PID: 8627 (graylog-sidecar)
CGroup: /system.slice/graylog-sidecar.service
├─8627 /usr/bin/graylog-sidecar
└─8652 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/filebeat.conf

Aug 31 12:48:16 collector systemd[1]: Started Wrapper service for Graylog controlled collector.
Aug 31 12:48:16 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:16+01:00” level=info msg=“Using node-id: 0c38f1bb-d012-41ef-8598-c8cfcb1fca4c”
Aug 31 12:48:16 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:16+01:00” level=info msg=“No node name was configured, falling back to hostname”
Aug 31 12:48:16 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:16+01:00” level=info msg=“Starting signal distributor”
Aug 31 12:48:26 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:26+01:00” level=info msg=“Adding process runner for: filebeat”
Aug 31 12:48:26 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:26+01:00” level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
Aug 31 12:48:26 collector graylog-sidecar[8627]: time=“2021-08-31T12:48:26+01:00” level=info msg="[filebeat] Starting (exec driver)"

tor · August 31, 2021, 12:20pm

grep -v “^#|^$” /etc/graylog/sidecar/sidecar.yml
server_url: “http://192.168.1.48:9400/api/”
server_api_token: “19r7feq83fbnpr3j95tcnh82ocnp3i6oqodvc9humchlma545fj3”
list_log_files:

“/Volumes/assets/logs/dataiosync/”
log_path: “/var/log/graylog-sidecar”
log_rotate_max_file_size: “10MiB”
log_rotate_keep_files: 10

tor · August 31, 2021, 1:35pm

So in terms of installing filebeat I just installed it and ran the service I did not configure filebeat in any way as I thought the generated filebeat.conf would take care of it. Do I need to configure and edit the filebeat.yml file and then run the filebeat service?

tmacgbay · August 31, 2021, 6:28pm

Please use the tools in the toolbar when you post code, it makes it SO much easier to read!

In particular the </> tool…

I may have been reading it incorrectly but did you install filebeat separately? The sidecar installation has filebeat built into the install, you should work from that and uninstall the standalone filebeat. Only the sidecar should be sending data (via it’s filebeat) to the Graylog server… Graylog manages the information and handles the transfer/storage (etc.) to the Elasticsearch server.

EDIT: Correction - sidecar Linux does NOT currently include the filebeat, you download and install from Elastic but don’t need to activate or start the service since sidecar manages that.

tor · August 31, 2021, 7:46pm

This is good to know that I do not need a separate filebeat install. I will remove this.

Do you think there may be conflicts and that is why the inputs in graylog server sees no messages being sent?

gsmith said in a post earlier I should check the sysyemctl status filebeat. But if I remove filebeat then this service will not be present anymore.

Also i followed the sidecar install docs and when it talked about filebeat it had a link to filebeat standalone. This made me think that I need to install this separately as well. Not very clear in the docs then.

tmacgbay · August 31, 2021, 8:31pm

So you are right, you do install the filebeat separately … I am used to the windows side where it is included. On the other hand you do not start the service, you are installing to place the binaries for Graylog to use. Here is the status of the installed and inactive filebeat on my ubuntu system

$ sudo systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://www.elastic.co/products/beats/filebeat

tor · August 31, 2021, 8:54pm

Thanks for that. The logs I’m sending have been created by a inhouse script to show data transfer of files from one server to another. Do you know if the logs have to be a certain format for graylog to ingest them? Or will graylog accept all logs via the beats input If the log location is specified in the filebeat.cfg file for sidecar?

gsmith · August 31, 2021, 9:53pm

Hello,

FileBeat will send log to a Graylog Input. you need to configure you FileBeat.yml file accordingly.

You would want to match the Graylog Input with the message being sent.

I believe so, but read this it will tell you what beats are accepted.
https://docs.graylog.org/en/4.0/pages/sending/input/beats.html

The community creates an additional wide range of beats.

tor · September 1, 2021, 9:56am

On my sidecar server i have turned off the filebeat service as the binary is only required as per this thread by tmacgbay.

Now are you saying I still need to configure the filebeat.yml file that comes with the standalone filebeat install when filebeat is installed?

tor · September 1, 2021, 10:07am

In the docs its says Mode of Operation

When the Sidecar is assigned a configuration via the Graylog web interface, it will write a configuration file into the collector_configuration_directory directory for each collector backend. E.g. if you assigned a Filebeat collector you will find a filebeat.yml file in that directory. All changes have to be made in the Graylog web interface. Every time the Sidecar detects an update to its configuration it will rewrite the corresponding collector configuration file. Manually editing these files is not recommended.

I do not see a filbeat.yml file only the filebeat.conf file that I created through graylog. Now should this be .conf file or a .yml file?

tor · September 1, 2021, 10:07am

location im looking in is /var/lib/graylog-sidecar/generated

tmacgbay · September 1, 2021, 11:22am

They way I have it set up:

Filebeat and Sidecar installed
Changed /etc/graylog/sidecar/sidecar.yml so it sees Graylog server and filebeat bin
Set only sidecar as active and started. (I never touched the filebeat.yml)

If your sidecar.yml is set up properly (example of mine below) once graylog-sidecar starts it will notify the Graylog server it is ready for comminutication. The rest of your configuration happens in the Graylog in system->sidecars->configuration. You need a Log Collector (example below) configuration (filebeat, linux) that you will use for the basis of your configuration. In the configuration you define what you want to send to Graylog (example below) then under System->Sidecars->Administration you should find the sidecar hostname, check off box of the collector type you want to apply, pull down the configure menu on the right and apply the configuration you want. Graylog handles the rest. The sidecar you installed will be told of the change, it will copy it the configuration locally from the Graylog server to /var/lib/graylog-sidecar/generated and use it. If your input is set up correctly, you should see data flowing in as it is created and found by the sidecar. Future changes to configuration are handled in Graylog GUI

sidecar.yml

$ grep -v "^#\|^$" /etc/graylog/sidecar/sidecar.yml
server_url: "http://cmg-gl01:9000/api/"
server_api_token: "<secret stuff>"
update_interval: 10
tls_skip_verify: true
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"
collector_binaries_whitelist:
  - "/usr/share/filebeat/bin/filebeat"

Log Collector:

Filebeat collector configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/auth.log
  ignore_older: 72h
  tags:
    - linux
# custom fields
  fields:
    workspace: hifreq

output.logstash:
   hosts: 
   - ${user.BeatsInput}

path:
  data: /var/cache/graylog-sidecar/filebeat/data
  logs: /var/log/graylog-sidecar

Topic		Replies	Views
Unabble to find the Greylog sidecar node logs Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx	11	1539	November 11, 2020
Filebeat and linux messages file Graylog Central (peer support) sidecar , filebeat-linux , nosendlogfblx	14	3120	August 27, 2020
Filebeat logs dropped out Graylog Central (peer support) pipeline-rules , route-to-streampl , debuggingpl , sidecar , filebeat-windows	10	3399	February 8, 2018
Graylog 3, Sidecar (1.0.1) WinLogEvt - No messages Graylog Central (peer support) sidecar , filebeat-windows , winlogbeat	4	1361	May 17, 2019
Graylog Server not receiving messages from sidecar Graylog Central (peer support) sidecar , filebeat-linux , basic-configuration	2	1665	March 23, 2022

Graylog sidecar and filebeat

Related topics