Graylog Filebeat log collector failed after MongoDB Restore

[duese]

Hello,

in my working Graylog ist the MongoDB crashed. Now I start with a clean DB and restore a working Backup from February. Unfortunately the log collector does not work, so that filebeat doesent work.
I miss the Direktory /etc/filebeat/ with the filebeat.yml, but before my restore in the Working Area, there wasn’t that Directory likewise.
Has anyone a Idea.

• Ubuntu 20.04
• Graylog 4.1.3
• MongoDB v4.0.25
• Elasticsearch 7.10.2

In one of the filebeat is that log:
2021-08-17T07:42:00.570+0200 INFO instance/beat.go:611 Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/var/lib/graylog-sidecar/collectors/filebeat/data] Logs path: [/var/lib/graylog-sidecar/collectors/filebeat/log]
2021-08-17T07:42:00.571+0200 INFO instance/beat.go:618 Beat UUID: 1e30b2fd-023a-4a04-9164-26b405e83244
2021-08-17T07:42:00.571+0200 INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2021-08-17T07:42:00.571+0200 INFO [beat] instance/beat.go:931 Beat info {“system_info”: {“beat”: {“path”: {“config”: “/usr/share/filebeat/bin”, “data”: “/var/lib/graylog-sidecar/collectors/filebeat/data”, “home”: “/usr/share/filebeat/bin”, “logs”: “/var/lib/graylog-sidecar/collectors/filebeat/log”}, “type”: “filebeat”, “uuid”: “1e30b2fd-023a-4a04-9164-26b405e83244”}}}
2021-08-17T07:42:00.571+0200 INFO [beat] instance/beat.go:940 Build info {“system_info”: {“build”: {“commit”: “5cd281153df1eb5e95a4a31994a7846d2c9493e8”, “libbeat”: “6.8.14”, “time”: “2021-02-02T18:46:23.000Z”, “version”: “6.8.14”}}}
2021-08-17T07:42:00.571+0200 INFO [beat] instance/beat.go:943 Go runtime info {“system_info”: {“go”: {“os”:“linux”,“arch”:“amd64”,“max_procs”:8,“version”:“go1.10.8”}}}
2021-08-17T07:42:00.572+0200 INFO [beat] instance/beat.go:947 Host info {“system_info”: {“host”: {“architecture”:“x86_64”,“boot_time”:“2021-08-16T12:10:28+02:00”,“containerized”:false,“name”:“kvit-graylog”,“ip”:[“127.0.0.1/8”,"::1/128",“172.22.23.239/24”,“fe80::a846:96ff:fe9d:d870/64”],“kernel_version”:“5.4.0-80-generic”,“mac”:[“aa:46:96:9d:d8:70”],“os”:{“family”:“debian”,“platform”:“ubuntu”,“name”:“Ubuntu”,“version”:“20.04.2 LTS (Focal Fossa)”,“major”:20,“minor”:4,“patch”:2,“codename”:“focal”},“timezone”:“CEST”,“timezone_offset_sec”:7200,“id”:“83cd4f68db0a403f8c24289cd8790e0b”}}}
2021-08-17T07:42:00.572+0200 INFO [beat] instance/beat.go:976 Process info {“system_info”: {“process”: {“capabilities”: {“inheritable”:null,“permitted”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“effective”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“bounding”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“ambient”:null}, “cwd”: “/”, “exe”: “/usr/share/filebeat/bin/filebeat”, “name”: “filebeat”, “pid”: 4306, “ppid”: 753, “seccomp”: {“mode”:“filter”,“no_new_privs”:true}, “start_time”: “2021-08-17T07:41:59.899+0200”}}}
2021-08-17T07:42:00.572+0200 INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.14
2021-08-17T07:42:00.572+0200 INFO [publisher] pipeline/module.go:110 Beat name: kvit-graylog
2021-08-17T07:42:00.572+0200 ERROR fileset/modules.go:118 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2021-08-17T07:42:00.572+0200 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-08-17T07:42:00.572+0200 INFO instance/beat.go:402 filebeat start running.
2021-08-17T07:42:00.572+0200 INFO registrar/registrar.go:134 Loading registrar data from /var/lib/graylog-sidecar/collectors/filebeat/data/registry
2021-08-17T07:42:00.574+0200 INFO [monitoring] log/log.go:152 Total non-zero metrics {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:0,“time”:{“ms”:5}},“total”:{“ticks”:10,“time”:{“ms”:16},“value”:10},“user”:{“ticks”:10,“time”:{“ms”:11}}},“handles”:{“limit”:{“hard”:524288,“soft”:1024},“open”:6},“info”:{“ephemeral_id”:“6358755f-40b0-4e8d-a331-bd242bb39079”,“uptime”:{“ms”:8}},“memstats”:{“gc_next”:4194304,“memory_alloc”:2412080,“memory_total”:3867864,“rss”:23019520}},“filebeat”:{“harvester”:{“open_files”:0,“running”:0}},“libbeat”:{“config”:{“module”:{“running”:0}},“output”:{“type”:“logstash”},“pipeline”:{“clients”:0,“events”:{“active”:0}}},“registrar”:{“states”:{“current”:0}},“system”:{“cpu”:{“cores”:8},“load”:{“1”:0.01,“15”:0,“5”:0.01,“norm”:{“1”:0.0013,“15”:0,“5”:0.0013}}}}}}
2021-08-17T07:42:00.574+0200 INFO [monitoring] log/log.go:153 Uptime: 9.594923ms
2021-08-17T07:42:00.574+0200 INFO [monitoring] log/log.go:130 Stopping metrics logging.
2021-08-17T07:42:00.574+0200 INFO instance/beat.go:412 filebeat stopped.
2021-08-17T07:42:00.575+0200 ERROR instance/beat.go:906 Exiting: Could not start registrar: Error loading state: Error decoding states: EOF

Now I get after creating a Symlink /usr/share/filebeat/bin/module of /usr/share/filebeat/module/

2021-08-17T10:48:57.473+0200 INFO instance/beat.go:611 Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/var/lib/graylog-sidecar/collectors/filebeat/data] Logs path: [/var/lib/graylog-sidecar/collectors/filebeat/log]
2021-08-17T10:48:57.474+0200 INFO instance/beat.go:618 Beat UUID: 1e30b2fd-023a-4a04-9164-26b405e83244
2021-08-17T10:48:57.474+0200 INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2021-08-17T10:48:57.474+0200 INFO [beat] instance/beat.go:931 Beat info {“system_info”: {“beat”: {“path”: {“config”: “/usr/share/filebeat/bin”, “data”: “/var/lib/graylog-sidecar/collectors/filebeat/data”, “home”: “/usr/share/filebeat/bin”, “logs”: “/var/lib/graylog-sidecar/collectors/filebeat/log”}, “type”: “filebeat”, “uuid”: “1e30b2fd-023a-4a04-9164-26b405e83244”}}}
2021-08-17T10:48:57.474+0200 INFO [beat] instance/beat.go:940 Build info {“system_info”: {“build”: {“commit”: “5cd281153df1eb5e95a4a31994a7846d2c9493e8”, “libbeat”: “6.8.14”, “time”: “2021-02-02T18:46:23.000Z”, “version”: “6.8.14”}}}
2021-08-17T10:48:57.474+0200 INFO [beat] instance/beat.go:943 Go runtime info {“system_info”: {“go”: {“os”:“linux”,“arch”:“amd64”,“max_procs”:8,“version”:“go1.10.8”}}}
2021-08-17T10:48:57.475+0200 INFO [beat] instance/beat.go:947 Host info {“system_info”: {“host”: {“architecture”:“x86_64”,“boot_time”:“2021-08-17T10:48:26+02:00”,“containerized”:false,“name”:“kvit-graylog”,“ip”:[“127.0.0.1/8”,"::1/128",“172.22.23.239/24”,“fe80::a846:96ff:fe9d:d870/64”],“kernel_version”:“5.4.0-80-generic”,“mac”:[“aa:46:96:9d:d8:70”],“os”:{“family”:“debian”,“platform”:“ubuntu”,“name”:“Ubuntu”,“version”:“20.04.2 LTS (Focal Fossa)”,“major”:20,“minor”:4,“patch”:2,“codename”:“focal”},“timezone”:“CEST”,“timezone_offset_sec”:7200,“id”:“83cd4f68db0a403f8c24289cd8790e0b”}}}
2021-08-17T10:48:57.475+0200 INFO [beat] instance/beat.go:976 Process info {“system_info”: {“process”: {“capabilities”: {“inheritable”:null,“permitted”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“effective”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“bounding”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”],“ambient”:null}, “cwd”: “/”, “exe”: “/usr/share/filebeat/bin/filebeat”, “name”: “filebeat”, “pid”: 1763, “ppid”: 763, “seccomp”: {“mode”:“filter”,“no_new_privs”:true}, “start_time”: “2021-08-17T10:48:56.980+0200”}}}
2021-08-17T10:48:57.475+0200 INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.8.14
2021-08-17T10:48:57.476+0200 INFO [publisher] pipeline/module.go:110 Beat name: kvit-graylog
2021-08-17T10:48:57.476+0200 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2021-08-17T10:48:57.476+0200 INFO instance/beat.go:402 filebeat start running.
2021-08-17T10:48:57.476+0200 INFO registrar/registrar.go:134 Loading registrar data from /var/lib/graylog-sidecar/collectors/filebeat/data/registry
2021-08-17T10:48:57.477+0200 INFO [monitoring] log/log.go:152 Total non-zero metrics {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:0,“time”:{“ms”:5}},“total”:{“ticks”:10,“time”:{“ms”:16},“value”:10},“user”:{“ticks”:10,“time”:{“ms”:11}}},“handles”:{“limit”:{“hard”:524288,“soft”:1024},“open”:6},“info”:{“ephemeral_id”:“6e659230-8846-4578-b5f7-bbfdebffe36f”,“uptime”:{“ms”:9}},“memstats”:{“gc_next”:4194304,“memory_alloc”:2355552,“memory_total”:3892432,“rss”:22208512}},“filebeat”:{“harvester”:{“open_files”:0,“running”:0}},“libbeat”:{“config”:{“module”:{“running”:0}},“output”:{“type”:“logstash”},“pipeline”:{“clients”:0,“events”:{“active”:0}}},“registrar”:{“states”:{“current”:0}},“system”:{“cpu”:{“cores”:8},“load”:{“1”:0.75,“15”:0.06,“5”:0.19,“norm”:{“1”:0.0938,“15”:0.0075,“5”:0.0238}}}}}}
2021-08-17T10:48:57.478+0200 INFO [monitoring] log/log.go:153 Uptime: 10.323724ms
2021-08-17T10:48:57.478+0200 INFO [monitoring] log/log.go:130 Stopping metrics logging.
2021-08-17T10:48:57.478+0200 INFO instance/beat.go:412 filebeat stopped.
2021-08-17T10:48:57.478+0200 ERROR instance/beat.go:906 Exiting: Could not start registrar: Error loading state: Error decoding states: EOF

Is there someone who can help me and had any idea?

[gsmith]

Hello && Welcome

I might be able to help.

@duese

registrar/registrar.go:134 Loading registrar data from /var/lib/graylog-sidecar/collectors/filebeat/data/registry

I’m not sure what’s really going on in your environment. Never had MongoDb crash before. So, I’m unsure exactly what happen with yours.
MongoDb keeps all metadata (i.e., Sidecar config) for Graylog, so when you replaced your MongoDb database you removed all you new configurations for that. My best suggestion right now is try to copy any configurations you have and reinstall your FileBeat.
Hope that helps

[duese]

Thanks for your answer.

After an VM Restore was my Graylog corrupted and the Mongodb Service didn’t start, so that I delete /var/lib/mongodb/* and after that I restore Data vom February. Now I can See the data in Graylog, but filebeat doesn’t work.
How do I reinstall Filebeat?
I mean, that was automatically installed for me with Graylog.
I’ve tried this and still have the same bug.
Repositories for APT and YUM | Filebeat Reference [7.14] | Elastic

If I remove the assignment OBS, Sophos, etc., the falling disappears and the sidecars overview shows runniung. However, this is not a solution.

[gsmith]

Hello,

@duese

so that I delete /var/lib/mongodb/* and after that

Just because something stop working, I would not delete it. My first choice would be the Forum before deleteing it. There are a lot of really good people here thats willing to help you out on there spare time.

@duese

How do I reinstall Filebeat?

Since you’re using Graylog Sidecar I would at this point re-install.
You can look here on how to do that.
https://docs.graylog.org/en/4.1/pages/sidecar.html#graylog-sidecar

Basically, you install it, connected it to Graylog Server and configure your shipper via Web UI. But I highly recommend you read that documentation first. Since you deleted MongDb you may run into other problems.
Hope that helps

[duese]

@gsmith

Just because something stop working, I would not delete it. My first choice would be the Forum before deleteing it. There are a lot of really good people here thats willing to help you out on there spare time.

I had read in the Forum, that someone delete the mongodb directory.
Now I restore the Virtual Machine in a Second Machine, before I delete the monodb directory.
When the VM backup was created everything was still working. But you see the mongod.service doesn’t work.

root@kvit-graylog:~# systemctl status mongod.service
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: core-dump) since Thu 2021-08-19 08:44:16 CEST; 3min 3s ago
Docs: https://docs.mongodb.org/manual
Process: 1519 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=dumped, signal=ABRT)
Main PID: 1519 (code=dumped, signal=ABRT)

Aug 19 08:44:16 kvit-graylog systemd[1]: Started MongoDB Database Server.
Aug 19 08:44:16 kvit-graylog systemd[1]: mongod.service: Main process exited, code=dumped, status=6/ABRT
Aug 19 08:44:16 kvit-graylog systemd[1]: mongod.service: Failed with result ‘core-dump’.

[gsmith]

Hello,

Sorry, I might have misunderstood your issue. I thought it was your log shipper FileBeat, but your real issue is still MongoDb. Is this correct?
This is unfortunate because Active: failed (Result: core-dump) This error is possible consequence of crashed or corrupted database which can be caused by terminating the database in a way which shouldn’t be used.

There are a couple things you can do first:
Check MongoDB permissions

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock

Then

sudo systemctl restart mongod
sudo systemctl status mongod

Now check your MongoDb logs for any errors/warnings.

If that doesnt help, execute a repair MongoDb.

sudo mongod --repair --dbpath /path/to/mongodb

Before you purge MongoDb, it would be wise to post your MongoDb logs and/or anything that pertains to this issue.

[duese]

Sorry for the misunderstanding, unfortunately my English is not the best.

@gsmith

There are a couple things you can do first:
Check MongoDB permissions

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock

Then

sudo systemctl restart mongod
sudo systemctl status mongod

Now check your MongoDb logs for any errors/warnings.

Unfortunately that was not enough and I had tried the repair before. However, if I do the customization again after the repair, I come back to the Graylog interface with all the data. Unfortunately, my sidecar doesn’t work.
Now I have this:

and this:
![Screenshot 2021-08-18 at 10-23-35 Graylog - Collectors Administration|6

@gsmith

There are a couple things you can do first:
Check MongoDB permissions

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock

Then

sudo systemctl restart mongod
sudo systemctl status mongod

Now check your MongoDb logs for any errors/warnings.

90x150](upload://2UPE03TTJNnr4obA0oM2Cszajw2.png)
For the current progress you have definitely made me a lot happier.
Somehow I hadn’t seen any light at the end of the tunnel before.

The Sidecar Service looks now:
root@kvit-graylog:~# systemctl status graylog-sidecar.service
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-08-20 10:37:55 CEST; 3min 39s ago
Main PID: 764 (graylog-sidecar)
Tasks: 23 (limit: 19106)
Memory: 38.9M
CGroup: /system.slice/graylog-sidecar.service
└─764 /usr/bin/graylog-sidecar

Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[filebeat] Unable to start collector after 3 tries, giving up!”
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[MascSE-Produktion] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding>
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=”[MascZS-Produktion] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding>
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[filebeat] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding states: >
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=”[Jenkins] Unable to start collector after 3 tries, giving up!"
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[ondis] Unable to start collector after 3 tries, giving up!”
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[Jenkins] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding states: E>
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=”[ondis] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding states: EOF>
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg=“[Sophos] Unable to start collector after 3 tries, giving up!”
Aug 20 10:38:49 kvit-graylog graylog-sidecar[764]: time=“2021-08-20T10:38:49+02:00” level=error msg="[Sophos] Collector output: Exiting: Could not start registrar: Error loading state: Error decoding states: EO>

[gsmith]

Hello,

@duese

For the current progress you have definitely made me a lot happier.
Somehow I hadn’t seen any light at the end of the tunnel before.

Your Welcome

Just to sum it up. MongDb is running but your problem is Graylog-Sidecar now?
If this is correct, double check your sidecar configuration file? I think this is YAML file so, make sure indents in the file are correct.

Maybe another option is to install the newest version of GL Sidecar. Looks like your running an older version. See if that works.
https://docs.graylog.org/en/4.1/pages/sidecar.html#installation
If you can post your sidecar config here.

[duese]

Exactly, that’s right. At least the created Collectors and the Log Collectors under sidecars do not work. My sidecar.yml looks like this:

# The URL to the Graylog server API.
server_url: "http://172.22.23.239:9000/api/"

# The API token to use to authenticate against the Graylog server API.
# This field is mandatory
server_api_token: "v4mbal318qbrirhueb82q3rsq2q8u05i39k4rrgt8bjkfccs9c0"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:/etc/graylog/sidecar/node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
#node_id: "file:/etc/graylog/sidecar/node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
node_name: "graylog"

# The update interval in seconds. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
#update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
#tls_skip_verify: false

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
#send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []

# Directory where the sidecar stores internal data.
#cache_path: "/var/cache/graylog-sidecar"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "/var/log/graylog-sidecar"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_whitelist:
#       - "/usr/bin/filebeat"
#       - "/opt/collectors/*"
#
# Example disable whitelisting:
#     collector_binaries_whitelist: []
#
# Default:
# collector_binaries_whitelist:
#  - "/usr/bin/filebeat"
#  - "/usr/bin/packetbeat"
#  - "/usr/bin/metricbeat"
#  - "/usr/bin/heartbeat"
#  - "/usr/bin/auditbeat"
#  - "/usr/bin/journalbeat"
#  - "/usr/share/filebeat/bin/filebeat"
#  - "/usr/share/packetbeat/bin/packetbeat"
#  - "/usr/share/metricbeat/bin/metricbeat"
#  - "/usr/share/heartbeat/bin/heartbeat"
#  - "/usr/share/auditbeat/bin/auditbeat"
#  - "/usr/share/journalbeat/bin/journalbeat"
#  - "/usr/bin/nxlog"
#  - "/opt/nxlog/bin/nxlog"

[gsmith]

Hello,

Couple questions.

1. Have you tried creating new API Tokens for those sidecars?
2. Have you tried installing a newer version of the sidecar?

Releases · Graylog2/collector-sidecar

Tested the Graylog Sidecar on a windows server in my lab.
Download the exe file and configure it through the installer. What I found was the server_api_token and with my environment I had to set this tls_skip_verify: true.

Steps taken:

1. Download new Sidecar on Server version 1.1.0
2. Configured a new Server API token for sidecar.
3. Started Graylog-Sidecar Application on server.

Results.

Here is a couple suggestion on your failing Sidecars.

• Make a new API and configured it in your sidecar (restart service).
• Check your filewall/s, either from your remote device or on your Graylog server.

EDIT: I was able to create a failing collector.
How I fixed it was in the Sidecar Overview then I clicked on the name.

This showed the Sidecar status. Should look like this.

Navigated to Log collectors then clicked on “edit” on the left side.
Change this.

Into this.

Now I have this.

Saved file and restart Graylog sidecar service. It might not be the same as your error or type of collector but it should show some type of error/warning that can help you out.

Hope that helps

[duese]

I have now done the sidecar update. The API key for the sidecar user is new and is also in the sidecar.yml accordingly.
The interface looks like this:

and this:

And on details this:

[gsmith]

Hello,

To be honest your not showing enough information here for me to help troubleshoot your issue.

• What does your Sidecar logs show?
• What configuration did you make?
• What did you configure and the steps you executed.
• Can you show your configuration?
• How did you install sidecar? Perhaps showing the steps.
• Was there any extra configuration you made during the installation?
• What does you Web UI look like under System/Sidecars/Configuration

Example:

Not starting a service can be multiple problems. Just showning unable to start collector does not give us good information to go on for helping out. What needs to happen is for you to show us all available information you see or can show us.

Since this is a collector, can you show the configuration for that? since the sidecar was last seen a few minutes ago I assume its in the configuration for the collector.
EDIT: Just a thought, what user did you use to create the API ? I used the following.

Also is it possible to execute just one collector instead of 6?
Do you have enough space on the server where the sidecar resides?

Hope that helps

[duese]

@gsmith

• What does your Sidecar logs show?
• What configuration did you make?
• What did you configure and the steps you executed.
• Can you show your configuration?
• How did you install sidecar? Perhaps showing the steps.
• Was there any extra configuration you made during the installation?
• What does you Web UI look like under System/Sidecars/Configuration

My Sidecar.log show:

time="2021-03-03T09:09:31Z" level=info msg="Starting signal distributor"
time="2021-03-03T09:09:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:09:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:10:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:11:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:12:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:13:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:14:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:15:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:16:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:17:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:41Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:18:51Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:19:01Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:19:11Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:19:21Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"
time="2021-03-03T09:19:31Z" level=error msg="[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized"

My configuration follows on the following pictures .
Sidecar I installed with:

wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.de
sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
sudo apt-get update && sudo apt-get install graylog-sidecar

If I only excute the MascSE, I have the same Problem if I start all.
For my API I use the Sidecar System User and his Token is in the sidecar.yml.
And my filesystem looks like this:

root@kvit-graylog:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               7.8G     0  7.8G   0% /dev
tmpfs                              1.6G  9.1M  1.6G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   42G   14G  75% /
tmpfs                              7.9G     0  7.9G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda2                          976M  203M  707M  23% /boot
/dev/loop1                          68M   68M     0 100% /snap/lxd/20326
/dev/loop0                          56M   56M     0 100% /snap/core18/2074
/dev/loop2                          33M   33M     0 100% /snap/snapd/12704
/dev/loop4                          71M   71M     0 100% /snap/lxd/21029
/dev/loop5                          56M   56M     0 100% /snap/core18/2128
/dev/loop6                          33M   33M     0 100% /snap/snapd/12883
tmpfs                              1.6G     0  1.6G   0% /run/user/2189

I like to send more log files, just wondering where I can find them.
For example, I still have the log file here that I found under
/var/lib/graylog-sidecar/collectors/filebeat/log/filebeat

:33.164+0200    INFO    [beat]  instance/beat.go:976    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 1910, "ppid": 754, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-08-24T12:30:32.560+0200"}}}
2021-08-24T12:30:33.165+0200    INFO    instance/beat.go:280    Setup Beat: filebeat; Version: 6.8.14
2021-08-24T12:30:33.165+0200    INFO    [publisher]     pipeline/module.go:110  Beat name: kvit-graylog
2021-08-24T12:30:33.165+0200    ERROR   fileset/modules.go:118  Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2021-08-24T12:30:33.165+0200    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-08-24T12:30:33.166+0200    INFO    instance/beat.go:402    filebeat start running.
2021-08-24T12:30:33.166+0200    INFO    registrar/registrar.go:134      Loading registrar data from /var/lib/graylog-sidecar/collectors/filebeat/data/registry
2021-08-24T12:30:33.167+0200    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":7}},"total":{"ticks":0,"time":{"ms":14},"value":0},"user":{"ticks":0,"time":{"ms":7}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":6},"info":{"ephemeral_id":"4172dd9b-6293-47ff-8b9f-c3c4fa162df2","uptime":{"ms":14}},"memstats":{"gc_next":4194304,"memory_alloc":1990136,"memory_total":3863336,"rss":21901312}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":2.74,"15":0.35,"5":0.97,"norm":{"1":0.3425,"15":0.0438,"5":0.1213}}}}}}
2021-08-24T12:30:33.167+0200    INFO    [monitoring]    log/log.go:153  Uptime: 15.290356ms
2021-08-24T12:30:33.167+0200    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2021-08-24T12:30:33.167+0200    INFO    instance/beat.go:412    filebeat stopped.

And after I’ve created the following symlink,

ln -s /usr/share/filebeat/module/ /usr/share/filebeat/bin/

it looks like this:

2021-08-25T09:53:05.807+0200    INFO    instance/beat.go:611    Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/var/lib/graylog-sidecar/collectors/filebeat/data] Logs path: [/var/lib/graylog-sidecar/collectors/filebeat/log]
2021-08-25T09:53:05.807+0200    INFO    instance/beat.go:618    Beat UUID: 1e30b2fd-023a-4a04-9164-26b405e83244
2021-08-25T09:53:05.807+0200    INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
2021-08-25T09:53:05.807+0200    INFO    [beat]  instance/beat.go:931    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat/bin", "data": "/var/lib/graylog-sidecar/collectors/filebeat/data", "home": "/usr/share/filebeat/bin", "logs": "/var/lib/graylog-sidecar/collectors/filebeat/log"}, "type": "filebeat", "uuid": "1e30b2fd-023a-4a04-9164-26b405e83244"}}}
2021-08-25T09:53:05.807+0200    INFO    [beat]  instance/beat.go:940    Build info      {"system_info": {"build": {"commit": "5cd281153df1eb5e95a4a31994a7846d2c9493e8", "libbeat": "6.8.14", "time": "2021-02-02T18:46:23.000Z", "version": "6.8.14"}}}
2021-08-25T09:53:05.807+0200    INFO    [beat]  instance/beat.go:943    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.10.8"}}}
2021-08-25T09:53:05.808+0200    INFO    [beat]  instance/beat.go:947    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-08-25T09:52:04+02:00","containerized":false,"name":"kvit-graylog","ip":["127.0.0.1/8","::1/128","172.22.23.239/24","fe80::a846:96ff:fe9d:d870/64"],"kernel_version":"5.4.0-81-generic","mac":["aa:46:96:9d:d8:70"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.2 LTS (Focal Fossa)","major":20,"minor":4,"patch":2,"codename":"focal"},"timezone":"CEST","timezone_offset_sec":7200,"id":"83cd4f68db0a403f8c24289cd8790e0b"}}}
2021-08-25T09:53:05.808+0200    INFO    [beat]  instance/beat.go:976    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 1915, "ppid": 757, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-08-25T09:53:05.240+0200"}}}
2021-08-25T09:53:05.808+0200    INFO    instance/beat.go:280    Setup Beat: filebeat; Version: 6.8.14
2021-08-25T09:53:05.809+0200    INFO    [publisher]     pipeline/module.go:110  Beat name: kvit-graylog
2021-08-25T09:53:05.809+0200    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
2021-08-25T09:53:05.809+0200    INFO    instance/beat.go:402    filebeat start running.
2021-08-25T09:53:05.809+0200    INFO    registrar/registrar.go:134      Loading registrar data from /var/lib/graylog-sidecar/collectors/filebeat/data/registry
2021-08-25T09:53:05.810+0200    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0},"total":{"ticks":10,"time":{"ms":15},"value":10},"user":{"ticks":10,"time":{"ms":15}}},"handles":{"limit":{"hard":524288,"soft":1024},"open":6},"info":{"ephemeral_id":"fd183e87-5eba-4e30-bc11-d0d51c6feb1f","uptime":{"ms":13}},"memstats":{"gc_next":4194304,"memory_alloc":2415632,"memory_total":3897680,"rss":22683648}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.7,"15":0.08,"5":0.22,"norm":{"1":0.0875,"15":0.01,"5":0.0275}}}}}}
2021-08-25T09:53:05.810+0200    INFO    [monitoring]    log/log.go:153  Uptime: 13.941085ms
2021-08-25T09:53:05.810+0200    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2021-08-25T09:53:05.810+0200    INFO    instance/beat.go:412    filebeat stopped.
2021-08-25T09:53:05.811+0200    ERROR   instance/beat.go:906    Exiting: Could not start registrar: Error loading state: Error decoding states: EOF

[gsmith]

Hello,

Thank you for all the information and what I did was to test this out in my lab on a Ubuntu 20. The remote server is called Keycloak. Since this a new installation, I installed Graylog Sidecar (1.1.0) made configurations to the sidecar YAML file and started it the service. I made sure that the sidecar was working correctly before I proceeded to configure FileBeat.

Below are the steps taken and any configuration I made in testing. I did show all errors that happened and how I corrected it. I followed the Graylog documentation for install Graylog-sidecar.

Installation Steps:

$ wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
$ sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
$ sudo apt-get update && sudo apt-get install graylog-sidecar

Configured Graylog-sidecar

$ vi /etc/graylog/sidecar/sidecar.yml
root@keycloak:~# grep -v "^#\|^$" /etc/graylog/sidecar/sidecar.yml
server_url: "http://8.8.8.8:9000/api/"
server_api_token: "115oppe0ghiftkq9o20bjcm99po4n6s0e7cge769gfuqul3iedml"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "keycloak"
tls_skip_verify: true
send_status: true
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10

Create Graylog-Sidecar service
$ sudo graylog-sidecar -service install

Start Graylog-sidecar service
$ sudo systemctl start graylog-sidecar

Graylog Sidecar Error Messages:

Had to change the configuration from http to https because that is what my lab graylog server uses.

server_url: https://8.8.8.8:9000/api/

FileBeat configuration

I copied the Linux FileBeat template and called it Keycloak, this way it matches the name of the server. Its only for my simplicity for testing this out and find any errors/warning that may occur.

Administration Configurations

As you can see I have my Keycloak template

Back to Administration section, checked the box next to Keycloak and applied configuration.

It seems I missed a step or two in the documentation. Like installing FileBeat .

I needed to install FileBeat. Here are the steps taken. The link for this is in the Graylog-Sidecar documentation. I just over looked it.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update && sudo apt-get install filebeat
sudo systemctl enable filebeat
sudo systemctl start filebeat

Second Issue:

I still did not get messages through so I checked the Filebeat Log file here.

vi /var/lib/graylog-sidecar/collectors/filebeat/log/filebeat

ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: write tcp 8.8.4.4:43622->8.8.8.8:5044: write: connection reset by peer

This was caused by the WRONG Input. I was using Syslog UDP INPUT port 5044 and I should have been using Beat’s INPUT port 5044. As shown below.

Results
Check to insure my Graylog-Sidecar is running without errors.

That was it and the reason I did this was to show any/all configuration made and how I resolved my issue.

Conclusion:
Make sure Graylog-sidecar is running first without errors.
You may need to start with one collector and lets say you use FileBeat. You don’t need to deleted anything just remove all the collectors from that Sidecar and just leave one. This is shown how to do this below (red box).

I believe you have a lot going on so taking baby steps will probably help you resolve the issues within your environment.

To be honest, deleting MongoDb was probably a bad idea but lessons learned.

Things I seen that may not be right.
I did not have to make a symlink at all. I did not see this in the documentation that I was following as shown below.
https://docs.graylog.org/en/4.0/pages/sidecar.html

@duese

And after I’ve created the following symlink,

ln -s /usr/share/filebeat/module/ /usr/share/filebeat/bin/

I realize this worked before but obviously it does not any more so to troubleshoot this issue you may need to backup and do one thing at a time then to install all these collectors.

Check list:

Check your Graylog-Sidecar log files
/var/log/graylog-sidecar

Check the status of your Graylog-Sidecar
Sudo systemctl status graylog-sidecar

Check you filebeat service
Sudo systemctl status filebeat

Check your Filebeat log file
vi /var/lib/graylog-sidecar/collectors/filebeat/log/filebeat

Hope that helps

[duese]

I am now one step further, unfortunately your execution did not help me this time. The installation of Filebeat is not necessary in my case, as the function is already in the sidecar installation.
I also think that deleting MongoDB was a good idea. I completely restored the deleted one by doing a restore from my VM backup where I had the last status. Then I brought the defective MongoDB back to life with a repair command. What I hadn’t noticed when I tried it for the first time was that the owner rights still had to be adjusted.
But now to my last step:
I found the solution to my problem here:

https://discuss.elastic.co/t/exiting-could-not-start-registrar-error-loading-state-error-decoding-states-eof/74430

However, I have the register file in:

/var/lib/graylog-sidecar/collectors/filebeat/data/registry

After I deleted it, everything is green and the filebeats are running. Now I’ll see whether all the data is processed and entered my dashboards. I may not be able to report on this until tomorrow, however, as the data is delivered once a day.
I hope the rest will go now. In any case, thank you very much for your help. In many share my salvation.

[gsmith]

Hello,

@duese

The installation of Filebeat is not necessary in my case, as the function is already in the sidecar installation.

I thought it should have been installed also. I used your installation that you stated for graylog-sidecar and FileBeat wasn’t install. Kind of weird because my other installations didn’t have this problem. I may have missed something.

@duese

wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.de
sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
sudo apt-get update && sudo apt-get install graylog-sidecar

@duese

What I hadn’t noticed when I tried it for the first time was that the owner rights still had to be adjusted.

Oh I see now, good catch.

When your issue is all resolve I’m curious to see if your solution worked. I haven’t had this problem before but incase someone does I can revert back to this post for answers.
keep us apprised and good luck.

[duese]

Now everything seems to be going very well again. Thanks again for the great support.

[gsmith]

That’s great, I also learned a something new.