Unit graylog-sidecar.service not found (Ubuntu Linux 20.04)

Description of your problem

Sidecar service is not created on client Linux machines.

Description of steps you’ve taken to attempt to solve the issue

https://docs.graylog.org/docs/sidecar
I install sidecar on my Ubuntu linux server 20.04:

$ wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
$ sudo dpkg -i graylog-sidecar-repository_1-2_all.deb
$ sudo apt-get update && sudo apt-get install graylog-sidecar

then i adapt the config file
/etc/graylog/sidecar/sidecar.yml
and then i find NO way to install the service in the documentation, only “andactivate the Sidecar as a system service”
but when i try to start it using

systemctl start graylog-sidecar

it gives me this error:
Failed to start graylog-sidecar.service: Unit graylog-sidecar.service not found.
This means that the .deb file does NOT install the service. WHY?

Environmental information

graylog-sidecar (1.1.0-1) on the client/target.
Client/target:
5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Server:
Version: 4.2.0+5adccc3, codename Noir
JVM: PID 46957, Private Build 1.8.0_292 on Linux 4.15.0-159-generic

Operating system information

Graylog server is on VM appliance downloaded from graylog site (september 2021).
the target server is Ubuntu 20.04
5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  • Ubuntu

Package versions

graylog-sidecar (1.1.0-1)

Is this normal that the DEB file does NOT install the sidecar service?
If yes, then your documentation is incomplete and cannot be used as is.
How to make it work? How to install or what is the correct procedure for Ubuntu Linux 20.04 targets?

I agree it’s not clear in the instructions but you need to use graylog-sidecar to install the service after the dpkg:

dude@elast-server:~$ sudo systemctl start graylog-sidecar
Failed to start graylog-sidecar.service: Unit graylog-sidecar.service not found.

dude@elast-server:~$ sudo graylog-sidecar -service install

dude@elast-server:~$ sudo systemctl enable graylog-sidecar
dude@elast-server:~$ sudo systemctl start graylog-sidecar
dude@elast-server:~$ sudo systemctl status graylog-sidecar
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
     Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-10-19 19:23:52 UTC; 9s ago
   Main PID: 58456 (graylog-sidecar)
      Tasks: 6 (limit: 4593)
     Memory: 1.9M
     CGroup: /system.slice/graylog-sidecar.service
             └─58456 /usr/bin/graylog-sidecar

Oct 19 19:23:52 elast-server systemd[1]: Started Wrapper service for Graylog controlled collector.
Oct 19 19:23:52 elast-server graylog-sidecar[58456]: time="2021-10-19T19:23:52Z" level=info msg="node-id file doesn't exist, generating a new one"
Oct 19 19:23:52 elast-server graylog-sidecar[58456]: time="2021-10-19T19:23:52Z" level=info msg="Using node-id: dae40a62-7777-4aed-b925-4a1db177726a"
Oct 19 19:23:52 elast-server graylog-sidecar[58456]: time="2021-10-19T19:23:52Z" level=info msg="No node name was configured, falling back to hostname"
Oct 19 19:23:52 elast-server graylog-sidecar[58456]: time="2021-10-19T19:23:52Z" level=info msg="Starting signal distributor"
2 Likes

Thanks, it worked. The service is created.
I start it, no messages.
Then i check the status and it “failed with exit code”:

root@hostname:/home/admin# systemctl status graylog-sidecar
● graylog-sidecar.service - Wrapper service for Graylog controlled collector
     Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2021-10-20 17:01:58 UTC; 5s ago
    Process: 3291 ExecStart=/usr/bin/graylog-sidecar (code=exited, status=1/FAILURE)
   Main PID: 3291 (code=exited, status=1/FAILURE)

Oct 20 17:01:58 hostname systemd[1]: graylog-sidecar.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 17:01:58 hostname systemd[1]: graylog-sidecar.service: Failed with result 'exit-code'.

/var/log/graylog-sidecar is empty.
Is there a topic on this failure, or maybe you can direct me to a solution, please?
Thanks

What are the results from:

journalctl -xefu graylog-sidecar

My guess is there is something else going on with your system other than the Graylog Sidecar install

OK, here is some output (i did not include all the repeated logs):

-- Logs begin at Tue 2021-02-02 15:13:40 UTC. --
Oct 20 17:01:58 hostname systemd[1]: Started Wrapper service for Graylog controlled collector.
-- Subject: A start job for unit graylog-sidecar.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit graylog-sidecar.service has finished successfully.
--
-- The job identifier is 868.
Oct 20 17:01:58 hostname graylog-sidecar[3291]: [ConfigFile] YAML config parsing failed on /etc/graylog/sidecar/sidecar.yml: yaml: line 74: did not find expected key. Exiting.
Oct 20 17:01:58 hostname systemd[1]: graylog-sidecar.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit graylog-sidecar.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Oct 20 17:01:58 hostname systemd[1]: graylog-sidecar.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit graylog-sidecar.service has entered the 'failed' state with result 'exit-code'.
Oct 20 17:03:58 hostname systemd[1]: graylog-sidecar.service: Scheduled restart job, restart counter is at 1.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit graylog-sidecar.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Oct 20 17:03:58 hostname systemd[1]: Stopped Wrapper service for Graylog controlled collector.
-- Subject: A stop job for unit graylog-sidecar.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit graylog-sidecar.service has finished.
--
-- The job identifier is 957 and the job result is done.
Oct 20 17:03:58 hostname systemd[1]: Started Wrapper service for Graylog controlled collector.
-- Subject: A start job for unit graylog-sidecar.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit graylog-sidecar.service has finished successfully.
--
-- The job identifier is 957.
Oct 20 17:03:59 hostname graylog-sidecar[3326]: [ConfigFile] YAML config parsing failed on /etc/graylog/sidecar/sidecar.yml: yaml: line 74: did not find expected key. Exiting.
Oct 20 17:03:59 hostname systemd[1]: graylog-sidecar.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit graylog-sidecar.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Oct 20 17:03:59 hostname systemd[1]: graylog-sidecar.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit graylog-sidecar.service has entered the 'failed' state with result 'exit-code'.
Oct 20 17:05:59 hostname systemd[1]: graylog-sidecar.service: Scheduled restart job, restart counter is at 2.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

Line 74 of sidecar.yml is:

#     collector_binaries_whitelist: []

“An empty list disables the white list feature.” - does this mean that there is no white list and everything is permitted? Then why error?

There was some other part of your YAML that is missing, that line is likely the last line it was trying to find the missing key on. Here is one of mine condensed without comments for comparison to what you might be missing. Feel free to post yours (anonymized of course) if you can’t find it.

$ cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"

server_url: "http://honkytonkGlog:9000/api/"
server_api_token: "SuperSecretAPITokenPlacedHere"
node_id: "file:/etc/graylog/sidecar/node-id"
update_interval: 10
tls_skip_verify: true
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"
collector_binaries_accesslist:
  - "/usr/share/filebeat/bin/filebeat"

Hello,
OK, i have tried your config (with my url and token). I have the same result.
line 74: did not find expected key. Exiting.
I tried
collector_binaries_accesslist:

  • “/usr/share/filebeat/bin/filebeat”
    I tried line 74:
    collector_binaries_whitelist:
    Always the same error in the same line 74:
    collector_binaries_whitelist:

Also, i do not find any directory/file listed at the end of the yml file.
/usr/share/filebeat/ does not exist, nor other variants.
I cannot find the binary in all these paths.

find / -name filebeat
gives me nothing

Sidecar on Linux is a shell, wrapped around filebeat or nxlog. You need to install one of those so you can configure them in Graylog. In the Sidecar docs, scroll down to Install Collectors and follow that. My guess, based on what you have said and without having looked at your sidecar config, is that it is looking for filebeat/nxlog.

When you do post code, please use the Forum Tools like </> to make the code easier to read. :slight_smile:

Ah, the filebeats are not installed automatically by sidecar!
OK, i got it installed and started, it appeared in /usr/bin/filebeat

root@hostname:/etc/graylog/sidecar# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2021-10-22 10:13:12 UTC; 4h 39min ago

I edited the /etc/graylog/sidecar/sidecar.yml:

cat /etc/graylog/sidecar/sidecar.yml | egrep -v "^\s*(#|$)"

server_url: "http://192.1.1.22:9000/api/"
server_api_token: "~token~"
node_id: "file:/etc/graylog/sidecar/node-id"
update_interval: 10
send_status: true
     list_log_files:
       - "/var/log/apache2"
       - "/var/log"
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"
collector_binaries_whitelist:
       - "/usr/bin/filebeat"
       - "/opt/collectors/*"

Still not starting:

-- The job identifier is 136042.
Oct 22 14:56:18 hostname graylog-sidecar[53920]: [ConfigFile] YAML config parsing failed on /etc/graylog/sidecar/sidecar.yml: yaml: line 74: did not find expected key. Exiting.
Oct 22 14:56:18 hostname systemd[1]: graylog-sidecar.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit graylog-sidecar.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Oct 22 14:56:18 hostname systemd[1]: graylog-sidecar.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit graylog-sidecar.service has entered the 'failed' state with result 'exit-code'.

Because i modified the yml file, the line 74 is now different:

collector_binaries_whitelist:
       - "/usr/bin/filebeat"     #  <-- Line 74
       - "/opt/collectors/*"

/var/log/graylog-sidecar is empty.
What the # is going on?

I am pretty sure Sidecar handles starting the filebeat service for you, but that is not your problem. Two things:

  1. YAML files are VERY picky about indentation. Make 100% sure you have that right.
  2. the log files you want to watch are not defined in the sidecar YAML file, they are set up in your sidecar configurations in Graylog UI (i.e. remove the list_log_files section)

The yaml file you are working on is really just intended to get the sidecar connected to the Graylog server and know where local binaries are. once it is connected properly (URL and Token plus some misc housekeeping settings) you go to the Graylog UI and build out a log collector/configuration that you can then apply to the sidecar under configuration. Here is a sample linux filebeats config.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- type: log
  enabled: true
  exclude_lines: ['bull_whip']
  paths:
    - /var/log/apache2/*
    - /var/log/*.log
  ignore_older: 72h
  tags:
    - linux
#############   field marker for shunting with stream rules
  fields:
    prod_env: false

output.logstash:
   hosts: 
   - ${user.BeatsInput}

path:
  data: /var/cache/graylog-sidecar/filebeat/data
  logs: /var/log/graylog-sidecar

OK, i have commented out the log files and collector_binaries_whitelist, uncommented and moved around

collector_binaries_whitelist: []

and it started.
In the status message it also says that:

level=warning msg="`collector_binaries_whitelist` is deprecated. Migrate your configuration to `collector_binaries_accesslist`."

This is recent change. So, to restrict the binaries, should i rename whitelist into accesslist?

you could give it a try! :stuck_out_tongue:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.