Failed to report collector status to server: Put "xxx"; dial tcp x.x.x.x:5044:i/o timeout

Graylog Central (peer support)
,
1

So I have been trying to get a filebeat sidecar working for linux, but to no avail.
I have already configured a Beats Input.

In the logs on the host machine I get this error: Failed to report collector status to server: Put "xxx"; dial tcp x.x.x.x:5044:i/o timeout.

Here’s my sidecar.yml file:

# The URL to the Graylog server API.
server_url: "https://x.x.x.x:5044"

# The API token to use to authenticate against the Graylog server API.
# This field is mandatory
server_api_token: "xxxxxxxxxxxxxxxxxxxxxxx"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:/etc/graylog/sidecar/node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
#node_id: "file:/etc/graylog/sidecar/node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
#node_name: ""

# The update interval in seconds. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
#update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
#send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []

# Directory where the sidecar stores internal data.
#cache_path: "/var/cache/graylog-sidecar"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "/var/log/graylog-sidecar"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
"/etc/graylog/sidecar/sidecar.yml"
# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_whitelist:
#       - "/usr/bin/filebeat"
#       - "/opt/collectors/*"
#
# Example disable whitelisting:
#     collector_binaries_whitelist: []
#
# Default:
# collector_binaries_whitelist:
#  - "/usr/bin/filebeat"
#  - "/usr/bin/packetbeat"
#  - "/usr/bin/metricbeat"
#  - "/usr/bin/heartbeat"
#  - "/usr/bin/auditbeat"
#  - "/usr/bin/journalbeat"
#  - "/usr/share/filebeat/bin/filebeat"
#  - "/usr/share/packetbeat/bin/packetbeat"
#  - "/usr/share/metricbeat/bin/metricbeat"
#  - "/usr/share/heartbeat/bin/heartbeat"
#  - "/usr/share/auditbeat/bin/auditbeat"
#  - "/usr/share/journalbeat/bin/journalbeat"
#  - "/usr/bin/nxlog"
#  - "/opt/nxlog/bin/nxlog"

And here’s my collector configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/*.log
  type: log
output.logstash:
   hosts: ["x.x.x.x:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

Regarding the first file, I have already tried the different approach of simply putting https://x.x.x.x:9000/api but it also doesn’t work. The sidecar is freshly installed so it’s running the newest version, graylog server’s version is 4.3.7

2

Hello,

The sidecar configuration is incorrect

Should look something like this.

root@ansible:/etc/graylog/sidecar# cat sidecar.yml | egrep -v "^\s*(#|$)"
server_url: "https://graylog_server_ip_address:9000/api/" 
server_api_token: "3jje4aq12jk6ot3vfr52ifscgoh0qmgqhhl2bn0m9hpvju4puh5"
node_id: "file:/etc/graylog/sidecar/node-id"
node_name: "testing"
tls_skip_verify: true
send_status: true
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10

Ensure the firewall and/or SELinux /apparmor is not interfering.
Port 5044 is for log shipper/s Graylog-sidecar is a different entity . Its a wrapper for the log shippers.

1 Like
3

@gsmith After doing a bit of digging, it turned out that SELinux was preventing some stuff:

 SELinux is preventing /usr/sbin/zabbix_agentd from getattr access on the file /proc/<pid>/cmdline. For complete SELinux messages run: sealert -l 930fbaf2-6a53-4e7d-b940-fcb17f1580d If you believe that zabbix_agentd should be allowed getattr access on the cmdline file by default.
                          Then you should report this as a bug.
                          You can generate a local policy module to allow this access.
                          Do
                          allow this access for now by executing:
                          # ausearch -c 'zabbix_agentd' --raw | audit2allow -M my-zabbixagentd
                          # semodule -X 300 -i my-zabbixagentd.pp

It wanted me to generate and enforce a new policy and so I did:

# ausearch -c 'zabbix_agentd' --raw | audit2allow -M my-zabbixagentd
# semodule -X 300 -i my-zabbixagentd.pp

However Zabbix doesn’t have to do anything with it, right? And it doesn’t, since after enforcing the new policy and restarting the sidecar service (server url corrected as you suggested), the error still persists.

Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert 13a3ca07-23d2-4868-bce0-c187f3e51408, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert e1008693-a3b6-4489-ae94-6ec2ec7a039f, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert f542a1d8-fd59-412f-8624-acef48c10231, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert 9b4d0bc5-1424-4599-a097-b5249395f3c4, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert 1a207006-6b2d-46b4-b268-e3fd485a53ab, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert c8805068-75f4-46da-9ea7-ce4a78f85b9c, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert 0dbd60a8-077e-48ef-a056-80d4c1731b83, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert 4d109fd5-86f0-4348-8e24-953d5c60a78a, it is allowed in current policy
Nov 23 16:26:14 foo@bar setroubleshoot[1477254]: Deleting alert a2257865-3101-44b5-9d42-20b50ed2f28c, it is allowed in current policy
Nov 23 16:26:47 foo@bar graylog-sidecar[1477098]: time="2022-11-23T16:26:47+01:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"https://192.168.8.21:9000/api/sidecars/35d2771a-8c5b-42aa-a1a4-2f05c504275b\": dial tcp x.x.x.x:9000: i/o timeout"
Nov 23 16:27:27 foo@bar graylog-sidecar[1477098]: time="2022-11-23T16:27:27+01:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"https://x.x.x.x:9000/api/sidecars/35d2771a-8c5b-42aa-a1a4-2f05c504275b\": dial tcp x.x.x.x:9000: i/o timeout"

From these messages I can gather that the new policy is being successfully enforced, but that doesn’t solve my issue. Looks like I solved a completely unrelated error :)). So since it isn’t an SELinux error, what can it also be?

Also looks like netcat can’t connect to the graylog server on port 900, since nc x.x.x.x 9000 doesn’t return anything at all. The Graylog server is on Network 192.x.x.x and the host machine is on 172.x.x.x. Maybe I need to allow the traffic from one machine to the other?

4

Hello,

Oh I see, might have a couple things going on preventing this to work correctly. By chance did you also check firewall/s?

Graylog ls on a different network? Can I ask what type of installation is this? Perhaps Docker?

172.16.0.0/12 IP addresses: 172.16.0.0 – 172.31.255.255
192.168.0.0/16 IP addresses: 192.168.0.0 – 192.168.255.255

You may need something to connect these two networks (i.e., Switch, router)
I have in the past created a bridge for my server stack using VLAN, something like this.

https://help.ubuntu.com/community/NetworkConnectionBridge

5

@gsmith Issue resolved! As I was laying down to sleep after a good dinner, I thought of one last idea. This morning I asked the network engineer to enable traffic from the host machine to the Graylog server.
Our network traffic is being managed by a different service alltogether and by default if you want traffic allowed from one network to the other you need to have that explicitly set. So that was the issue, as soon as the new firewall rules were enforced, my sidecar connected and no more errors!
Thanks for helping me out!

6

@cesq

Glad you resolved the issue :+1:

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.