Yesterday I upgraded to Graylog 5.0. Due to requirements, I also upgraded mongodb from 4.0 > 4.2 > 4.4 > 5.0. I also upgraded Java 11 > Java 17. Elasticsearch was already on 7.10. Ever since, once an hour on the hour, I get an error in server.log relating to beat input.
The client sending the logs seems to be random. The input seems to be working. Logs are still being received from all clients. I am just curious as to why this started happening upon upgrade.
What does your sidecar.yml file look like on the affected machines? That is the configuration part of Sidecar that creates the connection for configurations and data transfer.
Please use the </> forum tool on posted logs/code, it makes it easier to read!
# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "https://10.7.x.x:9000/api"
# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "1mon2r9p2isuma5leb....."
# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: ""
# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10
# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true
# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true
# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
# list_log_files:
# - "/var/log/nginx"
# - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []
# Directory where the sidecar stores internal data.
#cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"
# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"
# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"
# The maximum number of old log files to retain.
#log_rotate_keep_files: 10
# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"
# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
# collector_binaries_accesslist:
# - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
# - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
# collector_binaries_accesslist: []
#
# Default:
# collector_binaries_accesslist:
# - "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
# - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
# - "C:\\Program Files\\Filebeat\\filebeat.exe"
# - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
# - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
# - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
# - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
# - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"
I think if it were the logstash output IP config, it would have been giving errors before the update and would error every time any of the servers using that config check in, instead of a random server once an hour
I think @gsmith is on to something. The output.logstash you is a different format than what I have seen… I am no expert in yml files but I do know they are VERY particular on spacing and indentation. much like the one @gsmith posted, mine looks like:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
# Define the output (we use Logstash for Graylog)
output.logstash:
hosts: ["10.7.44.14:5044"]
and I am still getting the error once an hour.
As stated, everything seems to be working fine regardless. I am receiving logs from all hosts using the config and I am able to restart the process remotely through graylog.
I might try to update the sidecars to 1.3 and see if that makes a difference.
