1. Describe your incident:
I installed the Graylog sidecar 1.3.0.1 on my windows sidecar using chocolatey and configured it to run winlogbeat. I did everything like in the documentation Graylog Sidecar
The sidecar and graylog seem to run without any problem, only winlogbeat doesn’t send any logs to graylog. (View the log below)
2. Describe your environment:
Windows: Windows Server 2019 (Only Powershell)
Graylog: Graylog 5.0.5
Graylog sidecar: 1.3.0.1 (1.4.0.1 not working: Sidecar installer 1.4.0.1 not working)
Graylog Instance: Debian 11, Mongodb 6.0, opensearch-2.4.1
Graylog and the graylog sidecar are running on AWS EC2 instances
The AWS load balancer I use and the security groups shouldn’t be the problem, I checked the connection with openssl
- Service logs, configurations, and environment variables:
Sidecar configuration:
# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "https://graylog-testserver-load-balancer-**********.eu-central-1.elb.amazonaws.com:443/api/"
# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "1j2077gg0ajhfvjq6sfrjp62kj3up4bcpmj2cn28j*******"
# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: "Test_Windows_VM_for_Graylog-****"
# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10
# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true
# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true
# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
# list_log_files:
# - "/var/log/nginx"
# - "/opt/app/logs"
#
# Default: empty list
#list_log_files:
# - "C:\Windows\System32\winevt\Logs"
# Directory where the sidecar stores internal data.
cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"
# Directory where the sidecar stores logs for collectors and the sidecar itself.
log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"
# The maximum size of the log file before it gets rotated.
log_rotate_max_file_size: "10MiB"
# The maximum number of old log files to retain.
log_rotate_keep_files: 10
# How long to wait for the config validation command.
collector_validation_timeout: "1m"
# Directory where the sidecar generates configurations for collectors.
collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"
# Range of windows drives which are checked for disk usage. If their usage extends 75% they will be reported
# in the sidecar's status report to the Graylog server. Set to "" to disable disk scanning.
# Default:
windows_drive_range: "CDEFGHIJKLMNOPQRSTUVWXYZ"
# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
# collector_binaries_accesslist:
# - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
# - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
# collector_binaries_accesslist: []
#
# Default:
collector_binaries_accesslist:
- "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
- "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
# - "C:\\Program Files\\Filebeat\\filebeat.exe"
# - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
# - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
# - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
# - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
# - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"
Winlogbeat configuration:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["graylog-testserver-load-balancer-**********.eu-central-1.elb.amazonaws.com:5044"]
index: winlogbeat
path:
data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\winlogbeat"}\data
logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
- windows
winlogbeat:
event_logs:
- name: Application
- name: Security
- name: System
Beats Input configuration:
* bind_address:0.0.0.0
* charset_name:UTF-8
* no_beats_prefix:false
* number_worker_threads:12
* override_source:*<empty>*
* port:5044
* recv_buffer_size:1048576
* tcp_keepalive:true
* tls_cert_file:*<empty>*
* tls_client_auth:disabled
* tls_client_auth_cert_file:*<empty>*
* tls_enable:false
* tls_key_file:*<empty>*
* tls_key_password:********
Winlogbeat logs (repetitive):
2023-03-30T08:55:15.191+0200 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044)) established
2023-03-30T08:55:15.193+0200 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-30T08:55:15.193+0200 INFO [publisher] pipeline/retry.go:223 done
2023-03-30T08:55:15.231+0200 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: lumberjack protocol error
2023-03-30T08:55:15.232+0200 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-30T08:55:15.232+0200 INFO [publisher] pipeline/retry.go:223 done
2023-03-30T08:55:15.439+0200 INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2023-03-30T08:55:15.439+0200 ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: client is not connected
2023-03-30T08:55:15.439+0200 INFO [publisher] pipeline/retry.go:217 done
2023-03-30T08:55:17.181+0200 ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: client is not connected
2023-03-30T08:55:17.181+0200 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044))
2023-03-30T08:55:17.181+0200 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-03-30T08:55:17.181+0200 INFO [publisher] pipeline/retry.go:223 done
2023-03-30T08:55:17.181+0200 INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
2023-03-30T08:55:17.181+0200 INFO [publisher] pipeline/retry.go:217 done
2023-03-30T08:55:17.184+0200 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044)) established
The Graylog sidecar logs don’t show any abnormalties and the sidecar is shown as running in graylog
3. What steps have you already taken to try and solve the problem?
I’ve tried to restart the sidecar and the input, I tried configuration changes of the sidecar, winlogbeat and the beats input, nothing helped. I tried to google for the error but I didn’t find any helpful information.
I configured filebeat on the sidecar after that for testing but it doesn’t send any logs, too.
4. How can the community help?
Can you tell me what the winlogbeat logs really mean or what the problem in general is? AmI missing something or may I have done something wrong during installation or have I configured something wrong? I’d apreciate any help to solve the problem.