Winlogbeat not working because of lumberjack protocol error

1. Describe your incident:
I installed the Graylog sidecar 1.3.0.1 on my windows sidecar using chocolatey and configured it to run winlogbeat. I did everything like in the documentation Graylog Sidecar
The sidecar and graylog seem to run without any problem, only winlogbeat doesn’t send any logs to graylog. (View the log below)

2. Describe your environment:
Windows: Windows Server 2019 (Only Powershell)
Graylog: Graylog 5.0.5
Graylog sidecar: 1.3.0.1 (1.4.0.1 not working: Sidecar installer 1.4.0.1 not working)
Graylog Instance: Debian 11, Mongodb 6.0, opensearch-2.4.1

Graylog and the graylog sidecar are running on AWS EC2 instances
The AWS load balancer I use and the security groups shouldn’t be the problem, I checked the connection with openssl

  • Service logs, configurations, and environment variables:

Sidecar configuration:

# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "https://graylog-testserver-load-balancer-**********.eu-central-1.elb.amazonaws.com:443/api/"

# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "1j2077gg0ajhfvjq6sfrjp62kj3up4bcpmj2cn28j*******"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.

# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"

# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: "Test_Windows_VM_for_Graylog-****"

# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files:
#       - "C:\Windows\System32\winevt\Logs"

# Directory where the sidecar stores internal data.
cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"

# The maximum size of the log file before it gets rotated.
log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
log_rotate_keep_files: 10

# How long to wait for the config validation command.
collector_validation_timeout: "1m"

# Directory where the sidecar generates configurations for collectors.
collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"

# Range of windows drives which are checked for disk usage. If their usage extends 75% they will be reported
# in the sidecar's status report to the Graylog server. Set to "" to disable disk scanning.
# Default:
windows_drive_range: "CDEFGHIJKLMNOPQRSTUVWXYZ"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_accesslist:
#       - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#       - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
#     collector_binaries_accesslist: []
#
# Default:
collector_binaries_accesslist:
- "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
- "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#  - "C:\\Program Files\\Filebeat\\filebeat.exe"
#  - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
#  - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
#  - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
#  - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
#  - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"

Winlogbeat configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["graylog-testserver-load-balancer-**********.eu-central-1.elb.amazonaws.com:5044"]
   index: winlogbeat
path:
  data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\winlogbeat"}\data
  logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
   - name: Security
   - name: System

Beats Input configuration:

* bind_address:0.0.0.0
* charset_name:UTF-8
* no_beats_prefix:false
* number_worker_threads:12
* override_source:*<empty>*
* port:5044
* recv_buffer_size:1048576
* tcp_keepalive:true
* tls_cert_file:*<empty>*
* tls_client_auth:disabled
* tls_client_auth_cert_file:*<empty>*
* tls_enable:false
* tls_key_file:*<empty>*
* tls_key_password:********

Winlogbeat logs (repetitive):

2023-03-30T08:55:15.191+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044)) established
2023-03-30T08:55:15.193+0200    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-30T08:55:15.193+0200    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-30T08:55:15.231+0200    ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: lumberjack protocol error
2023-03-30T08:55:15.232+0200    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-30T08:55:15.232+0200    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-30T08:55:15.439+0200    INFO    [publisher]     pipeline/retry.go:213   retryer: send wait signal to consumer
2023-03-30T08:55:15.439+0200    ERROR   [logstash]      logstash/async.go:280   Failed to publish events caused by: client is not connected
2023-03-30T08:55:15.439+0200    INFO    [publisher]     pipeline/retry.go:217     done
2023-03-30T08:55:17.181+0200    ERROR   [publisher_pipeline_output]     pipeline/output.go:180  failed to publish events: client is not connected
2023-03-30T08:55:17.181+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044))
2023-03-30T08:55:17.181+0200    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
2023-03-30T08:55:17.181+0200    INFO    [publisher]     pipeline/retry.go:223     done
2023-03-30T08:55:17.181+0200    INFO    [publisher]     pipeline/retry.go:213   retryer: send wait signal to consumer
2023-03-30T08:55:17.181+0200    INFO    [publisher]     pipeline/retry.go:217     done
2023-03-30T08:55:17.184+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(async(tcp://graylog-testserver-load-balancer-328612651.eu-central-1.elb.amazonaws.com:5044)) established

The Graylog sidecar logs don’t show any abnormalties and the sidecar is shown as running in graylog

3. What steps have you already taken to try and solve the problem?
I’ve tried to restart the sidecar and the input, I tried configuration changes of the sidecar, winlogbeat and the beats input, nothing helped. I tried to google for the error but I didn’t find any helpful information.
I configured filebeat on the sidecar after that for testing but it doesn’t send any logs, too.

4. How can the community help?
Can you tell me what the winlogbeat logs really mean or what the problem in general is? AmI missing something or may I have done something wrong during installation or have I configured something wrong? I’d apreciate any help to solve the problem.

The error is indicating your beat cannot communicate with the graylog input on TCP 5044.

Can you verify the following:

  1. Your graylog cluster has a beats input listening on TCP 5044, AND that input is started
  2. Your load balancer is properly configured to forward this traffic to the intended graylog server

For troubleshooting can you bypass the load balancer and communicate directly with the graylog input?

Can you also test if you can communicate with TCP 5044?

nc -z -v target_hostname 5044

Hey @Malle

I see one of your issues which is the port you configured “443” should be 9000.
found here.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.