Unable to get Server 2019 Event Viewer logs into Graylog via Sidecar and winlogbeat

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Hey, all. I’m in the process of bringing up a new Graylog Open 7 Core instance running on a pair of Almalinux 9 (RHEL 9 equivalent) servers. I’ve been able to ingest syslog messages from a number of my Almalinux 9 servers via rsyslog with no problems so I know it seems to be working.

What I’m having a problem with is getting my Windows Server 2019 system’s Event Viewer logs into Graylog via a sidecar and winlogbeat. I’m hoping it’s a simple configuration issue with either the log collector configuration in Graylog or with the sidecar.yml on the Windows server itself.

2. Describe your environment:

  • OS Information: Graylog Server and Data node are running on Almalinux 9

  • Package Version: graylog-server-7.0.2-1.x86_64; Winlogbeat 1.5.1 (downloaded from the Graylog downloads repository)

  • Service logs, configurations, and environment variables: Please see following comments to this thread for additional information.

3. What steps have you already taken to try and solve the problem?

  • I’ve read and reread the documentation on configuring Sidecar and collectors.
  • A lot of searching on Google and looking at results that seem to be similar to what I’m seeing.
  • Confirming that it’s not a firewall issue by dropping the firewall on both ends.
  • Taking TCP packet traces.

4. How can the community help?

Hopefully, due to my unfamiliarity with the product, it’s a simple configuration issue that I’ve overlooked.

I’ll be posting configs, etc. in following comments to this thread.

My sidecar.yml file (with a fake IP) from C:\Program Files\Graylog\sidecar:

# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "http://192.168.3.1:9000/api/"

# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "1j22od64d8ns0465m25limd4ke63s6s60dggdr6uc8hhcpe2ko7d"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: "server2019"

# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: true

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []

# Directory where the sidecar stores internal data.
#cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# How long to wait for the config validation command.
#collector_validation_timeout: "1m"

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"

# Range of windows drives which are checked for disk usage. If their usage extends 75% they will be reported
# in the sidecar's status report to the Graylog server. Set to "" to disable disk scanning.
# Default:
# windows_drive_range: "CDEFGHIJKLMNOPQRSTUVWXYZ"

# A list of tags to assign to this sidecar. Collector configuration matching any of these tags will automatically be
# applied to the sidecar.
# Default: []
tags: [windows, server2019]

# Example:
#    tags:
#    - apache-logs
#    - dns-logs

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_accesslist:
#       - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#       - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
#     collector_binaries_accesslist: []
#
# Default:
# collector_binaries_accesslist:
#  - "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
#  - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#  - "C:\\Program Files\\Filebeat\\filebeat.exe"
#  - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
#  - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
#  - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
#  - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
#  - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"
#  - "C:\\Program Files\\nxlog\\nxlog.exe"

The collector configuration (with fake IP) from the Graylog server:

# Required settings
fields_under_root: true
fields.collector_node_id: <node name>
fields.gl2_source_collector: <node id>


output.logstash:
   hosts: ["192.168.3.1:5044"]
path:
  data: <sidecar spool directory>\data
  logs: <sidecar spool directory>\logs
tags:
 - windows
winlogbeat:
  event_logs:
   - name: Application
     ignore_older: 96h
   - name: System
     ignore_older: 96h
   - name: Security
     ignore_older: 96h
   - name: Setup
     ignore_older: 96h
   - name: ForwardedEvents
     forwarded: true
     ignore_older: 96h
   - name: Microsoft-Windows-Windows Defender/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-Sysmon/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-TerminalServices-LocalSessionManager/Operational
     ignore_older: 96h
   - name: Microsoft-Windows-PowerShell/Operational
     ignore_older: 96h
   - name: windows PowerShell
     ignore_older: 96h

In terms of network setup:

  • Graylog server is at 192.168.3.1
  • Windows Server 2019 is at 192.168.1.1

(Not the real IPs but they are on two different /24 subnets.)

From the Windows Server host, I can browse to http://192.168.3.1:9000/api/ and the page displayed shows:

{"cluster_id":"60f4275d-f9a5-41ea-b1cb-af68eaa0c2a3","node_id":"7e9e49e0-962b-4c5d-bd7f-758ff7856651","version":"7.0.2+7a367fe","tagline":"Manage your logs in the dark and have lasers going and make it look like you're from space!"}

Firewall on the Graylog server is as follows:

[root@graylog-serv ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: cockpit dhcpv6-client ssh syslog
  ports: 9000/tcp 514/tcp 5044/tcp
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

But, as I said previously, I did try this with firewalls down on both sides and didn’t see any change in behavior.

I ended up doing a TCP packet trace from the Windows server and what I see is a lot of TCP RST and retransmit messages.

So, I’m kind of at a loss as to what’s going on. I’m really hoping it’s a simple configuration issue.

Thanks in advance to anyone who is able to help out. :folded_hands:

Almost forgot to mention, when I look at System > Sidecars in Graylog, it does show the status of “server2019” as “running”.

But, when I click on the “show messages” button, there’s nothing coming in.

If you root around the folders in the sidecar install on the windows server can you find that your config file for winlogbeat has been delivered to the server successfully? I forget the exact folder path.

I think I found what you were asking for. I found it in “C:\Program Files\Graylog\sidecar\generated\694441437d6dff70fa0a961e”

Required settings

fields_under_root: true
fields.collector_node_id: server2019
fields.gl2_source_collector: f86752e8-8e0a-4398-8736-1cf690dd1266

output.logstash:
hosts: [“192.168.3.1:5044”]
path:
data: C:\Program Files\Graylog\sidecar\generated\694441437d6dff70fa0a961e\data
logs: C:\Program Files\Graylog\sidecar\generated\694441437d6dff70fa0a961e\logs
tags:

windows
winlogbeat:
event_logs:

name: Application
ignore_older: 96h

name: System
ignore_older: 96h

name: Security
ignore_older: 96h

name: Setup
ignore_older: 96h

name: ForwardedEvents
forwarded: true
ignore_older: 96h

name: Microsoft-Windows-Windows Defender/Operational
ignore_older: 96h

name: Microsoft-Windows-Sysmon/Operational
ignore_older: 96h

name: Microsoft-Windows-TerminalServices-LocalSessionManager/Operational
ignore_older: 96h

name: Microsoft-Windows-PowerShell/Operational
ignore_older: 96h

name: windows PowerShell
ignore_older: 96h


Okay so the config is getting to the machine, thats awesome.

I would look at the beats input in graylog next. Are you sure its started (they go to setup mode first which is why I say that, so people think often its running then its not). If its running then go into input diagnostics, do you see any problems in there, messages being dropped etc.

That was it. I’ll have to look at the Sidecar instructions again because I must have missed a step where I was supposed to configure a Beats input. I’m getting Event Viewer logs in now.

Thank you so much for the nudge in the right direction. It’s working now!

Which instructions were you following?

I was using this: Install Sidecar and Deploy Collectors

And, in reviewing the instructions again, I completely overlooked the creation of an input.

That’s what I get for skimming while under pressure to get this deployed as quickly as I could. lol.

Thanks again for your help, @Joel_Duffield!