I have been reviewing forum posts here and scouring the Internet for suggestions most of the day, but have reached a point that I have run out of ideas on how to solve a peculiar issue with a Sidecar-Winlogbeat installation I am testing.
I call this peculiar because it does not quite match any other issues I have seen posted and because the installation is 70% working. I have tried Sidecar versions 1.0.1 and 1.0.2 (not at the same time) on a Windows 2012 R2 domain controller and it is configured to collect/forward the System, Application, and Security events. There is no issue applying a configuration from Graylog and the System and Application events forward to Graylog without any issues at all. The Security events will not forward no matter what I have tried which includes changing permissions, GPOs, etc.
I have installed the Sidecar agent on a Windows 2019 Standard server (not a DC) and several Windows 2012 Standard servers (not a DCs) and the System, Application, and Security events forward to Graylog without an issue. This leads me to believe it is something specific to domain controllers and I am working on getting access to a 2016 domain controller to see what happens.
I noticed on the 2012 domain controller where I am having the issue, the .winglogbeat.yml file does not show the Security or ForwardedEvents logs even though it is listed in the configuration file.
The collector configuration is very vanilla (IP address redacted):
fields_under_root: true
fields.collector_node_id: {sidecar.nodeName}
fields.gl2_source_collector: {sidecar.nodeId}
I am running sidecar on our DC’s and get the security events just fine. Are you running the sidecar service under a different account? Ours run under the “Local System” account. My working configuration is below (nicely formatted with posting tools )
You could check the sidecar.log and winlogbeat log files at C:\Program Files\Graylog\sidecar\logs that might tell you something about the problem it’s having. If you see something odd and can’t figure it out - post it up… but not the WHOLE log file(s) and please use the formatting tools.
Also check C:\Program Files\Graylog\sidecar\generated\winlogbeat.conf - that should match the configuration you have applied to the server.
Working Windows DC configuration:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["${user.BeatsInput}"]
path:
data: C:\Program Files\Graylog\sidecar\cache\winlogbeat\data
logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
- DC
winlogbeat:
event_logs:
- name: Application
- name: System
- name: Security
- name: DFS Replication
- name: Directory Service
- name: DNS Server
- name: File Replication Service
My collector configuration looks very similar to yours with some minor changes in the tags used and not collecting the DFS, Directory Service, DNS, or File Replication events.
graylog-sidecar.exe and winlogbeat.exe are both running as services under the “Local System” account on the domain controller.
The generated winlogbeat.conf file matches my collector configuration in Graylog.
The log winlogbeat has an error reference access issues to the Security event log. Based on that I have tried many different suggestions I have found online. It seems specific to domain controllers and something within a security policy that I have not figured out yet. Here is a snippet of the log with the error.
2020-01-09T11:08:31.044-0800 INFO instance/beat.go:273 Setup Beat: winlogbeat; Version: 6.4.2
2020-01-09T11:08:31.046-0800 INFO pipeline/module.go:98 Beat name: stage-spark
2020-01-09T11:08:31.046-0800 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\ProgramData\winlogbeat\.winlogbeat.yml
2020-01-09T11:08:31.046-0800 INFO instance/beat.go:367 winlogbeat start running.
2020-01-09T11:08:31.047-0800 INFO [monitoring] log/log.go:114 Starting metrics logging every 30s
2020-01-09T11:08:31.055-0800 WARN beater/eventlogger.go:104 EventLog[Security] Open() error. No events will be read from this source. Access is denied.
2020-01-09T11:08:33.079-0800 INFO beater/eventlogger.go:73 EventLog[System] successfully published 2 events
I am still working on getting access to a Windows 2016 DC to see if that will help eliminate the OS version from the situation.
)