Long-time graylog user here - I began with 0.8.8 - but I’m really struggling with ingesting Windows logs into 3.1 for whatever reason and I could use some help. Completely new setup - did not upgrade old graylog environment.
I can ingest logs from my linux environment just fine - syslog-ng, apache, etc, all seem to work. But from Windows I keep running into indexing errors such as this one from GL’s server.log:
WARN [Messages] Failed to index message: index=<graylog_2> id=<963a5de0-f72c-11e9-86c6-00155d14b13c> error=<{"type":"mapper_parsing_exception","reason":"failed to parse field [level] of type [long] in document with id '963a5de0-f72c-11e9-86c6-00155d14b13c'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"Information\"
The error above is coming from a Windows Server 2012 host running Sidecar 1.0.2-1 with the following Winlogbeat configuration:
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
fields.source: ${sidecar.nodeName}
path:
data: 'C:\Program Files\Graylog\sidecar\cache\winlogbeat\data'
logs: 'C:\Program Files\Graylog\sidecar\logs'
tags:
- windows
winlogbeat.event_logs:
- name: Application
- name: System
- name: Security
output.logstash:
hosts: ["192.168.20.106:5045", "192.168.20.107:5045", "192.168.20.232:5045"]
Here’s the global beats input:
* bind_address: 0.0.0.0
* no_beats_prefix: true
* number_worker_threads: 2
* override_source: *<empty>*
* port: 5045
* recv_buffer_size: 1048576
* tcp_keepalive: false
* tls_cert_file: *<empty>*
* tls_client_auth: disabled
* tls_client_auth_cert_file: *<empty>*
* tls_enable: false
* tls_key_file: *<empty>*
* tls_key_password: ********
If I have Winlogbeat output to a file on the sending host (output.file) here is an example of what is logged (this particular server handles NPS/RADIUS):
{"@timestamp":"2019-10-25T12:11:08.000Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"message":"A RADIUS message was received from the invalid RADIUS client IP address 192.168.36.11.","level":"Error","log_name":"System","source":"REDACTED","host":{"name":"REDACTED"},"tags":["windows"],"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"computer_name":"REDACTED.REDACTED.com","collector_node_id":"REDACTED","gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","source_name":"NPS","record_number":"968918","event_id":13,"keywords":["Classic"],"event_data":{"param1":"192.168.36.11"},"type":"wineventlog"}
{"@timestamp":"2019-10-25T12:11:10.000Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"tags":["windows"],"gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","event_data":{"param1":"192.168.36.11"},"record_number":"968919","keywords":["Classic"],"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"computer_name":"REDACTED.REDACTED.com","type":"wineventlog","collector_node_id":"REDACTED","source":"REDACTED","host":{"name":"REDACTED"},"log_name":"System","source_name":"NPS","level":"Error","event_id":13,"message":"A RADIUS message was received from the invalid RADIUS client IP address 192.168.36.11."}
{"@timestamp":"2019-10-25T12:12:13.226Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"thread_id":2280,"version":1,"level":"Information","event_id":6272,"source_name":"Microsoft-Windows-Security-Auditing","provider_guid":"{54999925-5478-4994-A5BA-3E3B0328C30D}","type":"wineventlog","record_number":"2158586","process_id":540,"task":"Network Policy Server","message":"Network Policy Server granted access to a user.\n\nUser:\n\tSecurity ID:\t\t\tS-1-5-21-36288392-3099998340-2610000361-7787\n\tAccount Name:\t\t\thost/REDACTED.REDACTED.com\n\tAccount Domain:\t\t\REDACTED\n\tFully Qualified Account Name:\REDACTED\\REDACTED$\n\nClient Machine:\n\tSecurity ID:\t\t\tS-1-0-0\n\tAccount Name:\t\t\t-\n\tFully Qualified Account Name:\t-\n\tOS-Version:\t\t\t-\n\tCalled Station Identifier:\t\tREDACTED\n\tCalling Station Identifier:\t\tREDACTED\n\nNAS:\n\tNAS IPv4 Address:\t\t192.168.37.32\n\tNAS IPv6 Address:\t\t-\n\tNAS Identifier:\t\t\t-\n\tNAS Port-Type:\t\t\tWireless - IEEE 802.11\n\tNAS Port:\t\t\t2\n\nRADIUS Client:\n\tClient Friendly Name:\t\REDACTED\n\tClient IP Address:\t\t\t192.168.37.32\n\nAuthentication Details:\n\tConnection Request Policy Name:\tMeraki Wifi\n\tNetwork Policy Name:\t\tMeraki Wifi - Machine Auth\n\tAuthentication Provider:\t\tWindows\n\tAuthentication Server:\t\tREDACTED.REDACTED.com\n\tAuthentication Type:\t\tEAP\n\tEAP Type:\t\t\tMicrosoft: Smart Card or other certificate\n\tAccount Session Identifier:\t\t31384136384437314338384141324246\n\tLogging Results:\t\t\tAccounting information was written to the local log file.\n\nQuarantine Information:\n\tResult:\t\t\t\tFull Access\n\tSession Identifier:\t\t\t-","computer_name":"REDACTED.REDACTED.com","opcode":"Info","event_data":{"NASIPv4Address":"192.168.37.32","FullyQualifiedSubjectMachineName":"-","SubjectMachineName":"-","SubjectUserSid":"S-1-5-21-36288392-302120-2617777361-7787","SubjectUserName":"host/REDACTED.REDACTED.com","QuarantineSessionIdentifier":"-","EAPType":"Microsoft: Smart Card or other certificate","CalledStationID":"REDACTED","LoggingResult":"Accounting information was written to the local log file.","NASIdentifier":"-","NASPort":"2","AccountSessionIdentifier":"313855554338384141324246","AuthenticationProvider":"Windows","AuthenticationType":"EAP","NASPortType":"Wireless - IEEE 802.11","ClientName":"REDACTED","CallingStationID":"REDACTED","MachineInventory":"-","AuthenticationServer":"REDACTED.REDACTED.com","NASIPv6Address":"-","QuarantineState":"Full Access","NetworkPolicyName":"Meraki Wifi - Machine Auth","SubjectMachineSID":"S-1-0-0","ClientIPAddress":"192.168.37.32","FullyQualifiedSubjectUserName":"REDACTED\\REDACTED$","ProxyPolicyName":"Meraki Wifi","SubjectDomainName":"REDACTED"},"collector_node_id":"REDACTED","host":{"name":"REDACTED"},"keywords":["Audit Success"],"log_name":"Security","source":"REDACTED","beat":{"hostname":"REDACTED","version":"6.4.2","name":"REDACTED"},"tags":["windows"],"gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23"}
{"@timestamp":"2019-10-25T12:12:13.226Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"host":{"name":"REDACTED"},"source_name":"Microsoft-Windows-Security-Auditing","collector_node_id":"REDACTED","keywords":["Audit Success"],"process_id":540,"computer_name":"REDACTED.REDACTED.com","tags":["windows"],"opcode":"Info","event_id":6278,"source":"REDACTED","gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","record_number":"2158587","level":"Information","event_data":{"ExtendedQuarantineState":"-","AuthenticationProvider":"Windows","QuarantineState":"Full Access","MachineInventory":"-","NASIdentifier":"-","AuthenticationType":"EAP","CalledStationID":"REDACTED","ClientIPAddress":"192.168.37.32","FullyQualifiedSubjectUserName":"REDACTED\\REDACTED$","NASPortType":"Wireless - IEEE 802.11","QuarantineHelpURL":"-","NASIPv4Address":"192.168.37.32","CallingStationID":"REDACTED","AuthenticationServer":"REDACTED.REDACTED.com","SubjectUserName":"host/REDACTED.REDACTED.com","NASPort":"2","SubjectUserSid":"S-1-5-21-36288392-3022222340-26161-7787","SubjectMachineSID":"S-1-0-0","NASIPv6Address":"-","ProxyPolicyName":"Meraki Wifi","SubjectDomainName":"REDACTED","SubjectMachineName":"-","EAPType":"Microsoft: Smart Card or other certificate","ClientName":"REDACTED","NetworkPolicyName":"Meraki Wifi - Machine Auth","QuarantineSessionID":"-","AccountSessionIdentifier":"3138413666643734141324246","QuarantineSystemHealthResult":"-","FullyQualifiedSubjectMachineName":"-"},"provider_guid":"{5485-5478-4994-A5BA-3E3B0328C30D}","task":"Network Policy Server","thread_id":2280,"message":"Network Policy Server granted full access to a user because the host met the defined health policy.\n\nUser:\n\tSecurity ID:\t\t\tS-1-5-21-36288392-3021999940-2618-7787\n\tAccount Name:\t\t\thost/REDACTED.REDACTED.com\n\tAccount Domain:\t\t\REDACTED\n\tFully Qualified Account Name:\tREDACTED\\REDACTED$\n\nClient Machine:\n\tSecurity ID:\t\t\tS-1-0-0\n\tAccount Name:\t\t\t-\n\tFully Qualified Account Name:\t-\n\tOS-Version:\t\t\t-\n\tCalled Station Identifier:\t\tREDACTED\n\tCalling Station Identifier:\t\tREDACTED\n\nNAS:\n\tNAS IPv4 Address:\t\t192.168.37.32\n\tNAS IPv6 Address:\t\t-\n\tNAS Identifier:\t\t\t-\n\tNAS Port-Type:\t\t\tWireless - IEEE 802.11\n\tNAS Port:\t\t\t2\n\nRADIUS Client:\n\tClient Friendly Name:\t\REDACTED\n\tClient IP Address:\t\t\t192.168.37.32\n\nAuthentication Details:\n\tConnection Request Policy Name:\tMeraki Wifi\n\tNetwork Policy Name:\t\tMeraki Wifi - Machine Auth\n\tAuthentication Provider:\t\tWindows\n\tAuthentication Server:\t\tREDACTED.REDACTED.com\n\tAuthentication Type:\t\tEAP\n\tEAP Type:\t\t\tMicrosoft: Smart Card or other certificate\n\tAccount Session Identifier:\t\t31384136384437384141324246\n\nQuarantine Information:\n\tResult:\t\t\t\tFull Access\n\tExtended-Result:\t\t\t-\n\tSession Identifier:\t\t\t-\n\tHelp URL:\t\t\t-\n\tSystem Health Validator Result(s):\t-","type":"wineventlog","log_name":"Security"}
And here’s something else that may or may not be helpful. I tested a RAW Plaintext/TCP input on that port and the logs were indexed, but it was just garbage / garbled non-ascii data. Maybe that’s to be expected from a beats source and a raw input, but I’m just learning beats (I did everything over syslog/udp in my old GL environment) so I found it interesting.
I have no extractors or pipeline rules.
3 centos7 node GL 3.1.2
3 centos7 node ES 6.8.3
2 centos7 node HAP LB 1.5.18 (GL’s UI plus certain incoming logs, but not Windows logs atm [although I intend to run them through here too, once things are stable])
Help! And TIA!