Failed to index messages coming from Windows hosts

Long-time graylog user here - I began with 0.8.8 - but I’m really struggling with ingesting Windows logs into 3.1 for whatever reason and I could use some help. Completely new setup - did not upgrade old graylog environment.

I can ingest logs from my linux environment just fine - syslog-ng, apache, etc, all seem to work. But from Windows I keep running into indexing errors such as this one from GL’s server.log:

    WARN  [Messages] Failed to index message: index=<graylog_2> id=<963a5de0-f72c-11e9-86c6-00155d14b13c> error=<{"type":"mapper_parsing_exception","reason":"failed to parse field [level] of type [long] in document with id '963a5de0-f72c-11e9-86c6-00155d14b13c'","caused_by":{"type":"illegal_argument_exception","reason":"For input string: \"Information\"

The error above is coming from a Windows Server 2012 host running Sidecar 1.0.2-1 with the following Winlogbeat configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
fields.source: ${sidecar.nodeName}

path:
  data: 'C:\Program Files\Graylog\sidecar\cache\winlogbeat\data'
  logs: 'C:\Program Files\Graylog\sidecar\logs'

tags:
 - windows

winlogbeat.event_logs:
   - name: Application
   - name: System
   - name: Security

output.logstash:
  hosts: ["192.168.20.106:5045", "192.168.20.107:5045", "192.168.20.232:5045"]

Here’s the global beats input:

* bind_address: 0.0.0.0
* no_beats_prefix: true
* number_worker_threads: 2
* override_source: *<empty>*
* port: 5045
* recv_buffer_size: 1048576
* tcp_keepalive: false
* tls_cert_file: *<empty>*
* tls_client_auth: disabled
* tls_client_auth_cert_file: *<empty>*
* tls_enable: false
* tls_key_file: *<empty>*
* tls_key_password: ********

If I have Winlogbeat output to a file on the sending host (output.file) here is an example of what is logged (this particular server handles NPS/RADIUS):

{"@timestamp":"2019-10-25T12:11:08.000Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"message":"A RADIUS message was received from the invalid RADIUS client IP address 192.168.36.11.","level":"Error","log_name":"System","source":"REDACTED","host":{"name":"REDACTED"},"tags":["windows"],"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"computer_name":"REDACTED.REDACTED.com","collector_node_id":"REDACTED","gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","source_name":"NPS","record_number":"968918","event_id":13,"keywords":["Classic"],"event_data":{"param1":"192.168.36.11"},"type":"wineventlog"}
{"@timestamp":"2019-10-25T12:11:10.000Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"tags":["windows"],"gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","event_data":{"param1":"192.168.36.11"},"record_number":"968919","keywords":["Classic"],"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"computer_name":"REDACTED.REDACTED.com","type":"wineventlog","collector_node_id":"REDACTED","source":"REDACTED","host":{"name":"REDACTED"},"log_name":"System","source_name":"NPS","level":"Error","event_id":13,"message":"A RADIUS message was received from the invalid RADIUS client IP address 192.168.36.11."}
{"@timestamp":"2019-10-25T12:12:13.226Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"thread_id":2280,"version":1,"level":"Information","event_id":6272,"source_name":"Microsoft-Windows-Security-Auditing","provider_guid":"{54999925-5478-4994-A5BA-3E3B0328C30D}","type":"wineventlog","record_number":"2158586","process_id":540,"task":"Network Policy Server","message":"Network Policy Server granted access to a user.\n\nUser:\n\tSecurity ID:\t\t\tS-1-5-21-36288392-3099998340-2610000361-7787\n\tAccount Name:\t\t\thost/REDACTED.REDACTED.com\n\tAccount Domain:\t\t\REDACTED\n\tFully Qualified Account Name:\REDACTED\\REDACTED$\n\nClient Machine:\n\tSecurity ID:\t\t\tS-1-0-0\n\tAccount Name:\t\t\t-\n\tFully Qualified Account Name:\t-\n\tOS-Version:\t\t\t-\n\tCalled Station Identifier:\t\tREDACTED\n\tCalling Station Identifier:\t\tREDACTED\n\nNAS:\n\tNAS IPv4 Address:\t\t192.168.37.32\n\tNAS IPv6 Address:\t\t-\n\tNAS Identifier:\t\t\t-\n\tNAS Port-Type:\t\t\tWireless - IEEE 802.11\n\tNAS Port:\t\t\t2\n\nRADIUS Client:\n\tClient Friendly Name:\t\REDACTED\n\tClient IP Address:\t\t\t192.168.37.32\n\nAuthentication Details:\n\tConnection Request Policy Name:\tMeraki Wifi\n\tNetwork Policy Name:\t\tMeraki Wifi - Machine Auth\n\tAuthentication Provider:\t\tWindows\n\tAuthentication Server:\t\tREDACTED.REDACTED.com\n\tAuthentication Type:\t\tEAP\n\tEAP Type:\t\t\tMicrosoft: Smart Card or other certificate\n\tAccount Session Identifier:\t\t31384136384437314338384141324246\n\tLogging Results:\t\t\tAccounting information was written to the local log file.\n\nQuarantine Information:\n\tResult:\t\t\t\tFull Access\n\tSession Identifier:\t\t\t-","computer_name":"REDACTED.REDACTED.com","opcode":"Info","event_data":{"NASIPv4Address":"192.168.37.32","FullyQualifiedSubjectMachineName":"-","SubjectMachineName":"-","SubjectUserSid":"S-1-5-21-36288392-302120-2617777361-7787","SubjectUserName":"host/REDACTED.REDACTED.com","QuarantineSessionIdentifier":"-","EAPType":"Microsoft: Smart Card or other certificate","CalledStationID":"REDACTED","LoggingResult":"Accounting information was written to the local log file.","NASIdentifier":"-","NASPort":"2","AccountSessionIdentifier":"313855554338384141324246","AuthenticationProvider":"Windows","AuthenticationType":"EAP","NASPortType":"Wireless - IEEE 802.11","ClientName":"REDACTED","CallingStationID":"REDACTED","MachineInventory":"-","AuthenticationServer":"REDACTED.REDACTED.com","NASIPv6Address":"-","QuarantineState":"Full Access","NetworkPolicyName":"Meraki Wifi - Machine Auth","SubjectMachineSID":"S-1-0-0","ClientIPAddress":"192.168.37.32","FullyQualifiedSubjectUserName":"REDACTED\\REDACTED$","ProxyPolicyName":"Meraki Wifi","SubjectDomainName":"REDACTED"},"collector_node_id":"REDACTED","host":{"name":"REDACTED"},"keywords":["Audit Success"],"log_name":"Security","source":"REDACTED","beat":{"hostname":"REDACTED","version":"6.4.2","name":"REDACTED"},"tags":["windows"],"gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23"}
{"@timestamp":"2019-10-25T12:12:13.226Z","@metadata":{"beat":"winlogbeat","type":"doc","version":"6.4.2"},"beat":{"name":"REDACTED","hostname":"REDACTED","version":"6.4.2"},"host":{"name":"REDACTED"},"source_name":"Microsoft-Windows-Security-Auditing","collector_node_id":"REDACTED","keywords":["Audit Success"],"process_id":540,"computer_name":"REDACTED.REDACTED.com","tags":["windows"],"opcode":"Info","event_id":6278,"source":"REDACTED","gl2_source_collector":"a7d5a715-17f7-4fa6-9039-d153493a1a23","record_number":"2158587","level":"Information","event_data":{"ExtendedQuarantineState":"-","AuthenticationProvider":"Windows","QuarantineState":"Full Access","MachineInventory":"-","NASIdentifier":"-","AuthenticationType":"EAP","CalledStationID":"REDACTED","ClientIPAddress":"192.168.37.32","FullyQualifiedSubjectUserName":"REDACTED\\REDACTED$","NASPortType":"Wireless - IEEE 802.11","QuarantineHelpURL":"-","NASIPv4Address":"192.168.37.32","CallingStationID":"REDACTED","AuthenticationServer":"REDACTED.REDACTED.com","SubjectUserName":"host/REDACTED.REDACTED.com","NASPort":"2","SubjectUserSid":"S-1-5-21-36288392-3022222340-26161-7787","SubjectMachineSID":"S-1-0-0","NASIPv6Address":"-","ProxyPolicyName":"Meraki Wifi","SubjectDomainName":"REDACTED","SubjectMachineName":"-","EAPType":"Microsoft: Smart Card or other certificate","ClientName":"REDACTED","NetworkPolicyName":"Meraki Wifi - Machine Auth","QuarantineSessionID":"-","AccountSessionIdentifier":"3138413666643734141324246","QuarantineSystemHealthResult":"-","FullyQualifiedSubjectMachineName":"-"},"provider_guid":"{5485-5478-4994-A5BA-3E3B0328C30D}","task":"Network Policy Server","thread_id":2280,"message":"Network Policy Server granted full access to a user because the host met the defined health policy.\n\nUser:\n\tSecurity ID:\t\t\tS-1-5-21-36288392-3021999940-2618-7787\n\tAccount Name:\t\t\thost/REDACTED.REDACTED.com\n\tAccount Domain:\t\t\REDACTED\n\tFully Qualified Account Name:\tREDACTED\\REDACTED$\n\nClient Machine:\n\tSecurity ID:\t\t\tS-1-0-0\n\tAccount Name:\t\t\t-\n\tFully Qualified Account Name:\t-\n\tOS-Version:\t\t\t-\n\tCalled Station Identifier:\t\tREDACTED\n\tCalling Station Identifier:\t\tREDACTED\n\nNAS:\n\tNAS IPv4 Address:\t\t192.168.37.32\n\tNAS IPv6 Address:\t\t-\n\tNAS Identifier:\t\t\t-\n\tNAS Port-Type:\t\t\tWireless - IEEE 802.11\n\tNAS Port:\t\t\t2\n\nRADIUS Client:\n\tClient Friendly Name:\t\REDACTED\n\tClient IP Address:\t\t\t192.168.37.32\n\nAuthentication Details:\n\tConnection Request Policy Name:\tMeraki Wifi\n\tNetwork Policy Name:\t\tMeraki Wifi - Machine Auth\n\tAuthentication Provider:\t\tWindows\n\tAuthentication Server:\t\tREDACTED.REDACTED.com\n\tAuthentication Type:\t\tEAP\n\tEAP Type:\t\t\tMicrosoft: Smart Card or other certificate\n\tAccount Session Identifier:\t\t31384136384437384141324246\n\nQuarantine Information:\n\tResult:\t\t\t\tFull Access\n\tExtended-Result:\t\t\t-\n\tSession Identifier:\t\t\t-\n\tHelp URL:\t\t\t-\n\tSystem Health Validator Result(s):\t-","type":"wineventlog","log_name":"Security"}

And here’s something else that may or may not be helpful. I tested a RAW Plaintext/TCP input on that port and the logs were indexed, but it was just garbage / garbled non-ascii data. Maybe that’s to be expected from a beats source and a raw input, but I’m just learning beats (I did everything over syslog/udp in my old GL environment) so I found it interesting.

I have no extractors or pipeline rules.

3 centos7 node GL 3.1.2
3 centos7 node ES 6.8.3
2 centos7 node HAP LB 1.5.18 (GL’s UI plus certain incoming logs, but not Windows logs atm [although I intend to run them through here too, once things are stable])

Help! And TIA!

the quickest fix would be to create a new index that is dedicated for windows logs (what is my suggestion) - cause the problem is that you have a field (level) that is created as a field that holds a number in elasticsearch (because that is the default on most unixoide systems) - but windows is having the same field that contains a string … so that can’t be ingested to the index as long as the field is a number …

1 Like

Thank you, Jan. That worked perfectly and I’m now kicking myself for not trying that sooner. I must confess I’m not as adept at the index model as I should be, but that’s now about to change.

Thank you again. Now I’m off to dive deeper into the documentation re indexes, streams, and pipelines.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.