Nxlog event consistency

[Jeff-Crowe]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

2. Describe your environment:

• OS Information:
  Graylog is running from a docker container within Ubuntu 22.04.2.
  Sidecar 1.4.0 is installed and running on a Windows 10 client.
• Package Version:
  Graylog version is 5.1.3
• Service logs, configurations, and environment variables:
  Nxlog configuration listed in Collector Configuration > nxlog:
  define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

Module xm_fileop When @daily Exec file_cycle('%ROOT%\data\nxlog.log', 7); Module xm_gelf # Avoid truncation of the short_message field to 64 characters. ShortMessageLength 65536 Module im_msvistalog PollInterval 20 SavePos True ReadFromLast True

Query <QueryList>\

[System[(Level=1 or Level=2 or Level=3)]]
[System[(EventID=20003)]]
[System[(Level=1 or Level=2 or Level=3)]]
[System[(EventID=19 or EventID=43)]]
[System[(EventID=137)]]

[System[(EventID=400)]]

*[System[(EventID=1150)]]

Module im_file File 'C:\Windows\MyLogDir\\*.log' PollInterval 1 SavePos True ReadFromLast True Recursive False RenameCheck False Exec $FileName = file_name(); # Send file name with each message Module om_udp Host x.x.x.x (obfuscated IP) Port 12201 OutputType GELF_UDP # These fields are needed for Graylog $gl2_source_collector = '${sidecar.nodeId}'; $collector_node_id = '${sidecar.nodeName}'; $Hostname = hostname_fqdn(); Path eventlog => gelf Path file => gelf

3. What steps have you already taken to try and solve the problem?

I’ve generated Events in the Windows client that I’m trying to see in Graylog, and have confirmed there’s no filters preventing the events I’m looking for.

4. How can the community help?

I’m trying to validate the nxlog configuration I’m currently using, but there seems to be Windows Events missing from the Graylog messages. Starting with Microsoft-Windows-Windows Defender/Operational, I am seeing event ID 1151 feed into Graylog hourly and it shows:
Endpoint Protection client health report (time in UTC):
Platform version: 4.18.23050.5
Engine version: 1.1.23060.1005
Network Realtime Inspection engine version: 1.1.23060.1005
Antivirus security intelligence version: 1.393.261.0

What I am not seeing is any other event related to Microsoft-Windows-Windows Defender/Operational, even though the nxlog config is only suppressing event ID 1150. When I initiate a Microsoft Defender scan, it creates an Event ID of 1000. I even downloaded a Malware test into the client which generates an event ID of 1116. Neither of those instances appear in Graylog. I’m looking for help to determine why not every event from Microsoft-Windows-Windows Defender/Operational appears in Graylog other than the 1150 event that I’m suppressing.

[Jeff-Crowe]

Adding this image since the config file didn’t come through clearly

[s0p4L1N]

Why do you have the value Query before <QueryList> on line 31 ?
Try also removing \ character at the end of your lines (31 to 42).

Here’s one of my nxlog sidecar input conf for event related to account management:

<Input windows_security_event>
    Module    im_msvistalog
    <QueryXML>
         <QueryList>
        
       <Query Id="0" Path="Security">
        <Select Path="Security">*[System[(EventID=4627)]]</Select>
        <Select Path="Security">*[System[(EventID=4703 or EventID=4704 or EventID=4705)]]</Select>
        <Select Path="Security">*[System[(EventID=4720)]]</Select>
        <Select Path="Security">*[System[(EventID &gt;=4722 and EventID &lt;=4735)]]</Select>
        <Select Path="Security">*[System[(EventID &gt;=4737 and EventID &lt;=4739)]]</Select>
        <Select Path="Security">*[System[(EventID &gt;=4741 and EventID &lt;=4767)]]</Select>
        <Select Path="Security">*[System[(EventID &gt;=4780 and EventID &lt;=4782)]]</Select>
        <Select Path="Security">*[System[(EventID=4793 or EventID=4794)]]</Select>
        <Select Path="Security">*[System[(EventID=4798 or EventID=4799)]]</Select>
        <Select Path="Security">*[System[(EventID=5376 or EventID=5377)]]</Select>
       </Query>
      
        </QueryList>
    </QueryXML>
    Exec      to_json();
</Input>

[Jeff-Crowe]

I’ve updated my config to remove that first Query value and without the \ at the end of each line, but still not seeing any Windows Defender events besides 1151

[s0p4L1N]

Here’s my working configuration for Windows Defender Logs:

define ROOT     C:\Program Files\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf\nxlog.d
define LOGDIR   %ROOT%\data

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

##### EXTENSION SECTION ########

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>


<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Extension json>
    Module    xm_json
</Extension>


##### INPUT SECTION ########

<Input windows_defender>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
         <!-- Inspired by Microsoft Documentation and/or IADGOV -->
            <Query Id="0" Path="Microsoft-Windows-Windows Defender/Operational">
                <!-- Modern Windows Defender event provider Detection events (1006-1009) and (1116-1119) -->
                <Select Path="Microsoft-Windows-Windows Defender/Operational">*[System[( (EventID &gt;= 1006 and EventID &lt;= 1009) )]]</Select>
                <Select Path="Microsoft-Windows-Windows Defender/Operational">*[System[( (EventID &gt;= 1116 and EventID &lt;= 1119) )]]</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec      to_json();
</Input>


##### OUTPUT SECTION ########


<Output gelf>
	Module om_tcp
	Host graylog.company.lan
	Port 12202
	OutputType  GELF_TCP
	<Exec>
	  # These fields are needed for Graylog
	  $gl2_source_collector = '${sidecar.nodeId}';
	  $collector_node_id = '${sidecar.nodeName}';
	</Exec>
</Output>


##### REDIRECT SECTION ########

# Tells to logged file to be redirected to GELF TCP input

#redirect some_name to output tcp
<Route route-1>
  Path windows_defender => gelf
</Route>

I tested by downloading EICAR file:

Local Event Log:

Graylog stream:

I use module xm_json, and NXLOG convert to json to parse data easily.

[Jeff-Crowe]

Ok, my file is pretty close to how that the one above is configured. I’m using the EICAR files as well to test for Malware. I can see that event in the client, but it doesn’t populate to Graylog. This is the only Windows Defender event that shows up

[s0p4L1N]

Can you try separate query id for each type of channel ?
What I have seen from here or here is that you cannot filter different channel from the same Query.

When you try to filter the channel through the GUI of Windows event Log, you can not filter another channel from the same query, only the selected channel, the process is the same for nxlog.

Try this new configuration:

<QueryXML>
        <QueryList>
               
              <Query Id="0" Path="Microsoft-Windows-Windows Defender/Operational">
                <Select Path="Microsoft-Windows-Windows Defender/Operational">*</Select>
                <Suppress Path="Microsoft-Windows-Windows Defender/Operational">*[System[(EventID=1150)]]</Suppress>
              </Query>

             <Query Id="1" Path="Application">
                <Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Suppress Path="Application">*[System[(EventID=20003)]]</Suppress>
             </Query>

             <Query Id="2" Path="System">
                <Select Path="System">*[System[(Level=1 or Level=2 or Level=3)]]</Select>
                <Select Path="System">*[System[(EventID=19 or EventID=43)]]</Select>
                <Suppress Path="System">*[System[(EventID=137)]]</Suppress>
             </Query>

             <Query Id="3" Path="Setup">
                <Select Path="Setup">*</Select>
             </Query>

             <Query Id="4" Path="Windows PowerShell">
                <Select Path="Windows PowerShell">*[System[(EventID=400)]]</Select>
             </Query>

        </QueryList>
    </QueryXML>

[Jeff-Crowe]

I updated my config to the above as well, but not seeing any change. For additional background, our org currently has Graylog version 2.4.7 with very similar Nxlog settings. I’m replacing that server with this one, but I’m just unclear what the reason for the difference is. Every intended Windows Defender/Operational log is populating in the old Graylog server based on a near similar Nxlog while the new server is being more selective

[Jeff-Crowe]

Turns out my issue was stemming from continual Indexer failures.

This was solved by going into the Index and rotating it. What caused the index issue and to prevent it is something else, but my missing logs have been solved by performing this.