Graylog Dropped Messages Logging

Graylog 3 is dropping some events, is there a log that will show why these are being dropped? In the standard /var/log/graylog-server/server.log I can’t find any reference for dropped messages.

I have disconnected all pipelines and there are no extractors on the input.
I created a specific input for troubleshooting and a nxlog config with an output file. (GELF 12202 tcp)
I have set nxlog to only pick up 2 event id’s (5379, 4776) 5379 comes through fine the 4776 does not ever show up in graylog.
In the nxlog_output file, both events are listed so something is happening when the event gets to graylog but I can not find any reference or reason why the event would be dropped.

nxlog config:

define ROOT C:\Program Files (x86)\nxlog
define SecurityEvents   4776, 5379

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension logrotate>
    Module  xm_fileop
    <Schedule>
        When    @daily
        Exec    file_cycle('%ROOT%\data\nxlog.log', 7);
     </Schedule>
</Extension>

<Extension fileop>
  Module  xm_fileop
</Extension> 

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension gelfExt>
  Module xm_gelf
  # Avoid truncation of the short_message field to 64 characters.
  ShortMessageLength 65536
</Extension>

<Input eventlog>
        Module im_msvistalog
        PollInterval 1
        SavePos True
        ReadFromLast True
        
        #Channel System
        <QueryXML>
          <QueryList>
           <Query Id='0'>
            <Select Path='Security'>*</Select>
            </Query>
          </QueryList>
        </QueryXML>
</Input>


<Input file>
	Module im_file
	File 'C:\Windows\MyLogDir\\*.log'
	#PollInterval 1
	#SavePos	True
	#ReadFromLast True
	#Recursive False
	#RenameCheck False
	#Exec $FileName = file_name(); # Send file name with each message
</Input>


<Output gelf>
	Module om_tcp
	Host 1.1.1.1
	Port 12202
	OutputType  GELF_tcp
	<Exec>
	  if ($EventID NOT IN (%SecurityEvents%)) drop ();
	  file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
	  # These fields are needed for Graylog
	  $gl2_source_collector = '${sidecar.nodeId}';
	  $collector_node_id = '${sidecar.nodeName}';
	</Exec>
</Output>


<Route route-1>
  Path eventlog => gelf
</Route>
<Route route-2>
  Path file => gelf
</Route>

nxlog Output File:

2020-09-17 09:34:11 SERVSERNAME.domain.com AUDIT_SUCCESS 4776 The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: USERX
Source Workstation: PCNAME
Error Code: 0x02020-09-17 09:34:11 SERVSERNAME.domain.com AUDIT_SUCCESS 4776 The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: USERX
Source Workstation: PCNAME
Error Code: 0x02020-09-17 09:34:12 SERVSERNAME.domain.com AUDIT_SUCCESS 4776 The computer attempted to validate the credentials for an account.

Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon Account: USERX
Source Workstation: SERVSERNAME
Error Code: 0x02020-09-17 09:34:15 SERVSERNAME.domain.com AUDIT_SUCCESS 5379 Credential Manager credentials were read.

Subject:
Security ID: S-1-5-21-1658086676-1236715088-4121974914-500
Account Name: USERX
Account Domain: SERVERNAME
Logon ID: 0x17AD7F025
Read Operation: Enumerate Credentials

This event occurs when a user performs a read operation on stored credentials in Credential Manager.

Input Config:
image

Thanks for your help.

So it looks like this was actually a 1000+ FIELDS issue. I could not locate wich field was causing the issue and my total fields for the index were 1013 so I increased it to 1200 and the messages started coming in.

I was getting the message below in the log but didn’t see any correlation to the other issue.
I followed the instructions here and it seems to be all good now.

2020-09-17T11:28:03.105-04:00 ERROR [Messages] Failed to index [5] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.
2020-09-17T11:28:03.105-04:00 WARN  [Messages] Failed to index message: index=<graylog_30> id=<5fad0b30-f8fa-11ea-b661-00505688d876> error=<{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [graylog_30] has been exceeded"}>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.