Logstash input with gelf , raise a parse error

logstash 6.8.9

nxlog conf

define ROOT     D:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data

<Extension _gelf>
	 Module      xm_gelf
</Extension>

<Extension _charconv>
	Module      xm_charconv
	AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
	Module      xm_exec
</Extension>

<Extension _fileop>
	Module      xm_fileop

	# Check the size of our log file hourly, rotate if larger than 5MB
	<Schedule>
		Every   1 hour
		Exec    if (file_exists('%LOGFILE%') and \
				   (file_size('%LOGFILE%') >= 5M)) \
					file_cycle('%LOGFILE%', 8);
	</Schedule>

	# Rotate our log file every week on Sunday at midnight
	<Schedule>
		When    @weekly
		Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
	</Schedule>
</Extension>

<Input in>
	Module      im_msvistalog
</Input>

<Output out>
	Module      om_udp    
	Host        172.16.1.170
	Port        6666
	OutputType  GELF

</Output>

<Route 1>
	Path in => out
</Route>

logstash config

input {
    gelf {  
            host => "0.0.0.0"
            port_udp => 6666
            use_udp => true
    }
}
output {
    kafka { 
            bootstrap_servers => "127.0.0.1:9092"
            topic_id  => "test" 
            max_request_size => 9044723
    }
}

log from logstash raise a error, LogStash::Json::ParserError: Invalid UTF-8 middle byte 0x22

[2020-06-08T18:05:33,000][ERROR][logstash.inputs.gelf ] JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Invalid UTF-8 middle byte 0x22
at [Source: (byte)“{“version”:“1.1”,”_Keywords":-9187343239835811840,“_EventType”:“ERROR”,“_SeverityValue”:4,“_Severity”:“ERROR”,“_EventID”:7000,“_SourceName”:“Service Control Manager”,“_ProviderGuid”:“{555908D1-A6D7-4695-8E1E-26931D2012F4}”,“_Version”:0,“_Task”:0,“_OpcodeValue”:0,“_RecordNumber”:234289,“_ProcessID”:820,“_ThreadID”:2872,“_Channel”:“System”,“_param1”:“THlpDrv”,“_param2”:“%%2”,“_EventReceivedTime”:“2020-06-08 18:05:17”,“_SourceModuleName”:“in”,“_SourceModuleType”:“im_msvistalog”,“host”:“DESKTOP-3N7O”[truncated 235 bytes]; line: 1, column: 598]>, :data=>“"{\"version\":\"1.1\",\"_Keywords\":-9187343239835811840,\"_EventType\":\"ERROR\",\"_SeverityValue\":4,\"_Severity\":\"ERROR\",\"_EventID\":7000,\"_SourceName\":\"Service Control Manager\",\"_ProviderGuid\":\"{555908D1-A6D7-4695-8E1E-26931D2012F4}\",\"_Version\":0,\"_Task\":0,\"_OpcodeValue\":0,\"_RecordNumber\":234289,\"_ProcessID\":820,\"_ThreadID\":2872,\"_Channel\":\"System\",\"_param1\":\"THlpDrv\",\"_param2\":\"%%2\",\"_EventReceivedTime\":\"2020-06-08 18:05:17\",\"_SourceModuleName\":\"in\",\"_SourceModuleType\":\"im_msvistalog\",\"host\":\"DESKTOP-3N7O189.com\",\"short_message\":\"\xE7\x94\xB1\xE4\xBA\x8E\xE4\xB8\x8B\xE5\x88\x97\xE9\x94\x99\xE8\xAF\xAF\xEF\xBC\x8CTHlpDrv \xE6\x9C\x8D\xE5\x8A\xA1\xE5\x90\xAF\xE5\x8A\xA8\xE5\xA4\xB1\xE8\xB4\xA5: \\r\\n\xE7\xB3\xBB\xE7\xBB\x9F\xE6\x89\xBE\xE4\xB8\x8D\xE5\",\"full_message\":\"\xE7\x94\xB1\xE4\xBA\x8E\xE4\xB8\x8B\xE5\x88\x97\xE9\x94\x99\xE8\xAF\xAF\xEF\xBC\x8CTHlpDrv \xE6\x9C\x8D\xE5\x8A\xA1\xE5\x90\xAF\xE5\x8A\xA8\xE5\xA4\xB1\xE8\xB4\xA5: \\r\\n\xE7\xB3\xBB\xE7\xBB\x9F\xE6\x89\xBE\xE4\xB8\x8D\xE5\x88\xB0\xE6\x8C\x87\xE5\xAE\x9A\xE7\x9A\x84\xE6\x96\x87\xE4\xBB\xB6\xE3\x80\x82\",\"timestamp\":1591610716,\"level\":3}"”}

where the config is wrong?

Maybe yu have problem with UTF-8 encoding. Check if your source message is correctly UTF-8 encoded, and doesn’t contains non-valid chars…

[ERROR] 2020-06-09 16:05:02.969 [Ruby-0-Thread-8: :1] gelf - JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Invalid UTF-8 middle byte 0x22
at [Source: (byte)“{“version”:“1.1”,”_Keywords":-9187343239835811840,“_EventType”:“ERROR”,“_SeverityValue”:4,“_Severity”:“ERROR”,“_EventID”:7000,“_SourceName”:“Service Control Manager”,“_ProviderGuid”:“{555908D1-A6D7-4695-8E1E-26931D2012F4}”,“_Version”:0,“_Task”:0,“_OpcodeValue”:0,“_RecordNumber”:237848,“_ProcessID”:776,“_ThreadID”:13244,“_Channel”:“System”,“_param1”:“THlpDrv”,“_param2”:“%%2”,“_EventReceivedTime”:“2020-06-09 16:04:45”,“_SourceModuleName”:“in”,“_SourceModuleType”:“im_msvistalog”,“host”:“DESKTOP-3N7”[truncated 236 bytes]; line: 1, column: 599]>, :data=>“"{\"version\":\"1.1\",\"_Keywords\":-9187343239835811840,\"_EventType\":\"ERROR\",\"_SeverityValue\":4,\"_Severity\":\"ERROR\",\"_EventID\":7000,\"_SourceName\":\"Service Control Manager\",\"_ProviderGuid\":\"{555908D1-A6D7-4695-8E1E-26931D2012F4}\",\"_Version\":0,\"_Task\":0,\"_OpcodeValue\":0,\"_RecordNumber\":237848,\"_ProcessID\":776,\"_ThreadID\":13244,\"_Channel\":\"System\",\"_param1\":\"THlpDrv\",\"_param2\":\"%%2\",\"_EventReceivedTime\":\"2020-06-09 16:04:45\",\"_SourceModuleName\":\"in\",\"_SourceModuleType\":\"im_msvistalog\",\"host\":\"DESKTOP-3N7O189.com\",\"short_message\":\"\xE7\x94\xB1\xE4\xBA\x8E\xE4\xB8\x8B\xE5\x88\x97\xE9\x94\x99\xE8\xAF\xAF\xEF\xBC\x8CTHlpDrv \xE6\x9C\x8D\xE5\x8A\xA1\xE5\x90\xAF\xE5\x8A\xA8\xE5\xA4\xB1\xE8\xB4\xA5: \\r\\n\xE7\xB3\xBB\xE7\xBB\x9F\xE6\x89\xBE\xE4\xB8\x8D\xE5\",\"full_message\":\"\xE7\x94\xB1\xE4\xBA\x8E\xE4\xB8\x8B\xE5\x88\x97\xE9\x94\x99\xE8\xAF\xAF\xEF\xBC\x8CTHlpDrv \xE6\x9C\x8D\xE5\x8A\xA1\xE5\x90\xAF\xE5\x8A\xA8\xE5\xA4\xB1\xE8\xB4\xA5: \\r\\n\xE7\xB3\xBB\xE7\xBB\x9F\xE6\x89\xBE\xE4\xB8\x8D\xE5\x88\xB0\xE6\x8C\x87\xE5\xAE\x9A\xE7\x9A\x84\xE6\x96\x87\xE4\xBB\xB6\xE3\x80\x82\",\"timestamp\":1591689884,\"level\":3}"”}


raw message has no invalid chars

some message can parse, some can not

I search the question for all day,still not work

I am sure , the field short_message of gelf is the reason why I get this error,in other language encoding,short_message will cut off full_message in somewhere that made parse error,it suppose a bug

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.