Hey!
I ran into a problem and can’t seem to find what’s causing it.
I installed Palo Alto content pack (https://marketplace.graylog.org/addons/cc7de2f6-e0d0-4446-ac2b-309426871055)
and ran into an indexer problem error.
MapperParsingException[failed to parse [EventDate]]; nested: IllegalArgumentException[Invalid format: "2017/08/01" is malformed at "/08/01"];
Now as it uses GROK patterns, I tested the logs, which come to Palo Alto collector, with a grok pattern tester and it works fine. When I test it in Graylog itself, it also parses log file correctly.
Error message from logs in debug mode:
2017-08-01T15:48:05.828+03:00 DEBUG [ProcessBufferProcessor] Starting to process message <a9d1c841-76b7-11e7-90b5-00505681d4c9>.
2017-08-01T15:48:05.828+03:00 DEBUG [MessageFilterChainProcessor] Applying filter [Extractor] on message <a9d1c841-76b7-11e7-90b5-00505681d4c9>.
2017-08-01T15:48:05.829+03:00 DEBUG [MessageFilterChainProcessor] Applying filter [Static field appender] on message <a9d1c841-76b7-11e7-90b5-00505681d4c9>.
2017-08-01T15:48:05.829+03:00 DEBUG [MessageFilterChainProcessor] Applying filter [Rulesfilter] on message <a9d1c841-76b7-11e7-90b5-00505681d4c9>.
2017-08-01T15:48:05.829+03:00 DEBUG [MessageFilterChainProcessor] Applying filter [StreamMatcher] on message <a9d1c841-76b7-11e7-90b5-00505681d4c9>.
2017-08-01T15:48:05.829+03:00 DEBUG [StreamMatcherFilter] Routed message <a9d1c841-76b7-11e7-90b5-00505681d4c9> to 0 streams.
2017-08-01T15:48:05.829+03:00 DEBUG [ProcessBufferProcessor] Finished processing message <a9d1c841-76b7-11e7-90b5-00505681d4c9>. Writing to output buffer.
2017-08-01T15:48:05.829+03:00 DEBUG [OutputBufferProcessor] Processing message <a9d1c841-76b7-11e7-90b5-00505681d4c9> from OutputBuffer.
2017-08-01T15:48:05.829+03:00 DEBUG [OutputBufferProcessor] Wrote message <a9d1c841-76b7-11e7-90b5-00505681d4c9> to all outputs. Finished handling.
[63]: index [graylog_18], type [message], id [a9d1c841-76b7-11e7-90b5-00505681d4c9], message [MapperParsingException[failed to parse [EventDate]]; nested: IllegalArgumentException[Invalid format: "2017/08/01" is malformed at "/08/01"];]
Example log message:
....<...<190>Aug 1 16:06:41 PA-FW1 : 1,2017/08/01 16:06:41,111111,TRAFFIC,start,1,2017/08/01 16:06:41,1.1.1.1,2.2.2.2,0.0.0.0,0.0.0.0,DMZ-TO-OUTSIDE,,,web-browsing,vsys1,outdmz,untrust,ethernet1/2,ethernet1/1.10,TO-GRAYLOG,2017/08/01 16:06:41,39768,1,37849,80,0,0,0x0,tcp,allow,359,281,78,4,2017/08/01 16:06:40,0,any,0,4001679,0x0,SE,SE,0,3,1,n/a
Grok pattern:
%{IPORHOST:source} +:* *%{BASE10NUM:Domain},%{DATE_US2:LoggedDate} +%{TIME:LoggedTime},%{NOTCOMMA:SerialNumber},%{NOTCOMMA:Type},%{NOTCOMMA:Subtype},%{NOTCOMMA:ConfigVersion},%{DATE_US2:EventDate} +%{TIME:EventTime},%{IP:SourceIP},%{IP:DestinationIP},(%{IP:NATSourceIP})?,(%{IP:NATDestinationIP})?,%{NOTCOMMA:RuleName},(%{NOTCOMMA:SourceUser})?,(%{NOTCOMMA:DestinationUser})?,(%{NOTCOMMA:Application})?,%{NOTCOMMA:VirtualSystem},%{NOTCOMMA:SourceZone},%{NOTCOMMA:DestinationZone},%{NOTCOMMA:IngressInterface},(%{NOTCOMMA:EgressInterface})?,%{NOTCOMMA:LogForwardingProfile},%{NOTCOMMA:UNWANTED},%{BASE10NUM:SessionID},%{BASE10NUM:RepeatCount},(%{BASE10NUM:SourcePort})?,(%{BASE10NUM:DestinationPort})?,(%{BASE10NUM:NATSourcePort})?,(%{BASE10NUM:NATDestinationPort})?,%{NOTCOMMA:Flags},%{NOTCOMMA:Protocol},%{NOTCOMMA:Action},%{BASE10NUM:Bytes;long},%{BASE10NUM:BytesSent;long},%{BASE10NUM:BytesReceived;long},%{BASE10NUM:Packets;long},%{NOTCOMMA:UNWANTED},%{BASE10NUM:ElapsedTime},%{NOTCOMMA:Category},%{BASE10NUM:Padding},%{BASE10NUM:Sequence},%{NOTCOMMA:ActionFlags},(%{NOTCOMMA:SourceLocation})?,(%{NOTCOMMA:DestinationLocation}),(%{BASE10NUM:UNWANTED})?,%{BASE10NUM:PktsSent;long},%{BASE10NUM:PktsReceived;long},(%{NOTCOMMA:SessionEndReason})?.*
Missing grok patterns to test at rokconstructor.appspot.com :
DATE_US2 %{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}
NOTCOMMA [^,]+
Thanks!