I have syslog input in place to manage Fortigate firewall logs.
When log message has url with “=” all goes to sh…t. Graylog then parses using key=value and soon we have over 1000 fields…
example of url-field: url: /CI0/01000170b4053790-f15ccb89-9c83-4f9d-a080-5fd73f0ace10-000000/sD7nfaNYjGkkaqRCxhU1wQmc9R1aHbs9XL2GFMpl_Lo=131
Above makes, IN ADDITION, field sD7nfaNYjGkkaqRCxhU1wQmc9R1aHbs9XL2GFMpl_Lo with value 131
url from the message-field: url="/CI0/01000170b4053790-f15ccb89-9c83-4f9d-a080-5fd73f0ace10-000000/sD7nfaNYjGkkaqRCxhU1wQmc9R1aHbs9XL2GFMpl_Lo=131"
I’ve tried using CEF-input, but it fails at the beginning to this: java.lang.IllegalStateException: Could not parse timestamp. '<143>Mar 09 2020 15:52:38'
Suggestions for the best fix?
I was thinking of using pipeline, any thoughts? I would like to fix the message, not delete all messages with url…
Tried with CEF UDP. need to modify somehow because graylog logs fill up with these now:
2020-03-11T11:45:00.576+02:00 WARN [MappedMessage] Could not transform CEF field [in] according to standard. Skipping. java.lang.NumberFormatException: For input string: "15787509737" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_242] at java.lang.Integer.parseInt(Integer.java:583) ~[?:1.8.0_242] at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_242] at org.graylog.plugins.cef.parser.CEFMapping.convertInteger(CEFMapping.java:249) ~[graylog.jar:?]
Does it try to parse it as INT, with mapping api I get type long… "in" : { "type" : "long" }
TCP -formats don’t work from Fortigate. fails parsing.
Maybe I need to try rawUDP (like now) but with Fortigates sending “syslog”. Then modify the URLs in logged messages before parsing k=v pairs (these with pipeline)…
MAYBE then I will not have graylog server logs without filling with warnings AND the number of fields remains sane.