# Fortinet CEF Formatting issues **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** pipeline-rules\ **Created:** [September 18, 2019, 8:45pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017 "2019-09-18T20:45:16Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 18, 2019, 8:45pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/1 "2019-09-18T20:45:16Z")
Hello, I’m currently forwarding Fortinet Fortigate, FortiClient, etc… logs to FortiAnalyzer and from FortiAnalyzer to Graylog in TCP CEF format. It appears there’s an issue where if one the keys in the body has a two character sub-name (e.g ad.vd=) , it doesn’t get parsed properly and gets appended to the previous key? Giving me fields like this: start = Sep 12 2019 17:20:15 ad. logver=N/A Instead of this: start = Sep 12 2019 17:20:15 ad.logver=N/A Example raw message: Sep 12 17:20:15 ems CEF:0|Fortinet|FortiClient-EMS|1.2|0| traffic|5|start=Sep 12 2019 17:20:15 ad.logver=N/A cat=traffic deviceSeverity=notice externalID=N/A dhost=N/A [ad.pcdomain=subdomain.company.com](http://ad.pcdomain=subdomain.company.com) ad.uid=N/A deviceExternalId=FCT8104243435915 ad.fgtserial=N/A ad.emsserial=FCTEMS0000000824 ad.regip=N/A shost=chrome.exe ad.srcproduct=Chrome src=10.10.100.101 spt=N/A ad.direction=outbound dst=N/A [ad.remotename=yahoo.com](http://ad.remotename=yahoo.com) dpt=443 [duser=remployee@subdomain.company.com](mailto:duser=remployee@subdomain.company.com) proto=6 in=N/A out=N/A ad.utmaction=userbrowsed ad.utmevent=webfilter ad.threat=Search ad.vd=root ad.fctver=1.0.1.0020 ad.os=cros ad.usingpolicy=RR Chromebook Student app=https request=/ ad.userinitiated=1 ad.browsetime=0.12 Found a similar issue reported here for logstash > **[Incomplete parsing by CEF codec?](https://discuss.elastic.co/t/incomplete-parsing-by-cef-codec/109483)** > > Taking input from some network elements that are forwarding CEF, and using the CEF codec, there's an issue where if one the keys in the body has a two character sub-name (e.g ad.vd=) , it doesn't get parsed properly and gets appended to the previous... What would be the best way to handle this? Thanks ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 18, 2019, 8:53pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/2 "2019-09-18T20:53:57Z")
This is then causing entries in /var/log/graylog-server/server.log like these 2019-09-18T16:52:32.515-04:00 WARN [MappedMessage] Could not transform CEF field [in] according to standard. Skipping. java.lang.NumberFormatException: For input string: “40 ad.” at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0\_222] at java.lang.Integer.parseInt(Integer.java:580) ~[?:1.8.0\_222] at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0\_222] at org.graylog.plugins.cef.parser.CEFMapping.convertInteger(CEFMapping.java:249) ~[graylog.jar:?] at org.graylog.plugins.cef.parser.CEFMapping.convert(CEFMapping.java:314) ~[graylog.jar:?] at org.graylog.plugins.cef.parser.MappedMessage.mapExtensions(MappedMessage.java:52) [graylog.jar:?] at org.graylog.plugins.cef.parser.MappedMessage.(MappedMessage.java:37) [graylog.jar:?] at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF(CEFCodec.java:128) [graylog.jar:?] at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:117) [graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) [graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_222] ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 19, 2019, 4:44pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/3 "2019-09-19T16:44:43Z")
I’ve also opened a ticket with Fortinet. According to the docs they should be appending with FTNTFGT and not ad. I can’t find any documentation or references to regarding appending custom keys with “ad.” Found in the FortiOS to CEF log field mapping guidelines Following is an example of the header and one key-value pair for extension from the Event VPN log in CEF: #Feb 12 10:31:04 syslog-800c CEF:0|Fortinet|Fortigate|v5.6.0|37127|event:vpn negotiate success|3|FTNTFGTlogid=0101037127 ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 19, 2019, 11:46pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/4 "2019-09-19T23:46:37Z")
According to Fortinet support appending ad. is by design. Anyone have any ideas on how to go about getting around this? ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 12:01am UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/5 "2019-09-20T00:01:10Z")
@jochen, would appending these custom key values to the equivalent of the CEFMapping.java in current Graylog fix this? > ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [September 20, 2019, 1:28pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/6 "2019-09-20T13:28:57Z")
you might want to use the processing pipeline rules to parse the log messages and not use the CEF parser. That SHOULD give the you power to work around that. As the CEF Parser is only working with messages that follow the rules while you can work around some glitches with the processing pipelines. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 1:44pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/7 "2019-09-20T13:44:02Z")
Thanks @jan. Would I just use a raw TCP input then? Any chance you have any brief example of what such a pipeline rule would look like? Still wondering about the question above? If that’s an option I might create a separate input and submit a pull request. Or create a plugin. I just haven’t had time to look into the code yet. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [September 20, 2019, 1:56pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/8 "2019-09-20T13:56:03Z")
sorry did not have the time to look in detail into this. But the following issue is showing how you can parse broken trend micro with the processing rules: > that might give you already enough to work on your issue. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 2:20pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/9 "2019-09-20T14:20:03Z")
Do you recall what type of input you used? Would I just use a raw input instead of a cef input? Thinking so since the cef input is catching the error before I can run a pipeline on it. ---
**Author:** ![BlueTeamNinja](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/blueteamninja/32/11535_2.png) [@BlueTeamNinja](https://community.graylog.org/u/BlueTeamNinja)\ **Post date:** [September 20, 2019, 3:33pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/10 "2019-09-20T15:33:57Z")
That’s exactly what he is saying. CEF Input basically means RAW INPUT with pre-built parsers. In this case, your parsers aren’t handling your messages properly - so you step back to RAW and create your own - using a Pipeline processing rule. It gets easier the more you do it. Some Grok and K/V parsing will handle that message no problem at all. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 7:36pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/11 "2019-09-20T19:36:13Z")
Thanks @jan and @BlueTeamNinja. Looks like this is going to work with little to no modification. Just needs to test across all the devices. ``` rule "type fortinet-faz-fgt-raw-cef-test" when has_field("rf_input_source") && to_string($message.rf_input_source) == "pure_cef_fgt" then let result = regex(("(CEF:.*)$"),to_string($message.message)); set_field("pure_cef", result["0"]); let message=split("\\|", to_string($message.pure_cef)); // split parts out of CEF set_field("device_vendor", message[1]); set_field("device_product", message[2]); set_field("device_version", message[3]); set_field("device_event_class_id", message[4]); set_field("name", message[5]); set_field("severity", message[6]); set_field("device_message", message[7]); //parse k-v message let kv = key_value(value: to_string(message[7]), trim_value_chars: "\"", trim_key_chars:"\"", delimiters:" ", kv_delimiters:"="); set_fields(kv); let gmsg = grok(pattern: "%{GREEDYDATA}msg=%{GREEDYDATA:message}", value: to_string(message[7]), only_named_captures: true); set_fields(gmsg); // cleanup remove_field("pure_cef"); remove_field("device_message"); remove_field("msg"); end ``` ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 7:49pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/12 "2019-09-20T19:49:22Z")
Looks like the only thing off so far is the start field. It’s showing Sep instead of Sep 20 15:32:07 Planning on converting the start time to a proper timestamp anyway. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 20, 2019, 10:25pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/13 "2019-09-20T22:25:28Z")
Any ideas on how to fix it not liking fields that have a space in them like: ad.os=Microsoft Windows 10 or ad.usingpolicy=RR Chromebook Student ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [September 21, 2019, 9:40am UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/14 "2019-09-21T09:40:53Z")
those are non default and not following any standards … what happens to a message with the spaces? what is the result in Graylog? Sorry I’m faster finding solutions if I can actually see something and not need my imagination to guess what it looks like. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [September 21, 2019, 4:51pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/15 "2019-09-21T16:51:07Z")
It’s taking fields like these and chopping it at the first space. It then is dropping non-keyed values until it hits the next keyed value, so the other keyed fields aren’t impacted. ad.os=Microsoft Windows 10 64-bit (build 17134) ad.usingpolicy=RR Staff start=Sep 21 2019 00:06:50 Is giving ad\_os=Microsoft ad\_usingpolicy=RR start=Sep ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [September 23, 2019, 5:44am UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/16 "2019-09-23T05:44:52Z")
do you mind opening a feature request. > **[Graylog2/graylog2-server](https://github.com/Graylog2/graylog2-server/issues)** > > Free and open source log management. Contribute to Graylog2/graylog2-server development by creating an account on GitHub. That should make it visible that the function (`kv_parser`) is working like it should - but should learn that values can contain the trim character - that your workcase is actually working in some case. I cut not give a guarantee if that will ever work. But you never know. ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [October 3, 2019, 11:45pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/18 "2019-10-03T23:45:50Z")
Looks like someone already opened a request for the main issue of ad. In the field names. I searched all over many times but didn’t find these until now. > The issue still remains that Fortinet also sends values like N/A when the value should be an integer. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [October 17, 2019, 11:45pm UTC](https://community.graylog.org/t/fortinet-cef-formatting-issues/12017/19 "2019-10-17T23:45:56Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.