Fortinet CEF Formatting issues


I’m currently forwarding Fortinet Fortigate, FortiClient, etc… logs to FortiAnalyzer and from FortiAnalyzer to Graylog in TCP CEF format.

It appears there’s an issue where if one the keys in the body has a two character sub-name (e.g ad.vd=) , it doesn’t get parsed properly and gets appended to the previous key?

Giving me fields like this:
start = Sep 12 2019 17:20:15 ad.

Instead of this:
start = Sep 12 2019 17:20:15

Example raw message:
Sep 12 17:20:15 ems CEF:0|Fortinet|FortiClient-EMS|1.2|0| traffic|5|start=Sep 12 2019 17:20:15 ad.logver=N/A cat=traffic deviceSeverity=notice externalID=N/A dhost=N/A ad.uid=N/A deviceExternalId=FCT8104243435915 ad.fgtserial=N/A ad.emsserial=FCTEMS0000000824 ad.regip=N/A shost=chrome.exe ad.srcproduct=Chrome src= spt=N/A ad.direction=outbound dst=N/A dpt=443 proto=6 in=N/A out=N/A ad.utmaction=userbrowsed ad.utmevent=webfilter ad.threat=Search ad.vd=root ad.fctver= ad.os=cros ad.usingpolicy=RR Chromebook Student app=https request=/ ad.userinitiated=1 ad.browsetime=0.12

Found a similar issue reported here for logstash

What would be the best way to handle this?


This is then causing entries in /var/log/graylog-server/server.log like these

2019-09-18T16:52:32.515-04:00 WARN [MappedMessage] Could not transform CEF field [in] according to standard. Skipping.
java.lang.NumberFormatException: For input string: “40 ad.”
at java.lang.NumberFormatException.forInputString( ~[?:1.8.0_222]
at java.lang.Integer.parseInt( ~[?:1.8.0_222]
at java.lang.Integer.parseInt( ~[?:1.8.0_222]
at org.graylog.plugins.cef.parser.CEFMapping.convertInteger( ~[graylog.jar:?]
at org.graylog.plugins.cef.parser.CEFMapping.convert( ~[graylog.jar:?]
at org.graylog.plugins.cef.parser.MappedMessage.mapExtensions( [graylog.jar:?]
at org.graylog.plugins.cef.parser.MappedMessage.( [graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF( [graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decode( [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage( [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent( [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent( [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent( [graylog.jar:?]
at [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$ [graylog.jar:?]
at [?:1.8.0_222]

I’ve also opened a ticket with Fortinet. According to the docs they should be appending with FTNTFGT and not ad. I can’t find any documentation or references to regarding appending custom keys with “ad.”

Found in the FortiOS to CEF log field mapping guidelines
Following is an example of the header and one key-value pair for extension from the Event VPN log in CEF:
#Feb 12 10:31:04 syslog-800c CEF:0|Fortinet|Fortigate|v5.6.0|37127|event:vpn negotiate

According to Fortinet support appending ad. is by design. Anyone have any ideas on how to go about getting around this?

@jochen, would appending these custom key values to the equivalent of the in current Graylog fix this?

you might want to use the processing pipeline rules to parse the log messages and not use the CEF parser.

That SHOULD give the you power to work around that. As the CEF Parser is only working with messages that follow the rules while you can work around some glitches with the processing pipelines.

1 Like

Thanks @jan. Would I just use a raw TCP input then?

Any chance you have any brief example of what such a pipeline rule would look like?

Still wondering about the question above? If that’s an option I might create a separate input and submit a pull request. Or create a plugin. I just haven’t had time to look into the code yet.

sorry did not have the time to look in detail into this. But the following issue is showing how you can parse broken trend micro with the processing rules:

that might give you already enough to work on your issue.

Do you recall what type of input you used? Would I just use a raw input instead of a cef input?

Thinking so since the cef input is catching the error before I can run a pipeline on it.

That’s exactly what he is saying.

CEF Input basically means RAW INPUT with pre-built parsers.
In this case, your parsers aren’t handling your messages properly - so you step back to RAW and create your own - using a Pipeline processing rule. It gets easier the more you do it.

Some Grok and K/V parsing will handle that message no problem at all.

Thanks @jan and @BlueTeamNinja. Looks like this is going to work with little to no modification.
Just needs to test across all the devices.

rule "type fortinet-faz-fgt-raw-cef-test"
has_field("rf_input_source") && to_string($message.rf_input_source) == "pure_cef_fgt"    

let result = regex(("(CEF:.*)$"),to_string($message.message));
set_field("pure_cef", result["0"]);

let message=split("\\|", to_string($message.pure_cef));

// split parts out of CEF
set_field("device_vendor", message[1]);
set_field("device_product", message[2]);
set_field("device_version", message[3]);
set_field("device_event_class_id", message[4]);
set_field("name", message[5]);
set_field("severity", message[6]);
set_field("device_message", message[7]);

//parse k-v message
let kv = key_value(value: to_string(message[7]), trim_value_chars: "\"", trim_key_chars:"\"", delimiters:" ", kv_delimiters:"=");

let gmsg = grok(pattern: "%{GREEDYDATA}msg=%{GREEDYDATA:message}", value: to_string(message[7]), only_named_captures: true);

// cleanup

Looks like the only thing off so far is the start field. It’s showing Sep instead of Sep 20 15:32:07

Planning on converting the start time to a proper timestamp anyway.

Any ideas on how to fix it not liking fields that have a space in them like:
ad.os=Microsoft Windows 10
ad.usingpolicy=RR Chromebook Student

those are non default and not following any standards … what happens to a message with the spaces? what is the result in Graylog?

Sorry I’m faster finding solutions if I can actually see something and not need my imagination to guess what it looks like.

It’s taking fields like these and chopping it at the first space. It then is dropping non-keyed values until it hits the next keyed value, so the other keyed fields aren’t impacted.

ad.os=Microsoft Windows 10 64-bit (build 17134)
ad.usingpolicy=RR Staff
start=Sep 21 2019 00:06:50

Is giving


do you mind opening a feature request.

That should make it visible that the function (kv_parser) is working like it should - but should learn that values can contain the trim character - that your workcase is actually working in some case. I cut not give a guarantee if that will ever work. But you never know.

Looks like someone already opened a request for the main issue of ad. In the field names. I searched all over many times but didn’t find these until now.

The issue still remains that Fortinet also sends values like N/A when the value should be an integer.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.