Fortinet CEF Formatting issues

rfinney · September 18, 2019, 8:45pm

Hello,

I’m currently forwarding Fortinet Fortigate, FortiClient, etc… logs to FortiAnalyzer and from FortiAnalyzer to Graylog in TCP CEF format.

It appears there’s an issue where if one the keys in the body has a two character sub-name (e.g ad.vd=) , it doesn’t get parsed properly and gets appended to the previous key?

Giving me fields like this:
start = Sep 12 2019 17:20:15 ad.
logver=N/A

Instead of this:
start = Sep 12 2019 17:20:15
ad.logver=N/A

Example raw message:
Sep 12 17:20:15 ems CEF:0|Fortinet|FortiClient-EMS|1.2|0| traffic|5|start=Sep 12 2019 17:20:15 ad.logver=N/A cat=traffic deviceSeverity=notice externalID=N/A dhost=N/A ad.pcdomain=subdomain.company.com ad.uid=N/A deviceExternalId=FCT8104243435915 ad.fgtserial=N/A ad.emsserial=FCTEMS0000000824 ad.regip=N/A shost=chrome.exe ad.srcproduct=Chrome src=10.10.100.101 spt=N/A ad.direction=outbound dst=N/A ad.remotename=yahoo.com dpt=443 duser=remployee@subdomain.company.com proto=6 in=N/A out=N/A ad.utmaction=userbrowsed ad.utmevent=webfilter ad.threat=Search ad.vd=root ad.fctver=1.0.1.0020 ad.os=cros ad.usingpolicy=RR Chromebook Student app=https request=/ ad.userinitiated=1 ad.browsetime=0.12

Found a similar issue reported here for logstash

What would be the best way to handle this?

Thanks

rfinney · September 18, 2019, 8:53pm

This is then causing entries in /var/log/graylog-server/server.log like these

2019-09-18T16:52:32.515-04:00 WARN [MappedMessage] Could not transform CEF field [in] according to standard. Skipping.
java.lang.NumberFormatException: For input string: “40 ad.”
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_222]
at java.lang.Integer.parseInt(Integer.java:580) ~[?:1.8.0_222]
at java.lang.Integer.parseInt(Integer.java:615) ~[?:1.8.0_222]
at org.graylog.plugins.cef.parser.CEFMapping.convertInteger(CEFMapping.java:249) ~[graylog.jar:?]
at org.graylog.plugins.cef.parser.CEFMapping.convert(CEFMapping.java:314) ~[graylog.jar:?]
at org.graylog.plugins.cef.parser.MappedMessage.mapExtensions(MappedMessage.java:52) [graylog.jar:?]
at org.graylog.plugins.cef.parser.MappedMessage.(MappedMessage.java:37) [graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF(CEFCodec.java:128) [graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:117) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:86) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

rfinney · September 19, 2019, 4:44pm

I’ve also opened a ticket with Fortinet. According to the docs they should be appending with FTNTFGT and not ad. I can’t find any documentation or references to regarding appending custom keys with “ad.”

rfinney · September 19, 2019, 11:46pm

According to Fortinet support appending ad. is by design. Anyone have any ideas on how to go about getting around this?

rfinney · September 20, 2019, 12:01am

@jochen, would appending these custom key values to the equivalent of the CEFMapping.java in current Graylog fix this?

github.com

Graylog2/graylog-plugin-cef/blob/master/src/main/java/org/graylog/plugins/cef/parser/CEFMapping.java

package org.graylog.plugins.cef.parser;

import com.google.common.collect.ImmutableMap;
import org.joda.time.DateTime;

import javax.annotation.Nullable;
import java.math.BigInteger;
import java.util.Map;
import java.util.function.Function;

@SuppressWarnings("unused")
public enum CEFMapping {
    // CEF Key Names For Event Producers
    act("act", "deviceAction", CEFMapping::convertString),
    app("app", "applicationProtocol", CEFMapping::convertString),
    c6a1("c6a1", "deviceCustomIPv6Address1", CEFMapping::convertIPv6Address),
    c6a1Label("c6a1Label", "deviceCustomIPv6Address1Label", CEFMapping::convertString),
    c6a2("c6a2", "deviceCustomIPv6Address2", CEFMapping::convertIPv6Address),
    c6a2Label("c6a2Label", "deviceCustomIPv6Address2Label", CEFMapping::convertString),
    c6a3("c6a3", "deviceCustomIPv6Address3", CEFMapping::convertIPv6Address),

This file has been truncated. show original

jan · September 20, 2019, 1:28pm

you might want to use the processing pipeline rules to parse the log messages and not use the CEF parser.

That SHOULD give the you power to work around that. As the CEF Parser is only working with messages that follow the rules while you can work around some glitches with the processing pipelines.

rfinney · September 20, 2019, 1:44pm

Thanks @jan. Would I just use a raw TCP input then?

Any chance you have any brief example of what such a pipeline rule would look like?

Still wondering about the question above? If that’s an option I might create a separate input and submit a pull request. Or create a plugin. I just haven’t had time to look into the code yet.

jan · September 20, 2019, 1:56pm

sorry did not have the time to look in detail into this. But the following issue is showing how you can parse broken trend micro with the processing rules:

that might give you already enough to work on your issue.

rfinney · September 20, 2019, 2:20pm

Do you recall what type of input you used? Would I just use a raw input instead of a cef input?

Thinking so since the cef input is catching the error before I can run a pipeline on it.

BlueTeamNinja · September 20, 2019, 3:33pm

That’s exactly what he is saying.

CEF Input basically means RAW INPUT with pre-built parsers.
In this case, your parsers aren’t handling your messages properly - so you step back to RAW and create your own - using a Pipeline processing rule. It gets easier the more you do it.

Some Grok and K/V parsing will handle that message no problem at all.

rfinney · September 20, 2019, 7:36pm

Thanks @jan and @BlueTeamNinja. Looks like this is going to work with little to no modification.
Just needs to test across all the devices.

rule "type fortinet-faz-fgt-raw-cef-test"
when
has_field("rf_input_source") && to_string($message.rf_input_source) == "pure_cef_fgt"    

then
let result = regex(("(CEF:.*)$"),to_string($message.message));
set_field("pure_cef", result["0"]);

let message=split("\\|", to_string($message.pure_cef));

// split parts out of CEF
set_field("device_vendor", message[1]);
set_field("device_product", message[2]);
set_field("device_version", message[3]);
set_field("device_event_class_id", message[4]);
set_field("name", message[5]);
set_field("severity", message[6]);
set_field("device_message", message[7]);

//parse k-v message
let kv = key_value(value: to_string(message[7]), trim_value_chars: "\"", trim_key_chars:"\"", delimiters:" ", kv_delimiters:"=");
set_fields(kv);

let gmsg = grok(pattern: "%{GREEDYDATA}msg=%{GREEDYDATA:message}", value: to_string(message[7]), only_named_captures: true);
set_fields(gmsg);

// cleanup
remove_field("pure_cef");
remove_field("device_message");
remove_field("msg");
end

rfinney · September 20, 2019, 7:49pm

Looks like the only thing off so far is the start field. It’s showing Sep instead of Sep 20 15:32:07

Planning on converting the start time to a proper timestamp anyway.

rfinney · September 20, 2019, 10:25pm

Any ideas on how to fix it not liking fields that have a space in them like:
ad.os=Microsoft Windows 10
or
ad.usingpolicy=RR Chromebook Student

jan · September 21, 2019, 9:40am

those are non default and not following any standards … what happens to a message with the spaces? what is the result in Graylog?

Sorry I’m faster finding solutions if I can actually see something and not need my imagination to guess what it looks like.

rfinney · September 21, 2019, 4:51pm

It’s taking fields like these and chopping it at the first space. It then is dropping non-keyed values until it hits the next keyed value, so the other keyed fields aren’t impacted.

ad.os=Microsoft Windows 10 64-bit (build 17134)
ad.usingpolicy=RR Staff
start=Sep 21 2019 00:06:50

Is giving

ad_os=Microsoft
ad_usingpolicy=RR
start=Sep

jan · September 23, 2019, 5:44am

do you mind opening a feature request.

That should make it visible that the function (kv_parser) is working like it should - but should learn that values can contain the trim character - that your workcase is actually working in some case. I cut not give a guarantee if that will ever work. But you never know.

rfinney · October 3, 2019, 11:45pm

Looks like someone already opened a request for the main issue of ad. In the field names. I searched all over many times but didn’t find these until now.

The issue still remains that Fortinet also sends values like N/A when the value should be an integer.

system · October 17, 2019, 11:45pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Replace or Trim from message field in pipeline rule Graylog Central (peer support) pipeline-rules	3	5047	September 27, 2019
FortiGate: I can get CEF logs over UDP and Syslog over TLS, but not CEF over TLS. Why? Graylog Central (peer support)	16	3328	May 2, 2023
Fortinet/Fortigate - TCP Raw Input and delimiter Graylog Central (peer support)	1	1023	September 30, 2020
Fortigate field mapping problem Graylog Central (peer support)	4	1464	March 25, 2020
Remove prefix in fieldnames Graylog Central (peer support)	4	589	July 12, 2022

Fortinet CEF Formatting issues

Related topics