@Joel_Duffield I have a much better understanding of the bug now (and I’ve updated the GitHub issue accordingly). I just don’t know if the bug is in the FortiGate firmware or Graylog. I suspect it is more likely in Graylog.
CEF messages are parsed correctly by Graylog over a CEF UDP input when a FortiGate firewall is configured to send CEF formatted logs over UDP. When the configuration is changed to send CEF logs over a TLS connection to a Graylog CEF TCP input, the connection is successful, and bytes in and bytes out are shown, but the message count remains at 0.
The Graylog server.log shows that parsing of the CEF message is failing because multiple CEF messages are being treated by the Graylog CEF parser as the timestamp of a single CEF message.
2023-04-15T16:03:32.467+01:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=b0211ff0-db9e-11ed-b17c-0242c81e95c5, messageQueueId=1989011, codec=CEF, payloadSize=332981, timestamp=2023-04-15T15:03:32.463Z, remoteAddress=/REDACTED9:22867} on input <643862a851727d49a5b23ea6>.
2023-04-15T16:03:32.468+01:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=b0211ff0-db9e-11ed-b17c-0242c81e95c5, messageQueueId=1989011, codec=CEF, payloadSize=332981, timestamp=2023-04-15T15:03:32.463Z, remoteAddress=/REDACTED9:22867}
java.lang.IllegalStateException: Could not parse timestamp. '705 <190>Apr 15 15:03:09 gateway CEF:0|Fortinet|Fortigate|v7.2.4|54000|utm:dns dns-query|2|deviceExternalId=REDACTED FTNTFGTeventtime=1681570989531024360 FTNTFGTtz=+0000 FTNTFGTlogid=1500054000 cat=utm:dns FTNTFGTsubtype=dns FTNTFGTeventtype=dns-query FTNTFGTlevel=information FTNTFGTvd=root FTNTFGTpolicyid=0 externalId=33799373 src=REDACTED spt=58665 FTNTFGTsrccountry=Reserved deviceInboundInterface=internal FTNTFGTsrcintfrole=lan dst=REDACTED9 dpt=53 FTNTFGTdstcountry=Reserved deviceOutboundInterface=root FTNTFGTdstintfrole=undefined proto=17 FTNTFGTprofile=default FTNTFGTsrcmac=REDACTED FTNTFGTxid=55650 FTNTFGTqname=www.google.com FTNTFGTqtype=A FTNTFGTqtypeval=1 FTNTFGTqclass=IN909 <189>Apr 15 15:03:09 gateway CEF:0|Fortinet|Fortigate|v7.2.4|54802|utm:dns dns-response|3|deviceExternalId=REDACTED FTNTFGTeventtime=1681570989531135600 FTNTFGTtz=+0000 FTNTFGTlogid=1501054802 cat=utm:dns FTNTFGTsubtype=dns FTNTFGTeventtype=dns-response FTNTFGTlevel=notice FTNTFGTvd=root FTNTFGTpolicyid=0 externalId=33799373 src=REDACTED spt=58665 FTNTFGTsrccountry=Reserved deviceInboundInterface=internal FTNTFGTsrcintfrole=lan dst=REDACTED9 dpt=53 FTNTFGTdstcountry=Reserved deviceOutboundInterface=root FTNTFGTdstintfrole=undefined proto=17 FTNTFGTprofile=default FTNTFGTsrcmac=REDACTED FTNTFGTxid=55650 FTNTFGTqname=www.google.com FTNTFGTqtype=A FTNTFGTqtypeval=1 FTNTFGTqclass=IN FTNTFGTipaddr=108.177.122.104, 108.177.122.105, 108.177.122.103, 108.177.122.106, 108.177.122.147, 108.177.122.99 msg=Domain is monitored act=pass FTNTFGTcat=41 FTNTFGTcatdesc=Search Engines and Portals1267 <189>Apr 15 15:03:09 gateway CEF:0|Fortinet|Fortigate|v7.2.4[...]<190>Apr 15 15:03:31 gateway CEF:0|Fortinet|Fortigate|v7.2.4|44546|event:system|2|deviceExternalId=REDACTED FTNTFGTeventtime=1681571011398864700 FTNTFGTtz=+0000 FTNTFGTlogid=0100044546 cat=event:system FTNTFGTsubtype=system FTNTFGTlevel=information FTNTFGTvd=root FTNTFGTlogdesc=Attribute configured duser=sean sproc=ssh(REDACTED) act=Edit FTNTFGTcfgtid=1013514248 FTNTFGTcfgpath=log.syslogd.setting FTNTFGTcfgattr=port[6514->5555]mode[reliable->udp] msg=Edit log.syslogd.setting 1350 <189>Apr 15 15:03:31'
at com.github.jcustenborder.cef.CEFParserImpl.parse(CEFParserImpl.java:162) ~[graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decodeCEF(CEFCodec.java:128) ~[graylog.jar:?]
at org.graylog.plugins.cef.codec.CEFCodec.decode(CEFCodec.java:117) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Unknown Source) [?:?]
For troubleshooting I replaced the CEF TLS input with a Syslog TCP input and kept the firewall configuration the same. Graylog received the message successfully and logged each CEF message separately as a raw syslog message, so the TLS connection between the firewall and Graylog is fine.
I’m not very familiar with CEF, but at first glance the raw log captured by the syslog input looks like a properly formatted CEF message to me.
Either the FortiGate is doing something different with the format of the CEF logs when TCP/TLS is used, or some internal Graylog process is breaking when it tries to read CEF from a TCP input, but I’m not sure which is the case.
and great work