Trouble parsing FortiGate CSV logs in Graylog using pipeline rules

How Are We Doing? Suggestion Box
1

Hi everyone!

[Context]

I’m working on ingesting FortiGate firewall logs into Graylog, and I’m having trouble parsing them when they are sent as CSV logs.

I’ve read that FortiGate logs can be sent to Graylog in two main ways:

  • Native FortiGate → Syslog → Graylog Syslog Input: In this case, fields are automatically extracted and everything works as expected.

  • Manually sending logs (CSV logs via file replay or a custom input): In this case, Graylog only sees a raw message string, and I need to parse the fields manually.

I’m currently working with the second approach, and the logs I’m working with look like this:

"itime=1765077471","date=2025-12-07","time=00:17:51","devid=FG5H0E5818900220","type=event","subtype=vpn","level=error","action=negotiate"

[The problem]

I’ve seen some ways to ingest these logs and found content packs that create extractors responsible for extracting specific fields. However, I would like to have something capable of identifying the fields by itself.

This is really important for me because I have different log sources, and they may contain different fields. I don’t want to specify or hardcode them every time.

[What I tried]

Since hard-coded extractors didn’t work well for my use case, I tried processing the message using a pipeline rule. After some research on the available functions, I used the following rule (and a few variations of the same idea):

rule "fortigate_csv_split"
when
  has_field("message")
then
  let kvs = key_value(
    value: to_string($message.message),
    delimiters: ",",
    kv_delimiters: "=",
    trim_value_chars: "\"",
    allow_dup_keys: true
  );

  set_fields(to_map(kvs));
end

This rule didn’t work in either the rule simulation or during real CSV ingestion. I tested both cases because I wasn’t completely sure whether my testing setup was correct. For the rule simulation, I used the same log example shown in the context section, using the “simple message” option.

I’m also a bit concerned because I’ve read about people having issues with edge cases, for example when a field value (a string) contains another string or special characters. I would like my solution to handle these cases correctly.

Thanks for reading!

TL;DR I’m ingesting FortiGate CSV logs into Graylog (not via native syslog) and trying to automatically extract key=value fields without hardcoding them. I attempted to use a pipeline rule with the key_value() function, but it doesn’t work correctly. I’m looking for a robust way to parse FortiGate CSV logs that handles quoted values and edge cases without relying on hardcoded extractors.

2

Hi! I found the solution to my own problem. The thing is, the rules’ language troubles with quotes. I tried using the different types (simple quote, double quote, etc), but I realized I don’t need them at all. So, the final solution was ignoring them.

Therefore, my final rule is:

rule "fortigate_csv_split"
when
  true
then
  let kvs = key_value(
    value: replace(to_string($message.message),"\""),
    delimiters: ",",
    kv_delimiters: "=",
    ignore_empty_values: false
  );
  set_fields(kvs);
end
3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.