How to correct syslog input that is missing timezone data?

[bharder]

FortiGate firewalls send syslog data without a timezone field. The date and time are correct, but since there is no timezone listed Graylog assumes it is UTC, and it converts the timestamp.

Here is an example of the message data the FortiGate sends:

date=2020-06-06 time=16:12:01

Here is the timestamp after Graylog has ingested it:

2020-06-06 11:12:01 -05:00

I’ve created a pipeline with the following rule:

rule "fix_timestamp"
when
  has_field("timestamp")
then
	let incorrect_timestamp = to_string($message.timestamp);
	let correct_date = substring(incorrect_timestamp, 0,11);
	let correct_time = to_string($message.time);
	let concat_date_time = concat(correct_date,correct_time);
	let corrected_timestamp = to_date(concat_date_time, "yyyy-MM-dd HH:mm:ss", "America/Chicago");
    set_field("timestamp", corrected_timestamp);
end

I’m not getting any errors, but logs in the stream don’t seem to be updating.

What am I doing wrong?
Is there a better way to do this?

Edit:
After looking at the raw syslog input using tshark, the FortiGate is sending timezone data, so I’m even more confused now.

date=2020-06-06 time=17:53:41 tz="-0500"

I edited my pipeline rule to this, just to see if I could get it to save the data to a new field, but I’m still not getting anything.

rule "fix_timestamp"
when
    true
then
    set_field("timestamp_custom", parse_date(concat(concat(to_string($message.date)," "),to_string($message.time)), "yyyy-MM-dd hh:mm:ss", "America/Chicago"));
end

Even trying a rule that doesn’t do anything other than set a custom field, isn’t working. New logs coming in the stream are not showing up with the “pipetest” field I’m trying to generate.

rule "fix_timestamp"
when
    true
then
    set_field("pipetest","rule processed");
end

But when I run the pipeline simulation it does show the rule is processing correctly.

Simulation results
These are the results of processing the loaded message. Processing took 672 µs.

Changes in original message 1b6e78e0-a856-11ea-bf02-b8ac6f83164c
Added fields
pipetest
rule processed

[bharder]

I actually got a working pipeline rule, but it seems to only be working in the pipeline simulation.

rule "fortigate timestamp"
when
    (has_field("tz") && has_field("date") && has_field("time")) 
then
    let build_message_0 = concat(to_string($message.date), " ");
    let build_message_1 = concat(build_message_0, to_string($message.time));
    let new_timestamp = parse_date(value:to_string(build_message_1), pattern:"yyyy-MM-dd HH:mm:sss", timezone:"America/Chicago");
    set_field("timestamp", new_timestamp);
end

---

Simulation results
These are the results of processing the loaded message. Processing took 740 µs.

Changes in original message 682fc281-a872-11ea-bf02-b8ac6f83164c
Mutated fields
timestamp
~~2020-06-06T17:53:41.000Z~~
2020-06-06T22:53:41.000Z

I don’t understand why it works on the Sim but doesn’t seem to apply on the stream though.

[bharder]

Finally fixed it.

I was running the pipeline on the stream I had built specifically for the firewall, but I guess that was out of order in the way the message are processed. I changed the pipeline to connect to “All Streams” instead, and it started working.

As I understand it, the problem was the message processing for routing to streams happens after the pipeline processing (at least in default config from my install), so it wasn’t receiving any stream data when I had it connected to the specific stream for the firewall.