Timestamp Replacement Issues (Ingestion timestamp vs log timestamp)

[nom666nom]

Essentially I’m trying to replace the ingestion timestamp with the actual timestamp in the logs. I’ve read many of the posts about this but I don’t seem to be making any progress.

As you can see in the screenshot below the Timestamp(far left) and timestamp(in the middle) are the same and are the time of ingestion, not the time of the log. I’ve created an extractor to copy the timestamp from the log and put it into the timestamp_orig(far right). As you can see in the screenshot.

I’ve already tried to copy/replace in the extractor to the timestamp field but that doesn’t change the timestamps.

I’ve then tried to make a pipeline to do it.

Pipeline Code:

rule “parse right timestamp”
when
true
then
let new_date = parse_date(to_string($message.timestamp_orig));
set_field(“timestamp”, new_date);
end

That doesn’t seem to be doing anything to my timestamps.

I feel like I’m missing something fairly simple and help would be much appreciated.

[nom666nom]

I did some troubleshooting and I can confirm that the pipeline is processing my messages, just isn’t doing what it’s supposed to do.

I change my pipeline rule to set a static value for a field.

rule “parse right timestamp”
when
has_field(“timestamp_orig”)
then
let new_date = parse_date(to_string($message.timestamp_orig));
set_field(“beat_hostname”, “taco”);
end

When I just change the set_field to the variable it reverts back the hostname and not the time stamp.

rule “parse right timestamp”
when
has_field(“timestamp_orig”)
then
let new_date = parse_date(to_string($message.timestamp_orig));
set_field(“beat_hostname”, new_date);
end

This leads me to believe that the parse_date is that part not working.

[nom666nom]

Success. The gl2_processing_error field was useful in figuring out what was going on since I wasn’t seeing anything in the logs.

For rule ‘parse right timestamp’: In call to function ‘parse_date’ at 5:19 an exception was thrown: Invalid format: “2019-04-05T19:08:08.513458-07:00” is malformed at “-07:00”

Basically this pointed to a format issue and everything in the timestamp needs to be accounted for in the parse cmd.

The following worked for the timestamps that rsyslog was forming.

rule “parse right timestamp”
when
has_field(“timestamp_orig”)
then
let new_date = parse_date(to_string($message.timestamp_orig), “yyyy-MM-dd’T’HH:mm:ss.SSSSSSZ”);
set_field(“timestamp”, new_date);
end

Hopefully this helps someone else.