Pipeline rule not working on rewrite of timestamp

[harryw]

Hi.

I’m having problems with a pipeline rule.
The rule is ;
rule “replace Graylog Timestamp for log messages”
when
has_field(“c-ip”)
then
let new_timestamp = parse_date(to_string($message.log_timestamp), “yyyy-MM-dd HH:mm:ss”);
set_field(“timestamp”, new_timestamp);
end
when
(has_field(“tags”) AND (to_string($message.tags) == “gra”))
then
let new_timestamp = parse_date(to_string($message.log_timestamp), “yyyy-MM-dd HH:mm:sss,SSS”);
set_field(“timestamp”, new_timestamp);
end

The first part of the works perfectly and rewrites the timestamp field for an IIS log record.
The second part doesn’t rewrite the timestamp for a non IIS log record.

The log record in question is;
EventReceivedTime
2018-08-31 09:45:52
FileName
d:\tablogs\jvgrs security\20180831.01.Security.RAWQGRA110V.log
SourceModuleName
5ac6e906e3397903966e288b
SourceModuleType
im_file
SourceName
ozchase
collector_node_id
rawqgra110v
file
20180831.01.security.rawqgra110v.log
level
6
log_timestamp
2018-08-31 09:45:51,908
message
2018-08-31 09:45:51,908 [19] INFO UNK0013O SecurityServiceServer - JVGRS.Security.Wcf.SecurityService/GetPublicKey End
size
119
source
rawqgra110v.rwwaq.com.au
tags
gra
timestamp
2018-08-31T01:45:52.000Z

Any help would be most appreciated.

Regards,

Harry W.

[jan (Jan Doberstein)]

only a single section of when ... then ... end is possible in one rule.

You need to have multiple rules if you have multiple when sections.

[harryw]

Ok, I have split the rule into two, IIS and Tablog.
The IIS rule is working perfectly.
The Tablog rule isn’t.
Here is the message for the rule that isn’t working.

2018-09-04 07:57:27.000	rawqgra110v.rwwaq.com.au
2018-09-04 07:57:27,310 [12] INFO UNK0013O SecurityServiceServer - JVGRS.Security.Wcf.SecurityService/GetPublicKey End

1cb83812-afd5-11e8-b3fa-005056aa4a6b

Received by : appliance-gelf-tcp on bdfcd88d / rawqclm121v.rwwaq.com.au

Stored in index : graylog_627

Routed into streams : * All messages

• jvgrs

EventReceivedTime : 2018-09-04 07:57:27

FileName : d:\tablogs\jvgrs security\20180904.01.Security.RAWQGRA110V.log

SourceModuleName : 5ac6e906e3397903966e288b

SourceModuleType : im_file

SourceName : ozchase

collector_node_id : rawqgra110v

file : 20180904.01.security.rawqgra110v.log

level : 6

log_timestamp : 2018-09-04 07:57:27,310

message : 2018-09-04 07:57:27,310 [12] INFO UNK0013O SecurityServiceServer - JVGRS.Security.Wcf.SecurityService/GetPublicKey End

size : 119

source : rawqgra110v.rwwaq.com.au

tags : gra

timestamp : 2018-09-03T23:57:27.000Z

Here is the rule.
rule “replace Graylog Timestamp for Tablog messages”
when
(to_string($message.SourceName) == “ozchase”)
then
let new_timestamp = parse_date(to_string($message.log_timestamp), “yyyy-MM-dd HH:mm:ss,SSS”);
set_field(“timestamp”, new_timestamp);
end

Also a matching message is ‘removed’ from Graylog when the rule is applied to the pipeline.
When I delete the rule from the pipeline the matching messages are in Graylog again.

[harryw]

Ok, a further update.
The message didn’t ‘disappear’ but was moved ‘forward’ in time (and the rule worked).
So, it looks like a time region code needs to be applied to the message.
I’m in Perth, Western Australia (maybe WST).
Any idea’s ?
Regards,

Harry W.

[harryw]

Also the message went forward 9 hours.

[harryw]

A further update.
I got the rule working but replacing the parse_date parameters with …
let new_timestamp = parse_date(to_string($message.log_timestamp), “yyyy-MM-dd HH:mm:ss,SSS”, “AU”, “Australia/Perth”);