Updating correct timestamp for logs

Graylog Central (peer support)
,
1

Hi guys, I am trying to correct timestamp for O365 logs. As per my understanding O365 logs have no timezone info so graylog treating it UTC 0, infact that should be UTC -6. I am trying to get over it by writing current parse time through pipeline below:

rule “correct timestamp for logs from O365”
when
has_field(field: “timestamp”)
then
let log_timestamp = $message.timestamp;
let timestamp = $message.timestamp;

let debug_message = concat(
                first: "timestamp before changing: ",
                second: to_string(timestamp));

debug(debug_message);

let new_date = parse_date( value: to_string(log_timestamp),
                       pattern: "yyyy-MM-dd HH:mm:ss Z");
set_field("timestamp", new_date);
let debug_message = concat(
                first: "timestamp after changing: ",
                second: to_string(timestamp));

debug(debug_message);

end

Graylog does the time zone conversion but still that is not good as the difference of 6 hours. I can get debug value for before changing but after is not even showing and is there an efficient way do it. Thanks in advance.

2

Try to include also your timezone, so graylog can parse date in right way:

    let new_date = parse_date(value: to_string($message.log_timestamp), pattern:"yyyy-MM-dd HH:mm:ss Z", timezone:"Europe/Bratislava");
    set_field("timestamp", new_date);

Set timezone parameter to your real timezone.

Also check that yo use correct pattern for date time format.

3

I am running 3.3.5 version. When I open a full log - I see 1d82cb00-f3e6-11ea-9f91-000c29c43263

Timestamp

2020-09-11 04:19:17.000

full_message

{“CreationTime”:“2020-09-11T04:19:17”,

timestamp

2020-09-10T23:19:17.000Z
It looks strange to me - when I just expand the logs in stream, all 3 timestamps are the same but when opened in the full log view by clicking log name; One differs.

2020-09-11T09:18:25.152+05:00 INFO [Function] PIPELINE DEBUG: timestamp before changing: 2020-09-11T04:16:51.000+05:00 Still can not see the after change debug message,
This is the timestamp that I see in debug and as per your suggestion, I applied all 4 parameters:

rule “correct timestamp for logs from O365”
when
has_field(field: “timestamp”)
then
let log_timestamp = $message.timestamp;
let timestamp = $message.timestamp;

let debug_message = concat(
                first: "timestamp before changing: ",
                second: to_string(timestamp));

debug(debug_message);
let new_date = parse_date( value: to_string(log_timestamp),
pattern: “yyyy-MM-ddTHH:mm:ss:SSSZ”, locale: “en-US”, timezone:“America/Chicago”);
set_field(“timestamp”, new_date);
let debug_message = concat(
first: "timestamp After changing: ",
second: to_string(timestamp));
debug(debug_message);

end

Thank you for your support.

4

to add - Following are the debug logs:

2020-09-11T10:42:26.598+05:00 DEBUG [PipelineInterpreter] [92ee2551-f3f1-11ea-9f91-000c29c43263] rule correct timestamp for logs from O365 matched running actions
2020-09-11T10:42:26.598+05:00 INFO [Function] PIPELINE DEBUG: timestamp before changing: 2020-09-11T05:38:29.000+05:00
2020-09-11T10:42:26.599+05:00 DEBUG [PipelineInterpreter] Encountered evaluation error, skipping rest of the rule: In call to function ‘parse_date’ at 12:19 an exception was thrown: Invalid format: “2020-09-11T05:38:29.000+05:00” is malformed at “.000+05:00”

let new_date = parse_date( value: to_string(timestamp),
pattern: “yyyy-MM-dd’T’HH:mm:ss:sssz”, locale: “en-US”, timezone:“America/Chicago”);
set_field(“timestamp”, new_date);

Any suggestions?

5

Finally got it working - In my guess the problem was graylog was considering log time UTC 0 and changing it to my required timezone but the actual log time was UTC-5, hence 5 hours difference. have to do following for solving it.

rule “correct timestamp for logs from O365”
when
has_field(field: “timestamp”)
then
let log_timestamp = $message.timestamp;
let timestamp = $message.timestamp;

let debug_message = concat(
                first: "timestamp before changing: ",
                second: to_string(timestamp));

debug(debug_message);
let mid_date = concat(
first: to_string(timestamp),
second: “+00:00”);
let new_date = parse_date( value: to_string(mid_date),
pattern: “yyyy-MM-dd’T’HH:mm:ss.SSS+SS:SSZ”, locale: “en-US”, timezone:“Asia/Karachi”);
set_field(“timestamp”, new_date);
let debug_message = concat(
first: "timestamp After changing: ",
second: to_string(new_date));
debug(debug_message);

end

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.