Fortigate graylog modifies timestamp?


(Tuumke) #1

Hi,

We have 2 Graylog servers running. 1 for production firewall with a manuall installation (Graylog 2.0.2 (4da1379) on VC1LOG001 (Oracle Corporation 1.8.0_77 on Linux 4.2.0-42-generic)) and 1 new deployed via OVF Graylog 2.4.5+8e18e6a on VC1LOG002 (Oracle Corporation 1.8.0_172 on Linux 4.4.0-127-generic)).

The timestamp on the old 2.0.2 graylog machine is fine.
timestamp = 2018-06-27T11:24:08.436Z
On the new one, it’s also good
timestamp = 2018-06-27T13:25:46.000Z

But, on the VC1LOG001 we see the Graylog timestamp as is, so 2018-06-27 11 : 24 : 08 but on VC1LOG002 it just keeps adding 2 hours so it would become 2018-06-27 15 : 25 : 46

graylog-ctl timezone is set to Europe/Amsterdam.

ubuntu@VC1LOG002:~ timedatectl Local time: Wed 2018-06-27 13:27:32 CEST Universal time: Wed 2018-06-27 11:27:32 UTC Timezone: Europe/Amsterdam (CEST, +0200) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: yes Last DST change: DST began at Sun 2018-03-25 01:59:59 CET Sun 2018-03-25 03:00:00 CEST Next DST change: DST ends (the clock jumps one hour backwards) at Sun 2018-10-28 02:59:59 CEST Sun 2018-10-28 02:00:00 CET ubuntu@VC1LOG002:~

How do get the timestamp from the log message to be displayed as graylog timestamp? Because now i can’t use the “Past 5 minutes”

From the “overview” page of both servers:
VC1LOG001
User admin: 2018-06-27 13:32:08 +02:00
Your web browser: 2018-06-27 13:32:08 +02:00
Graylog server: 2018-06-27 13:32:08 +02:00
VC1LOG002:
User admin: 2018-06-27 13:33:11 +02:00
Your web browser: 2018-06-27 13:33:11 +02:00
Graylog server: 2018-06-27 13:33:11 +02:00

-edit-
Found Incorrect Timestamp on logs recieved SYSLOG UDP
But am not sure how to edit this…

-edit2-
Would this get me a correct timestamp:
rule “fortigate timestamp”
when
has_field(“devname”)
then
let build_message_0 = concat(to_string($message.date), “T”);
let build_message_1 = concat(build_message_0, to_string($message.time));
let build_message_2 = concat(build_message_1, “Z”);
let new_timestamp = parse_date(build_message_2, “yyyy-MM-dd HH:mm:ssZ”);
set_field(“timestamp”, new_timestamp);
end


(Jan Doberstein) #2

I would try your pipeline rule - what can happen if that does not work?


(Tuumke) #3

I did but it’s not working :frowning:


(Jan Doberstein) #4

any errors in the Graylog log?

Does it work if you write it in a different field (not timestamp) ?

Did you tried the to_date() surrounding the new timestamp?


(Tuumke) #5

I tried this:
rule “fortigate timestamp”
when
has_field(“devname”)
then
let build_message_0 = concat(to_string($message.date), “T”);
let build_message_1 = concat(build_message_0, to_string($message.time));
let build_message_2 = concat(build_message_1, “Z”);
let new_timestamp = parse_date(build_message_2, “yyyy-MM-dd HH:mm:sssZEurop/Amstedam”);

set_field("timestamp_test", new_timestamp);

end
But i dont see a field timestamp_test

-edit-
Couldnt i add a static field for Timezone or something to the input?


(Jan Doberstein) #6

I use the following rule to correct the timstamp on cisco devices

rule "cisco (3.1) correct timestamp IOS"
// we want to create ISO8601 Timestamps
// make 'Feb 15 2015 13:33:22.111 UTC' ISO8601
when
    has_field("cisco_message") AND
    has_field("log_date") AND
    grok(pattern: "%{MONTH} %{MONTHDAY} %{YEAR} %{TIME}", value:to_string($message.log_date)).matches == true
then
    let time = parse_date(value:to_string($message.log_date), pattern:"MMM dd yyyy HH:mm:ss.SSS", timezone:"UTC");
    set_field("timestamp",time);
end

Details on that in the posting about: https://jalogisch.de/2018/working-with-cisco-asa-nexus-on-graylog/

Maybe this can help you to find a solution. In addition you already commented on the feature issue that would be one way to solve your issue https://github.com/Graylog2/graylog2-server/issues/3853


(Tuumke) #7

Timestamp is still +2 and timestamp in the logmessage still has Z behind it.
afbeelding

Using this now:
rule “fortigate timestamp”
when
has_field(“devname”) && has_field(“date”) && has_field(“time”)
then
let build_message_0 = concat(to_string($message.date), " ");
let build_message_1 = concat(build_message_0, to_string($message.time));
let new_timestamp = parse_date(value:to_string(build_message_1), pattern:“yyyy-MM-dd HH:mm:sss”, timezone:“Europ/Amstedam”);
set_field(“timestamp”, new_timestamp);
end
In which logfiles should i look?


#8

I think, the problem is the processor order. Is this case, the pipeline processor probably musst works before the Message Filter Chain


(Tuumke) #9

I think you are referring to the stages? Because right now, it’;s in stage 1.
-edit-
Noticed a type in the rule.
I modified to a stage 0 that checks for field
stage 1 the tries to apply the timestamp changes. No luck so far…


#10

No, I mean this

You find it under the Configurations

In this case i think, you have to change it to 1. Pipeline Processor and 2. Message Filter Chain

http://docs.graylog.org/en/2.4/pages/pipelines/usage.html


(Tuumke) #11

I thought i’d try something else. I add a static field “timestamp_test” with value “null” to the input.
Then i changed my pipeline rule to modify that field and this works.


But timestamp_test is same as timestamp but the graylog timestamp on the left is still +2 hours…

-edit-
Processing order:


#12

ok, then that’s not the problem


#13

I’m wondering why your pipeline works, because the docs says:

On the Configurations page, you need to enable the Pipeline Processor message processor and, if you want your pipelines to have access to static fields set on inputs and/or fields set by extractors, set the Pipeline Processor after the Message Filter Chain.

That’s my experience, too

EDIT:

Now I think I know that the pipeline works in spite of the order: You are not accessing the value of a static field.


(Konrad Merz) #14

The problem here is related to this bug:

Where your right timestamp will not be renderd in to your timezone but keept in UTC.
This bug was addressed in 3.0:


(Tuumke) #15

Okai, and how do i get 3.0?
https://www.graylog.org/downloads only shows 2.4.5
Or can i do these changes manually?


(Jochen) #16

You’ll wait until it’s released.


(Jan Doberstein) #17

you might notice that this is a DISPLAY fix. The timezone of the messages is correct always - the full time.

Taking your picture from Fortigate graylog modifies timestamp?

The first timestamp is translated to the Users local timezone (set in the profile for normal user) which is for the root user whatever is written to the configuration. The second field just displays the time as it is saved in the Elasticsearch. Which is UTC.

That will be fixed in 3.0, the display of the timestamps will be unified.


(system) #18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.