@timestamp field format in Graylog GUI Search doesn't show as UTC ISO8601 sent by filebeat beat

[armesh (Armesh Singh)]

Environment:

• Ubuntu 22.04.2 LTS
• Graylog 5.0.6
• Filebeat 8.7.0

Incident:

I made an input to receive logs from Filebeat.
Filebeat is picking telegram*.txt

Filebeat sent over the log at 2023-04-28 12:26:27 PM GMT +8.
Filebeat ONLY sends @timestamp in UTC timezone. In my case it should be
“@timestamp”:“2023-04-28T04:26:27.729Z”

However in Graylog GUI I keep seeing 2023-04-28 12:26:27.738 in @timestamp field.

This made me think Filebeat is sending faulty @timestamp.
After debugging, I found Filebeat is indeed passing correct “@timestamp”:“2023-04-28T04:26:27.729Z” to Graylog.

Now I suspected Graylog is secretly modifying @timestamp value by adding 8 hours and changing its format (Timezone settings for my Graylog GUI is TimezoneAsia/Kuala_Lumpur, which is GMT +8).
After lots wasted time I found the original @timestamp from Filebeat is preserved and can be seen when I export the search results out of Graylog GUI in CSV format.
CSV Export Picture

This is insanely confusing and time wasting.
Is there a way to make @timestamp show exactly what was captured in original form in Graylog GUI Search?

[gsmith]

Hey @armesh

Yes i Know it can get frustrating, and no Graylog is not secretly adjusting your timestamp.
couple things you can check.

1. Graylog configuration file setting for time zone is correct.root_timezone = Asia/Kuala_Lumpur
2. Under users and Teams check the user logged on has the correct settings for Timezone.

Just so you know this issue been around for a while , you can check it out here

[armesh (Armesh Singh)]

I see, that sucks.
I think a proper Log Management System should show the exact value captured from the log fields, instead of decorating it as something else.

Almost all the posts in this forum are discussing on their problems parsing their log timestamp correctly in Graylog, which is not my issue.

In my case timestamp is parsed correctly by Graylog.
But @timestamp value is displaying in different format than what the original UTC ISO8601 datetime sent by filebeat
But upon csv exporting the search results I can see the value sent by filebeat in @timestamp field in orignal UTC ISO8601 datetime.

For other readers.
timestamp field is generated when Graylog stamp the time on your logs.
@timestamp field is value sent by Beats to Graylog with all other fields.

I think I will open an issue for this in GitHub.
Its not appropriate for LMS to behave this way.

[gsmith]

Hey @armesh

Sorry your having hard time, I double checked my lab and it seams I do not have this issue. I have not parsed timestamp field/s nor create any piepline/extractors. Im not sure if you know this but Elasticsearch/Opensearch defaults to UTC, So depeneding on what timezone your in, you may have to adjust or recheck date/time on nodes/devices that are sending or receiveing logs. Thats all i have for ya.

This issue with wrong timestamp is not limited to Graylog as shown here

[armesh (Armesh Singh)]

You are having the same issue as me.
The @timestamp field in your Graylog GUI shows 2023-04-28 21:30:16.726

But that is not the string that filebeat sent to you.
Try pushing filebeat output to file, then open the file and see what string value is in @timestamp

output.file:
  path: "/opt"
  filename: filebeat_testing

@timestamp is a ISO8601 datetime string, in UTC timezone.

Graylog GUI is not displaying it as such, but modifying it to other format in the GUI display.

I think its ok for Graylog to format the green marked value in human friendly date format converted into your user account timezone.
But any display touch up on the blue marked @timestamp value is a big no for a LMS.

[gsmith]

Hey @armesh

Ok I see what your saying now. I got it.

@armesh

Filebeat sent over the log at 2023-04-28 12:26:27 PM.
Filebeat ONLY sends @timestamp in UTC timezone. In my case it should be
“@timestamp”:“2023-04-28T04:26:27.729Z”

I was going off this statement because the difference in time.

[armesh (Armesh Singh)]

Ok I probably figured why it is behaving this way.
Graylog identifies @timestmap field as date (instead of String), so that is why it is auto formatting it into other format.

Is there a way to disable this?

[gsmith]

@armesh

Is there a way to disable this?

I havent tried, not 100% sure.

[gsmith]

@armesh

so that is why it is auto formatting it into other format.

yes, depending on what you want to do.

You can check that link out , there are multipe way to adjust timestamp.

@gsmith

, you can check it out here

EDIT: I forgot to add, so if you want to change date --> string you need to go into Elasticsearch/Opensearch find you @timestamp field, a custom template is perfered.

Example:

{
  "winlogbeat-2023.04.27": {
    "mappings": {
      "@timestamp": {
        "full_name": "@timestamp",
        "mapping": {
          "@timestamp": {
            "type": "date"
          }
        }
      }
    }
  }
}

TBH it will take some work.