Issue with Timestamp in GELF Input

Graylog Central (peer support)
, , , ,
1

We have got an issue with showing Messages correctly in the search.
It only shows messages when I select “Search in all messages”. The
messages it shows cannot be sorted on the timestamp, I mean you can
click it and it does something, but the messages are not sorted
properly.
There is no histogram data to show, no matter what I search or gets
displayed in my search results. It behaves like it is not able to
extract the Timestamp although Timestamps are displayed.

All Timezones are set to Berlin, User, Server.

User admin:
2017-05-08 11:14:20 +02:00
Your web browser:
2017-05-08 11:14:20 +02:00
Graylog server:
2017-05-08 11:14:20 +02:00

Recalculating Indices did not help at all.

Searching with absolute time values instead of relative ones does not show results.

My only connector right now is a GELF one, Windows System is sending messages with nxlog.

graylog.PNG983×640 49.9 KB

Has anyone encountered stuff like this or any lead on what to check for this odd behavior?
Is there a special instruction for extractors on the timestamp?
The Log that gets transmitted is a Windows Syslog.

2

Hej @MatthiasStukenberg

did you checked the messages you have already in Graylog and what timestamp they show?

Did you have the correct timestamp as part of the message? If yes, you can extract that and copy over to the timestamp field with a pipeline.

regards
Jan

3

Hey jan, thanks for your quick reply.

The timestamp I get is like this: 2017-05-05T12:06:59.000Z
When I see the messages via search, from my point of view, the timestamp does look fine, although I am a newb to graylog.

Kind regards,
Matthias

4

Hej Matthias,

it should work. if you receive messages are the timestamps the current ones?

Did you use nxlog for a specific reason or did you just follow some tutorial that suggest that? Did you already know our blog posting how to work with windows, use sidecar and filebeat?

5

Thank you, Yes, the time stamps in graylog match the ones on the windows system. As you see in the screenshot they do look pretty good…
No I haven’t seen the blog entry yet, but I will definitively give it a try! My current setup has been suggested by some youtube video, so I fiddeled around until I got it working, and it works somehow, since it is receiving messages.
Will report back tomorrow…

6

My follow up on this one: My Beats input (winlogbeat) shows me the same issues, no Histogram data, not able to sort on timestamp, though the timestamps look fine. Any directions what I can check for msiconfiguration?
Edit: one noticcable change: winlogbeat shows me milliseconds, the Gelf one did not.
Edit2: Timestamp in message for example:
2017-03-25T18:53:59.751Z

7

Hey @jan , may I ask once more for your assistance? Does this look like an elasticsearch issue to you? Is there any obvoious next step I should do for troubleshooting?

8

Hej @MatthiasStukenberg

something along the way of the message might modify the message. My troubleshoot would be to look at each step:

  • the raw messages
  • the extractors on the input
  • the pipelines
  • how the message is save in elasticsearch

to find where the timestamp modification is made and what is not working correctly. When not sitting at the environment to look at everything it is like asking a :crystal_ball:

9

I ran a tool from https://github.com/elastic/es2unix and did a search on my elasticsearch, the output looks like:

[root@adsvlog01 bin]# es -u http://10.3.1.63:9200 search -v "2017-05-15" timestamp
score   index     type    id                                   timestamp
0.91248 graylog_0 message ba292791-3465-11e7-8fbd-00505688c2c9 ["2017-05-09 03:15:1
0.91248 graylog_0 message 202581e2-35f8-11e7-8fbd-00505688c2c9 ["2017-05-11 03:15:4
0.91248 graylog_0 message 994c03e1-339c-11e7-9f42-00505688c2c9 ["2017-05-08 03:15:3
0.91248 graylog_0 message 98ab5710-339c-11e7-9f42-00505688c2c9 ["2017-05-08 03:15:3
0.89695 graylog_0 message 2da95b10-346e-11e7-8fbd-00505688c2c9 ["2017-05-09 04:15:4
0.89695 graylog_0 message ae534c82-35ef-11e7-8fbd-00505688c2c9 ["2017-05-11 02:15:1
0.89695 graylog_0 message b21bed40-35ef-11e7-8fbd-00505688c2c9 ["2017-05-11 02:15:2
0.84379 graylog_0 message 0c36d530-33a5-11e7-9f42-00505688c2c9 ["2017-05-08 04:15:5
0.77149 graylog_0 message 5dde8577-364f-11e7-8fbd-00505688c2c9 ["2017-04-15 05:15:0
0.77149 graylog_0 message 5dde8579-364f-11e7-8fbd-00505688c2c9 ["2017-04-15 05:15:0
 Total: 138547

Does this suggest that elasticsearch handels the timestamp internally wrong?

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.