Sort by Timestamp not work

[david_graylog13 (david)]

Hi.
We need to order the logs by Timestamp. please look screenshot when I sort by Timestamp filed ‍‍Ascending or descent nothing happens!!

For example In this screenshot should
Authorization was successful
before
Requset finished in 56.999 ms 200 application/json; charset=utf-8
To be placed, unfortunately, this is not the case with the Timestamp field.
Where am I wrong?
thanks a lot.

[david_graylog13 (david)]

Please help me guys…
This order is very important to me, please…
Sorry to mention you…
@jan @tmacgbay @macko003 @benvanstaveren

[macko003]

Be patient, it is a community support, noone get selary for it.
I suggest to check your elasticsearch, and the field type. Meybe it doesn’t handle the millis.
Or try to use the default timestamp field.

[david_graylog13 (david)]

I’m so sorry . you are right. Please excuse me.
I will check, thanks again.

[david_graylog13 (david)]

Hi agian
I’m using a pipeline get Timestamp from message body and save to new filed(New_Time)
my pipeline:

rule "CreateNewFormatTimestamp"
when
 from_input("5e8c0edb50401d5943c76fcf")
 then
//"Timestamp":"2020-04-14T17:26:47.6146998+04:30"
let GetTime = ($message.Timestamp);
//2020-04-14T16:39:01.214
let GetTime = substring(to_string(GetTime), 0, 23);
set_field("New_Time", parse_date(value:to_string(GetTime), pattern:"yyyy-MM-dd'T'HH:mm:ss.SSS"));
end   

Then checked Elasticsearch type field.

type is date but sort does not works yet!

Also, this field (New_Time) is not shown in the list of fields left side SORTING.

Thank you for your patience.

[macko003]

check the function with the original timestamp field.
After check the timestamp field mapping in elasticsearch.
check the date format in source field and in yours.
(just random tips)

[david_graylog13 (david)]

There was no difference…

[david_graylog13 (david)]

Another case, when i create new field with this pipeline:

rule "CreateNewFormatTimestamp"
when
 from_input("5e8c0edb50401d5943c76fcf")
 then
let GetTimeApp = ($message.Timestamp);
set_field("AOrginTimeStampApp", GetTimeApp);
end   

in Elasticsearch type field is date but in graylog (3.2.4 version) type is Unknown.

I do not understand. why is this happening?
thanks.

[macko003]

2 things
please check the mappint to the original timestamp filed.
Check it with lower letters, in my system the timestmp field written by with lowercase letters (I’m not sure it can cause problem).
+1 you can check the pipeline with debug functions.
+1 and I see you showed an index’s mapping. Are you sure the message is in this index?
+1 did you do it on a wildcarded index name, and you rotated the indices after the mapping?

[david_graylog13 (david)]

rule "CreateNewFormatTimestamp"
when
 from_input("5e8c0edb50401d5943c76fcf")
 then
let GetTime = ($message.timestamp);
let debug_message = concat("orginal timestamp is: ", to_string($message.timestamp));
debug(debug_message);
end

debug orginal timestamp:

INFO  [Function] PIPELINE DEBUG: orginal timestamp is: 2020-04-17T14:20:51.400Z

Are you sure the message is in this index? yes @macko003 i’m sure.

I use the absolute index name in API elasticsearch.

Another case again this section show correct timestamp (with millisecond) but in the field not show millisecond.

.

[tmacgbay]

Code:

@david_graylog13

let GetTime = ($message.Timestamp);

Debug:

@david_graylog13

let GetTime = ($message.timestamp);

but… you are using $message.Timestamp and debugging against $message.timestamp. If capitalization matters, and I think it does, you need to debug against the capitalized Timestamp… or change all things to lowercase timestamp.

Also - since you are working in a search that is related to fields in the message, stick with fields in the message rather than the capitalized Timestamp that looks like a Record Timestamp.

Might be wrong but that’s what it looks like to me.

[david_graylog13 (david)]

Hi @tmacgbay. you right, GetTime in post number 8, is a ‍Timestamp from the application, but after suggest @macko003 I used original timestamp then used in post number 9
let GetTime = ($message.timestamp);
so last debug it shows us original timestamp.

let GetTime = ($message.timestamp);
let debug_message = concat("orginal timestamp is: ", to_string($message.timestamp));
debug(debug_message);

Thank you for your attention

[encarta]

David, sorry to get into your thread as I’m facing the same issue.

I have similar issue with ascending type of sorting. But in my case, I sorted on timestamp It’s never sorted correctly. It’s always sorted descending.

version : v3.2.4+a407287
Takim

[david_graylog13 (david)]

Hi.
This problem still exists. Is there no solution? Is this a problem just for me and @encarta?
I want to make sure; my configuration; my timestamp; my elasticsearch; my application log or is everything else wrong or is it a problem with GaryLog; for example a bug?
thanks agian.

[konrad (Konrad Merz)]

Hi @david_graylog13, @encarta,

my best guess right now is that you have a compound value. Which means that at somepoint you ingested the timestamp field with a different type than date like string and after that elasticsearch can’t sort anymore.

Please have a look in to your field list in the sidebar and check the displayed type there:

If you find that instead of the calendar icon (which indicates date type) you find a ? then you have a compound type.

- Konrad

[david_graylog13 (david)]

Thanks, @konrad. I checked. timestamp filed is correct and type is date, even Gettime field is correct too (the second screenshot). (GetTime field separated using a pipeline).

but when sort by Timestamp not work

[jan (Jan Doberstein)]

he @david_graylog13, @macko003, @tmacgbay, @encarta

this is a known issue and it is fixed. Means in the next 3.2 release (hopeful next week) this will be gone

Persist message list search type's sort option. (Backport of #7759 for 3.2)

So this is also part on 3.3 and that was the reason @konrad did not see this in his dev. environment. But as this was “small” it slipped out of sight from me and I did not remember.