How to create field "ts_name" and connect it to the portion of the log lines between two timestamps(start and end)?

[dmirc]

Hi,

Basically I have a big log with a lot of lines that are all connected to some of the Tests in Jenkins(for example ID: Ts4567, Ts7890, Ts1234, etc).

Now the point what I want to achieve is for example: I want to write in search ts_name: Ts4567 and then in output(Messages block in Graylog GUI), there should be listed only lines that are connected to this field, between two timestamps(start and the end).

How should I make this happen? Via Pipeline maybe? If someone can write the example I would appreciate it much. Thanks!

Let’s say the parameters are:

ID of the Test: Ts4567
Start: 2019-12-03T07:49:44,702
End: 2019-12-03T07:52:14,463

Note: Not every line consist Ts4567, so I can’t just type “Ts4567” in the search and get all the lines. I need to use timestamps and somehow connect these lines to the field, and after that search for it.

Timestamp is in message field. Look example of a log line bellow:

message
2019-12-03T07:50:43,011 TRACE o.a.k.c.p.i.ProducerBatch [kafka-producer-network-thread | producer-3] Successfully produced messages to dev_module_0_storage_priority-0 with base offset 30498.

[macko003]

Unfortunately graylog cant make connection between logs.
May be you can write script what make elasticsearch querres to find the start and and time, Than write it back to the log line.

[dmirc]

something like this wont work? (for now it doesn’t but maybe im close?)

"
rule “parse event timestamp”
when
to_date (value: “2019-12-03T07:49:50,751”) < to_date (value: $message.newtimestamp) && to_date (value: “2019-12-03T07:50:43,895”) > to_date (value: $message.newtimestamp)
then
set_field(“test_id”, “Ts5963”);
end
"

[dmirc]

This is solution:

rule “parse event timestamp and add field”
when
parse_date(to_string($message.newtimestamp),“YYYY-MM-dd’T’HH:mm:ss.SSS”) >=
parse_date(to_string(“2019-12-03T07:49:50.751”),“YYYY-MM-dd’T’HH:mm:ss.SSS”) &&
parse_date(to_string($message.newtimestamp),“YYYY-MM-dd’T’HH:mm:ss.SSS”) <= parse_date(to_string(“2019-12-03T07:50:43.895”),“YYYY-MM-dd’T’HH:mm:ss.SSS”)
then
set_field(“nts”, “TS-5963”);
end