Pipeline parsing and setting correct timestamp

Graylog Central (peer support)
1

Hello,

I like to parse and set the correct timestamp to logmessages with the pipeline feature. I have all ready fields with hour,minutes,seconds, day of month and so on…

My first pipeline stage is detecting the logs (for example application server type a). The second stage is parsing the logmessage with the proper grok pattern.

Now I have the issue, that the logmessage timestamp and the graylog (elasticsearch) timestamp differ a view seconds. My idea is to use the parsed fields from stage two and set the correct timestamp in stage three.

My idea is a rule like this:

rule "Appserver Parsing - Timestamp"
when
 true
then
  let new_timestamp = parse_date(
        value: to_string($message.year) + "-" + to_string($message.month) + "-" + to_string($message.day) + " " + to_string($message.hour) + ":" + to_string($message.minute) + ":" + to_string($message.seconds),
        pattern: "yyyy-MM-dd hh:mm:ss",
        timezone: to_string("Europe/Berlin")
  );
  set_field("AAA_pipline_timestamp", new_timestamp);
  // If the timestamp is correct, rename the field
end

Does this is the correct way? Is an other way a better solution?

I have many different log formats and think this will be a simple way to get the values in fileds and then set the correct timestamp.

Kind regards
Julian

2

Hi Julian,

according to the guys from graylog and the community this is the preferred solution.

See here for example: Searching imported logs by log timestamp, not time Graylog received the log

Do you use this already and is it working? (The set_field("timestamp", new_timestamp); part)

I have the problem that I can’t store the DateTime object into graylogs timestamp field. If I do this graylog seems to start dropping the messages without any error.

Kind regards,
Thomas

3

I have the same problem when I want to replace timestamp field with my timestamp from message.

4

With what value (and type) did you try to replace the “timestamp” field?
What was the error message you’ve received?

5

I use this pipeline

rule “parse event timestamp”
when
true
then
set_field(“timestamp”,parse_date(to_string($message.test),“yyyy-MM-dd’T’hh:mm:ss.SSSZ”));
end

And extractor with name test from this message

2017-05-22T03:10:16+00:00 itc2000 daemon info itcTransceiver[1320]: SBCWorkflow: now 1970-Jan-03 17:50:18.831346 (0.-247636)

and I got extracted timestamp from message → test: 2017-05-22T03:10:16.000Z

Then with pipeline want to replace timeline with this data from test extractor.

1 Like
6

That rule works for me.

Are you sure, that the “test” message field contains a date which exactly matches the pattern you’ve used in the parse_date() function (without any leading or trailing characters)?

Are you also sure that the order of processors is correct (first extractors, then message processing pipelines)?

7

This is my Log message

message

2017-05-22T03:10:16+00:00 itc2000 daemon info itcTransceiver[1320]: Timer: Transmission start - 23710608us - 0x1169cb90 (now: 23578441us)

This is test extractor from message

test

2017-05-22T03:10:16+00:00

and this is timestamp that I want to replace

timestamp

2017-06-27T12:08:01.000Z

This is regular expression that I use to extract timestamp from message to test

^([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{2}:[0-9]{2}).*$

Do you see some error?

Error on graylog log

facility
runit-service

gl2_processing_error
For rule ‘parse event timestamp’: In call to function ‘parse_date’ at 5:26 an exception was thrown: Invalid format: “”

level
6

message
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]

source
graylog-server

timestamp
2017-07-03T08:00:21.392Z

8

Can you send your configuration for timestamp change?

9

This clearly shows that you’re trying to parse an empty string (“”) as timestamp, which obviously doesn’t match the date pattern in your parse_date() function.

Maybe you should ensure that the “test” field really contains a parseable string.

10

I found a workaround! I can use extracted fielt to create error chart

11

No it doesn’t work I was think that I can use it by extracted field

like this:

Timestamp_ITC4, Query: source:filesystem AND message:“SC2000SyncCard.Error”

But still show time when is upload to graylog not time from extractor off message time.

What to do, I’am desperate!

12

What do you think about this pipeline ? You just have to set the TZ for the now function.
No extractor needed.

rule “timestamp_now+2"
when
true
then
let new_date = parse_date(substring(to_string(now(”+0100")),0,23), “yyyy-MM-dd’T’HH:mm:ss.SSS”);
set_field(“timestamp”, new_date);
end