Use Timestamp of logfiles

[supermario18b]

Hi everyone,

I’m trying to parse the timestamp from logfiles and use them as timestamp of Graylog messages (instead of the ingestion time).
As I understand the “right” way is to use an extractor to read the string which contains the timestamp and put in a variable. Then a Pipeline should be used to parse the content of the variable and set it as Graylog timestamp. That’s what I tried with no success:

This the regular expression I used as extractor:

^([0-9]{4}-[0-9]{2}-[0-9]{2}\s*[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}).*$

This shoud be parse a timestamp like this:

2018-08-09 13:12:26,285

I put the result in event_ts. I tried to add a date converter but it didn’t work. At the moment I use any, should I?

Then I added a rule:

rule “parse event timestamp”
when
true
then
let new_date = parse_date(to_string($message.event_ts), “yyyy-MM-dd hh:mm:ss,SSS”);
set_field(“timestamp”, new_date);
end

Now I can’t see any result when I click “search” and in the server.log there is:

[Messages] Failed to index message: index=<graylog_0> id=<23c859a2-9bc7-11e8-85f4-08002754a6aa> error=<{“type”:“mapper_parsing_exception”,“reason”:“failed to parse [event_ts]”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“Invalid format: “2018-08-09 13:26:56,022” is malformed at " 13:26:56,022"”}}>

It seems that the extractor worked because in the logfile I can see “2018-08-09 13:26:56,022” but I can’t understand how to solve the “malformed” error .

Could you please help me?

If you need more details, let me know it.

[jan (Jan Doberstein)]

the timestamp need to be ISO8601 and the ,SSS does not fit into this.

[supermario18b]

Can I solve this? I have an old installation of Graylog where there is any Pipelines and I don’t know why, but it shows the right timestamp (timestamp of logs with comma).

In the Graylog stream it ends with .SSS but in the logs it ends with ,SSS but they have the same value
Thanks

[supermario18b]

Ok… I don’t know what happened but it works.
I canceled the pipeline and the rule. In the Extractor configuration I put the result of the regular expression in “timestamp” field instead of "event_ts.

Now when I make a search I see the Graylog time with .SSS and the logfile time with ,SSS. The time is the same.

Is it correct to do so?

P.S. maybe it is correct: http://docs.graylog.org/en/2.4/pages/extractors.html#the-standard-date-converter
If yes, I didn’t understand why use pipelines