Extract timestamp from message

Background

I am trying to override the default timestamp Graylog assigns messages with the one written in the message itself.

To achieve this I read the official docs:

https://docs.graylog.org/en/3.3/pages/extractors.html#the-standard-date-converter

However I am still unable to correctly set the timestamps.

Messages

Most of messages start like the following:

2021-02-14T06:07:58.521009+00:00 myApp00_rest_of_log

There are other types of messages that don’t adhere to this standard, but that is for latter.

To extract the timestamp from the message I have created the following extractor:

The RegEx does it’s job nicely, however it’s the converter that’s killing it.

Problem

As you can see I am using the converter:

yyyy-MM-ddTHH:mm:ss.S

This doesn’t work. I have also tried the following variations:

yyyy-MM-dd'T'HH:mm:ss.S
yyyy-MM-dd HH:mm:ss.S
yyyy-MM-dd'T'HH:mm:ss.SSSSSS
yyyy-MM-dd HH:mm:ss.SSSSSS
yyyy-MM-ddTHH:mm:ss.SSSSSS

At this point you might have picked up a couple of things:

  1. I have 2 problems: the “T” character in the middle and the precision that comes after the seconds (has 6 numbers, instead of just 3 (\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}))
  2. I am totally lost and I am just guessing random combinations at this point

OK but flawed solution

The “OK” solution would be to use the “Flexible Date” converter, however if I do that I will lose the precision that comes after the seconds (those 6 decimals mentioned in point 1).

Unfortunately for me, I need that amount of precision in my system, so I can’t just cut out the last 6 decimals.

Questions

  1. How can I fix my converter so it can properly override the Timestamp with the precision I need?

With the following converter:

yyyy-MM-dd'T'HH:mm:ss.SSSSSS

I am getting the following error:

java.lang.IllegalArgumentException: Invalid format: "2021-02-14T04:20:10.421162" is malformed at "162"`
at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:160) ~[graylog.jar:?]
at org.graylog2.inputs.converters.DateConverter.convert(DateConverter.java:87) ~[graylog.jar:?]`
at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:260) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:247) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:77) [graylog.jar:?]
at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:136) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:121) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:93) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]

Which is surprising as S is a fraction of a second and according to DateTime it should be able to go up to 9 S:

What am I missing?

It is my belief that Graylog does not support Timestamps with microsecond precision. I was unable to confirm this however. Has anyone ever made a microsecond Timestamp work on Graylog?

Could this be a bug?

I tried to look into this and have probably only confused myself more… I’m not a java developer but I was curious about this, so I looked at the source on github and I came across this issue:

So I was like… cool… it’s currently not, but they are looking to fix that. So I took a look at the

DateTime.Formats.TIMESTAMP_TZ format because it was DATETIME_TZ, and they changed it to TIMESTAMP_TZ. but when I tracked it down (again… not a Java guy) I see that every instance of DataTime is from the org.joda.time.DateTime library and from what I can tell, joda-time does not support microseconds.

Please take this with a grain of salt as again I am not a Java developer… perhaps @jan or @aaronsachs could shed some light on this.

you are 100% correct with your findings @cawfehman

Jan

Do we know in which official version this PR will be in?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.