Failed to parse date field

Graylog Central (peer support)
1

Graylog 4.1.10, Elastic 6.8.22

Hey,

I am ingesting log messages from custom software. Most messages seem to come through just fine but then there are some that fail due to “failed to parse date field”.

An example of failing message:

[2022-01-20T10:39:10,699][DEBUG][o.e.a.b.TransportShardBulkAction] [2CEfq1M] [graylog_37][0] failed to execute bulk item (index) index {[graylog_deflector][message][6f60b1e4-79cc-11ec-ae42-0050568815d4], source[{“gl2_accounted_message_size”:436,“SourceModuleType”:“im_file”,“level”:6,“log”:"{{dateFormat alertMessage.timestamp format=‘dd.MM.yyyy HH:mm:ss’}}",“gl2_remote_ip”:“X.X.X.X”,“gl2_remote_port”:57242,“streams”:[“5ed618ec5d8362036e809302”,“5e7b0ac45d8362099efe24ae”,“000000000000000000000001”,“5c94dee4f7f28c69cba861e1”],“gl2_message_id”:“01FSVA7EZZYKJQ30R185K54FBF”,“source”:“worker02”,“message”:"{“log”:"{{dateFormat alertMessage.timestamp format=‘dd.MM.yyyy HH:mm:ss’}}\n",“stream”:“stdout”,“time”:“2022-01-20T08:39:06.344427Z”}",“gl2_source_input”:“5bbc795d11973e0a7f631721”,“EventReceivedTime”:“2022-01-20 10:39:09”,“SourceModuleName”:“alert”,“stream”:“stdout”,“gl2_source_node”:“cb4a9a8c-e692-4e55-947f-8fc89edef6b4”,“time”:“2022-01-20 10:39:06”,“timestamp”:“2022-01-20 08:39:09.000”}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [time] of type [date] in document with id ‘6f60b1e4-79cc-11ec-ae42-0050568815d4’

So, I understand the problem is this part in original log: “time”:"2022-01-20T08:39:06.344427Z

Previously, I didn’t have a custom format for the field “time” and I saw this kind of explanation after that error:
Caused by: java.lang.IllegalArgumentException: Invalid format: “2022-01-20 00:00:17” is malformed at " 00:00:17"

This suggests that “2022-01-20T08:39:06.344427Z” gets successfully transformed into “2022-01-20 08:39:06” but is not parsed after that?

So I tried fixing it by creating a custom format for the field and managed to get a change in explanation:
Caused by: java.lang.IllegalArgumentException: failed to parse date field [2022-01-20 11:03:36] with format [8yyyy-MM-dd’T’HH:mm:ss.SSSSSS||date_optional_time||8yyyy-MM-dd’T’HH:mm:ss.SSSSSSSSS||8yyyy-MM-dd’ 'HH:mm:ss||date_time_no_millis]
Caused by: java.time.format.DateTimeParseException: Failed to parse with all enclosed parsers

I suppose ES 6 doesn’t support format strict_date_optional_time_nanos as graylog complains about that when trying to add it, that’s why there is the nanosecond pattern as well as. I have tried with and without 8 in front as well.

I have tried several format strings that being the latest, but still it fails on maybe 5 messages out of 2000. I almost thought I got it after quoting that space in “correct looking” format, but no.

Any suggestions?? :slight_smile:

Thanks!

2

Hello,

I need at ask you a couple question about your environment.
What type of input are you using to ingest your logs?
The logs sent from your application are they compatible with the type of INPUT your using now?
Are you using any extractors?

ES 6 can, but this depends on how you set up your environment.
For example here is a list of other community members with similar issues or perhaps the same issue BTW the list is long :smiley:

https://community.graylog.org/search?q=timestamp

3

Hey,

thanks for your help!

My input is GELF TCP, basically docker stderr forwarded by nxlog from the host. There were some random issues with docker logspout so we ended up with this arrangement. I don’t know if it would work better with Docker sending logs directly to Graylog with gelf but our Docker CE doesn’t support multiple drivers so that’s why we currently have this setup.

I do have 1 extractor there which I believe is considered for the message as it contains string “alert” but it hasn’t been written for that line:

Condition
    Will only attempt to run if the message includes the string type='alert'
Configuration
    list_separator: ,
    kv_separator: =
    key_prefix:
    key_separator: _
    replace_key_whitespace:
    key_whitespace_replacement: _

That was goog, I hadn’t thought of that and extractors at all.

Now I could try to see what that extractor does with a problematic message but I can’t load it there, no matter what I put to messageID and index fields it says “Not found”…

So, a secondary question: regarding this elasticsearch/graylog.log entry, what should I enter into messageID and index fields on Edit extractor page to try the extractor on it??

[2022-01-24T08:29:17,686][DEBUG][o.e.a.b.TransportShardBulkAction] [2CEfq1M] [graylog_38][2] failed to execute bulk item (index) index {[graylog_deflector][message][f40190d2-7cde-11ec-ae42-0050568815d4],source[{"gl2_accounted_message_size":356,"SourceModuleType":"im_file","level":6,"log":"Debug: request, error, close","gl2_remote_ip":"a.b.c.d","gl2_remote_port":60606,"streams":"000000000000000000000001","5c94dee4f7f28c69cba861e1","5ed618ec5d8362036e809302"],"gl2_message_id":"01FT5CCGJXPW229RDSK4PEBY49","source":"worker01","message":"{\"log\":\"Debug: request, error, close \\n\",\"stream\":\"stderr\",\"time\":\"2022-01-24T06:29:14.462979Z\"}","gl2_source_input":"5bbc795d11973e0a7f631721","EventReceivedTime":"2022-01-24 08:29:15","SourceModuleName":"mgmt-api","stream":"stderr","gl2_source_node":"cb4a9a8c-e692-4e55-947f-8fc89edef6b4","time":"2022-01-24 08:29:14","timestamp":"2022-01-24 06:29:15.000"}]}

You are correct, the list with timestamp issues is rather long… :smiley:

4

Hello,

If your going to search for the message ID use this.

gl2_message_id: 01FT5CCGJXPW229RDSK4PEBY49

Double check your Log shipper configuration if your using Nxlog /w GELF TCP/TLS should look something like this.

This config is without certificates.

Nxlog_config 
define ROOT C:\Program Files (x86)\nxlog
define LOGFILE C:\Program Files (x86)\nxlog\data\nxlog.log

<Extension gelf>
    Module      xm_gelf
 </Extension>

<Output out>
    Module      om_tcp
    Host        graylog.domain.com
    Port        51412
    OutputType  GELF_TCP  <--  "should look like this"   
    Exec $Hostname = hostname_fqdn();
    Exec $FullMessage = $raw_event;
    #Exec        to_syslog_snare();
</Output>

I’m wonder that the file in question may not be formatted correctly to use GELF_TCP.
With Nxlog configuration you can create a separate input , output and route for that application with a different GL input. Just an idea.

EDIT:

From what I see your using Stream “000000000000000000000001”, Index name shown above [graylog_38] and messageID 01FT5CCGJXPW229RDSK4PEBY49

image
image1433×466 30.8 KB

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.