Error mapper_parsing_exception on gl2_original_timestamp after upgrading to 6.3.3

[cklg]

Hello Graylog community,

I recently upgraded my Graylog instance from 6.1.8 (OpenSearch 2.15.0) to 6.3.3 (OpenSearch 2.19.3) and have a weird indexing error that I’m unable to solve, which seems to be linked to a date format.

1. Describe your incident:

When opening the Indexer failure page (/system/indices/failures), I see every second a massive amount of errors that look like this:

OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [gl2_original_timestamp] of type [date] in document with id '7917c080-a2b7-11f0-9d2d-02420a2b0c04'. Preview of field's value: '643698-11-02 06:43:34.000']]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [643698-11-02 06:43:34.000] with format [uuuu-MM-dd HH:mm:ss.SSS]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=Text '643698-11-02 06:43:34.000' could not be parsed at index 0]];

As you can see, it seems like OpenSearch try to parse the following string to date:

643698-11-02 06:43:34.000

using the format uuuu-MM-dd HH:mm:ss.SSS. Obviously, something is wrong with this date, but I’ve no idea what.

And I also have no clue where this gl2_original_timestamp field comes from, I didn’t have it with v6.1.8.

Any idea how I could fix this?

2. Describe your environment:

• OS Information: RedHat 8 with Docker

• Package Version: 6.3.3

3. What steps have you already taken to try and solve the problem?

I tried using the Index Set Field Type Profiles page (/system/indices/field-type-profiles) to force this field to be a string and not a date, but when I do so, it cause Graylog to be unable to rotate indexes. I had to delete my forced type conversion to solve this.

4. How can the community help?

Did any of you have a similar issue or know where it could be coming from?

Thanks in advance for your help.

[Joel_Duffield]

Your most likely seeing this `gl2_original_timestamp` causees data loss · Issue #23025 · Graylog2/graylog2-server · GitHub

It doesn’t look to have been dealt with yet, I’ll see if there is a work around available.

[cklg]

Thank you Joel, it is indeed exactly the same problem. I’ll follow-up on GitHub then.

[Joel_Duffield]

That field will be in that format and exist before it hits your pipeline rules, so for now you can make a pipeline that looks for the bad formating and fixes them so that they can be stored properly.

[cklg]

Good suggestion, thanks!

I tried this approach but it looks like the pipeline condition (Message has field gl2_original_timestamp) fails to be true.

I essentially try to transform it to the current date if the field exists.

But then, although the evaluation of this pipeline is done (I can see Stage 0: Throughput: 43 msg/s), the rule itself never runs (I can see Throughput: 0 msg/s for the actual rule).

It feels like since gl2_original_timestamp isn’t a normal field, it fails to be detected in incoming messages.