Graylog Indexer Failures

[spardok]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
After about 3 or so days I will start getting very specific indexer failures, if I rotate the index it will stop, and then start up after around 3 days. Failures are below:

OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [winlogbeat_winlog_event_data_param2] of type [date] in document with id ‘df08e816-2ccf-11ef-a0ba-0242ac120004’. Preview of field’s value: ‘running’]]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [running] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [winlogbeat_winlog_event_data_param2] of type [date] in document with id ‘ce8379b1-2ccf-11ef-a0ba-0242ac120004’. Preview of field’s value: ‘stopped’]]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [stopped] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

OpenSearchException[OpenSearch exception [type=mapper_parsing_exception, reason=failed to parse field [winlogbeat_winlog_event_data_param2] of type [date] in document with id ‘937fe6a0-2ccf-11ef-a0ba-0242ac120004’. Preview of field’s value: ‘0x2DC7796084477C44A40CC07C48D6263D’]]; nested: OpenSearchException[OpenSearch exception [type=illegal_argument_exception, reason=failed to parse date field [0x2DC7796084477C44A40CC07C48D6263D] with format [yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis]]]; nested: OpenSearchException[OpenSearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

Errors like this keep repeating.

2. Describe your environment:

• OS Information:
  Ubuntu 22.04 LTS
• Package Version:
  Graylog Open 6.0.3 with Graylog Datanode 6.0.3 and MongoDB 7.0.11 in Docker Compose.
  This issue happened on GL 5.2 as well with Mongo 6.0
• Service logs, configurations, and environment variables:
  Evs are default other than passwords, API goes through Nginx.
  Using Sidecar with winlogbeat, beats importer uses TLS through Graylog

3. What steps have you already taken to try and solve the problem?
From my understanding of the error it is throwing parsed data that should not be in the date field in the date field, and I have no idea why it just randomly starts doing it. I’ve looked at other threads about index errors with parsing and the only thing I could really find was the opensearch parse file was using the wrong format, I could not find anything about putting data in the wrong field. So I am very much stumped.

4. How can the community help?
Hopefully figuring out why opensearch is putting winlogbeat non-date field data into the date field.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[Wine_Merchant]

Hello @spardok,

Assuming winlogbeat_winlog_event_data_param2 will not always contain a recognisable date format, one route to stop this would be to set the field data type to string so that it will also capture the contents.

Use the sidebar on the left hand side of the screen within the search context, search for the given the field and alter as below. Be sure to check the rotate index option for the change to take effect.

[spardok]

Thanks @Wine_Merchant for the reply.

Do you have any idea as to why after rotating the indices that field will not start throwing errors until about 2-3 days? It’s just seemingly odd that it would start up and then non-stop throw that error instead of being consistent from the start of a rotation.

Edit: When I go to change this I get two errors,
An error when I first click on “change field type” which throws “Could not load stream: Loading stream failed with status: FetchError: There was an error fetching a resource: Not Found. Additional Information: Stream <000000000000000000000004> not found!”
And error “Not allowed to change type in indices: [Graylog Events, Graylog System Events]”

The correct index set does show as checked in the UI though and I can uncheck it.

Edit 2: This seems to happen if I try to change any field. I am using the “Default Index” if that matters.

I cycled the docker compose, still the same issue. I have not tried just rotating indicies.The cluster and Graylog all show green aside from the indexer failures from “winlogbeat_winlog_event_data_param2”

[Wine_Merchant]

@spardok the field might contain the correct date format but not consistently, when a field without a templated mapping is written to an index in Opensearch it will give a best guess as to what that field type should be. If the first instance of a log written to the index has a field that contains a correctly formatted date then it will be stored as type date, if the next message to be written to the same index has the same field but the contents are an incorrectly formatted date then an indexing error will be thrown.

As for this error

There was an error fetching a resource: Not Found. Additional Information: Stream <000000000000000000000004> not found!”
And error “Not allowed to change type in indices: [Graylog Events, Graylog System Events]”

It appears as though you have highlighted the 'Graylog Events, Graylog System Events" when attmepting to change the field type, unselect this and you should be able to make the change. If you are entering the context of the field mapping menu while filtering by stream in search then this will be the issue.

[spardok]

@Wine_Merchant thank you for the detailed information, I’ve been waiting a few days to see if the error was thrown again after the change. It looks like changing the field type did indeed fix it.

What’s odd is I was not filtering by stream in search and “Graylog Events, Graylog System Events” do not show in the list selected, instead they are invisible. I didn’t notice the total selected at the top left and the ability to use the drop-down to just deselect everything (even if it’s invisible).