Elasticsearch parse exception/Indexer failures

Graylog Tech Challenges
1

I’ve the following error:

ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [apache_timestamp] of type [date] in document with id 'a0511820-1532-11ec-a2d4-0050568c8ce4'. Preview of field's value: 'Tue Sep 14 10:06:09.999727 2021']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=failed to parse date field [Tue Sep 14 10:06:09.999727 2021] with format [strict_date_optional_time||epoch_millis]]]; nested: ElasticsearchException[Elasticsearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

does that mean, Graylog/Elasticsearch dropped the hole message? How can I figure out which server are sending logs? (In my opinion its a bug that a Log-Managementsystem is dropping(!!) logs)

2

Hello,

What is your environment, meaning what is your configuration to ingest logs and how you sending them?
As for your error this looks like elasticsearch expects the [apache_timestamp] field to be a date. Either you have a mapping misconfiguration, or it could be a generic ISO datetime parser, where the date must include the year at a minimum, and the time (separated by T), is optional. Examples: yyyy-MM-dd’T’HH:mm:ss.SSSZ or yyyy-MM-dd."

The date parser defaults to strict_date_optional_time or epoch_millis.
Did you create a template for mapping?

Edit: I did a quick google maybe this might help.

3

I send logs normally via filebeat, but some “older” webserver have strange configs with something lke |nc syslog.example.de:5514 so and it is not that easy to find them all… it would be much easier to know which IP/Source it was. it is also bad, that graylog/elastic drops this messages (for me this is a bug)

4

Hello,

I feel your frustration on logs dropped. In my lab I’m testing out Elastic-stack, Open Distro For Elasticsearch → AWS Opensearch. Depending on the device and what logs are shipped, I run into the same problems with each of them. Sorry I cant be more help.

5

This is something we’re aware of and have a couple of issues open to address. The ideal approach is that we implement dead letter queues for elastic where we take the parts of a message that can’t be parsed and drop them

2 Likes
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.