Lots of Indexer failures recently

Graylog Central (peer support)
,
1

Hi,

I just noticed that I recently have lots of Indexer failures like this:

{"type":"mapper_parsing_exception","reason":"Field [_timestamp] is a metadata field and cannot be added inside a document. Use the index API request parameters."}

Elasticsearch version: 6.8.13
Graylog Version: 3.3.8

I don’t know where those errors are coming from - and why I suddenly get them.

Also interesting: Only the Graylog master node has this errors in the log. I have a setup with 2 Graylog nodes. The other node does not have such errors.

I have no idea where this “_timestamp” field suddenly comes from.

2

Because the messages are not indexed (-> not stored in ES), I have no chance to figure out which messages cause this issue.

Is there a way to show the faulty message in the Graylog log files?

3

Elasticsearch doesn’t allow field names to start with _ so those are being stripped. Once they are removed the field name becomes “timestamp”, which is a system-defined metadata field that can’t be contained in a document. That’s why you’re seeing indexer failures.

You can view the input to see messages that contain the field to help determine the source.

4

I use Logstash to feed Graylog. I double checked my configuration; I am not actively sending a field named “_timestamp”. I have no idea where this suddenly comes from.

All I get in the Graylog server log are lines like this:

“… id [b9a30272-9030-11eb-a94f-0242620bd9d0], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=Field [_timestamp] is a metadata field and cannot be added inside a document. Use the index API request parameters.]]] …”

Documents with this ID don’t exist in Elasticsearch (of course not, because of index failure), so I cant see which documents failed.

It would help to get some more information, e.g. who (IP) is sending this faulty message, how the faulty message looks like, etc. Is there a way to get this kind of information in the Graylog Server?

5

I added some debug information via Pipelines and Rules to log the faulty messages to the Graylog server log.

{"@timestamp":"2021-04-02T11:02:27.194+00:00", "message":"Ingesting ...}

To my surprise, the message does not have a field named _timestamp - it has a field named @timestamp. Is there some sort of conversion going on in the Graylog server from @ to _?

6

@ is not supported as a character in a field name either, so the net result is the same. Try replacing the symbol with some other prefix in the pipeline to allow a unique fieldname to be created. Perhaps source_timestamp?

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.