Issues with Elasticsearch 7.10.1

Graylog Central (peer support)
1

I updated from:

Graylog: 3.3.4
Elasticsearch: 6.8.11

to:

Graylog: 4.0.5
Elasticsearch: 7.10.1

At first all looked good, but then I noticed that I was losing messages. In the Graylog server log there are lots of lines like this:

2021-03-23 00:18:33,566 ERROR: org.graylog.storage.elasticsearch7.MessagesAdapterES7 - Failed to index [3] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[778]: index [graylog_2456], type [_doc], id [ebb1ca65-8b64-11eb-85eb-0242863a4b0e], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [_version] of type [_version] in document with id ‘ebb1ca65-8b64-11eb-85eb-0242863a4b0e’. Preview of field’s value: ‘1’]]; nested: ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=Field [_version] is a metadata field and cannot be added inside a document. Use the index API request parameters.]];]

Where does this field “_version” come from? And why is it bothering ES now?

I use a custom mapping that is defined with this call:

curl -X PUT 'http://localhost:9200/_template/graylog-custom-mapping?pretty' -H 'Content-Type: application/json' -d '
{
  "template": "graylog_*",
  "mappings" : {
    "message" : {
      "properties" : {
        "http_method" : {
          "type" : "text",
          "index" : false
        },
        "exception" : {
          "type": "text",
          "analyzer": "standard"
        },
        "fullsize_bytes" : {
          "type" : "long"
        },
        "duration_sec" : {
          "type" : "double"
        }
      }
    }
  }
}

’

2

Hello,

So what does the web interface say about this error?

After you upgraded did you try to rotate you indices? This fixed my issue.

3

So what does the web interface say about this error?

I quickly rolled back because it was a productive environment - so I had no chance to check in the web interface. But I assume would say pretty much what the server log says:

type=mapper_parsing_exception, reason=Field [_version] is a metadata field and cannot be added inside a document. Use the index API request parameters.

After you upgraded did you try to rotate you indices? This fixed my issue.

This was what I tried first, but it didn’t solve the issue.

4

Are you working with a VM ( Virtual Machine) by chance? If so maybe clone it , adjust the network config and try upgrading again. TBH I would never upgrade my production Graylog without testing it first on another server.

1 Like
6

TBH I would never upgrade my production Graylog without testing it first on another server.

I tested it before and it looked OK. It was only after I sent real productive log data that I noticed that ~10% of the logs are missing.

7

I havent tried a custum template/s yet but have you seen this?

I’m just started learning ES so Im unsure. The error seams like something with a mapping for the property value within your value object property.

Maybe someone here with more experience with ES can jump in.
Sorry I cant be more help.

8

I still don’t know where this field “_version” comes from. Is this a Graylog internal field?

9

Initially I was thinking this might be surfaced add part of the Graylog Schema Graylog Information Model Schema — Graylog (GIM) Schema 0.0.1 documentation. But my searches on mobile aren’t really proving effective. If you search for _version in the search bar, does it return anything?

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.