Reindex graylog data to elastic 5.x

Good morning everyone,

I am trying to migrate graylog to a new version. Currently I have the elasticsearch indexes in version 1.7, I can read them from elasticsearch 2.x but I need to reindex to be able to read them in versions 5.x or higher.

For this I am trying to use the reindexing API, but I am getting the following error several times:

"{
“index”: “graylog_2023_re”,
“type”: “message”,
“id”: “f27395d1-92d6-11eb-9a08-525400f612cd”,
“cause”: {
“type”: “mapper_parsing_exception”,
“reason”: “Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.”
},
“status”: 400
},
{
"

Can somebody help me? Thank you.

Hi @lmrc

I don’t know about you, but I think the better way to do this is via Kibana, I’m not a big fan of UI in general to administrate things, but normally Kibana is pretty handy hepling execute internal stuf in ES.

Considere this.

Thank you very much for your reply.

I’m new in the ES world and I didn’t know that you can reindex using kibana.

How can I do it? I install Kibana and it can be done from its own UI or do I have to install an additional plugin?

Thank you.

Hi @lmrc

I’m very sorry for losting your time, but what I said in last comment was made without confirm if it’s possible to be done via Kibana, apparently it isn’t.

However, I saw in this link that you can fix this error following this step:

Message 1:
“org.elasticsearch.index.mapper.MapperParsingException: Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters”
The document source to index contains a field called “_id” which is reserved as metadata inside Elasticsearch. According to Elasticsearch documentation, each document has metadata associated with it, such as the _index , mapping _type , and _id metadata fields. Here is a complete list of metadata fields in Elasticsearch 7.9:

Category Fields
Identity metadata fields _index / _type / _id
Document source metadata fields _source / _size
Indexing metadata fields _field_names / _ignored
Routing metadata field _routing
Other metadata field _meta

Suggestions:

  1. Rename the field to avoid the conflicts.

Hope It helps.

It’d probably also be useful to share what call you’re making via the API @lmrc . You might also check on the ES forums–seems like some folks there are having a similar issue: How to change a documents _id during a reindex - Elasticsearch - Discuss the Elastic Stack.

1 Like

Hi @reimlima @aaronsachs ,

Thank you so much for the reply.

It looks like it’s because of the ID field, but I don’t really know how to replace it with the scripts I’ve found.
I will ask in the elastic forums, maybe they can give me a possible solution.
.
Again, thank you very much, greetings.

1 Like