Reindex graylog data to elastic 5.x

Development
1

Good morning everyone,

I am trying to migrate graylog to a new version. Currently I have the elasticsearch indexes in version 1.7, I can read them from elasticsearch 2.x but I need to reindex to be able to read them in versions 5.x or higher.

For this I am trying to use the reindexing API, but I am getting the following error several times:

"{
“index”: “graylog_2023_re”,
“type”: “message”,
“id”: “f27395d1-92d6-11eb-9a08-525400f612cd”,
“cause”: {
“type”: “mapper_parsing_exception”,
“reason”: “Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters.”
},
“status”: 400
},
{
"

Can somebody help me? Thank you.

2

Hi @lmrc

I don’t know about you, but I think the better way to do this is via Kibana, I’m not a big fan of UI in general to administrate things, but normally Kibana is pretty handy hepling execute internal stuf in ES.

Considere this.

3

Thank you very much for your reply.

I’m new in the ES world and I didn’t know that you can reindex using kibana.

How can I do it? I install Kibana and it can be done from its own UI or do I have to install an additional plugin?

Thank you.

4

Hi @lmrc

I’m very sorry for losting your time, but what I said in last comment was made without confirm if it’s possible to be done via Kibana, apparently it isn’t.

However, I saw in this link that you can fix this error following this step:

Message 1:
“org.elasticsearch.index.mapper.MapperParsingException: Field [_id] is a metadata field and cannot be added inside a document. Use the index API request parameters”
The document source to index contains a field called “_id” which is reserved as metadata inside Elasticsearch. According to Elasticsearch documentation, each document has metadata associated with it, such as the _index , mapping _type , and _id metadata fields. Here is a complete list of metadata fields in Elasticsearch 7.9:

Category Fields
Identity metadata fields _index / _type / _id
Document source metadata fields _source / _size
Indexing metadata fields _field_names / _ignored
Routing metadata field _routing
Other metadata field _meta

Suggestions:

  1. Rename the field to avoid the conflicts.

Hope It helps.

5

It’d probably also be useful to share what call you’re making via the API @lmrc . You might also check on the ES forums–seems like some folks there are having a similar issue: How to change a documents _id during a reindex - Elasticsearch - Discuss the Elastic Stack.

1 Like
6

Hi @reimlima @aaronsachs ,

Thank you so much for the reply.

It looks like it’s because of the ID field, but I don’t really know how to replace it with the scripts I’ve found.
I will ask in the elastic forums, maybe they can give me a possible solution.
.
Again, thank you very much, greetings.

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.