Reindexing automatically for migration from Elastic 6.8 (indices) to OpenSearch

Graylog Central (peer support)
, ,
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
When I try to upgrade/migrate our Graylog/Mongo/Elastic to OpenSearch, I have two big hurdles :

  • first one is I have too much shards (way too much, ~1500 and I already lowered them wherever I could, coming from ~2700 shards) and my question is, what is the “best” way to configure a graylog server with one node wich is also a one node Elastic ? Meaning : how much shards, how much indices ?
  • second one, how to simply move all my indices through the paces to get them to be compatible with the migration without losing documents : I have ~1,400M Documents on ~400 indices (~40 templates (system>indices on graylog)), around 50 of them are actively written. I got the reindexing API, but this is tedious to do to each of them, and it takes around 2x the disk space (before talking about any elastic snapshot…) My question : isn’t there some angel around here that has a good bash to launch to get all of them copied, reindexed, deleted, recreated, re-reindexed with the mappings, the settings and all that still showing up ?

2. Describe your environment:

  • OS Information: RedHat 8

  • Package Version: Graylog 5.2, MongoDB 6, Elastic 7.10 (update path to OpenSearch 1.3 and then 2.12)

  • Service logs, configurations, and environment variables: vanilla (all on the same VM)

3. What steps have you already taken to try and solve the problem?
Tried the brute force (rotate to the moon) but I loose every document I needed. Which makes me think there would be something to try with rotation and reindexation to new rotation… :S Not tried that yet.

4. How can the community help?
Bash script, how-to, guidelines, even if it is a hard stop because my config is too wicked to stay put…

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @TechSys

Another community member while back posted what to do. I followed those instructions and worked great.

Also this Documentation work very well.

https://go2docs.graylog.org/5-0/planning_your_deployment/upgrading_to_opensearch.htm

What I did was migrate everything from one CentOS7 node to another Ubuntu node and performed an upgrade from ES 7.10 to OS v1.3. My update path was a little different (1.3 → 2.0 → 2.5)

I would not go past opensearch=2.5.0, There have been some issues/bugs FYI

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.