Move Elasticsearch Cluster without (major) downtime

Due to lack of storage, we have to move our log storage to another server. Extending the Elasticsearch cluster is unfortunately not an option because our second bare metal server has a lot more storage which would result in different Elasticsearch node sizes.
I was planning on moving the following way:

  1. Setting up Elasticsearch on new server
  2. Stopping Graylog
  3. Adjusting config accordingly to new Elasticsearch cluster
  4. Starting Graylog again
  5. Moving the indices of the old cluster, reindex or elasticdump?

Are the any errors I should expect after changing the Elasticsearch cluster? I’d like to keep all inputs etc.
Any suggestions or problems you would see with that?

If you are going to migrate i would recommend to migrate away from elasticsearch. The most recent version of elasticsearch supported in graylog is 7.10 which went end of life may 2022.

The basic idea is:

  1. Upgrade elastic to 7.10 (if not already)
  2. re-index data (note: this is only needed if you upgraded elasticsearch)
  3. build new server, install OpenSearch
  4. copy data folder from elasticsearch to opensearch
    • For example, from /var/lib/elasticsearch/nodes
    • to (on new server) /var/lib/opensearch/nodes
  5. Set owner of folder
    • e.g. chown -R opensearch:opensearch /var/lib/opensearch/nodes (-R means recursive and will apply to all child files and folders)

This document covers the process in greater detail: Guide Index

IF you only want to move the data and not change versions or migrate to opensearch:

  1. build new server, install Elasticsearch 7.10.2
  2. copy data folder to new server
    • For example, from /var/lib/elasticsearch/nodes
  3. Set owner of folder
    • e.g. chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/nodes (-R means recursive and will apply to all child files and folders)

Hope that helps.

1 Like

Thanks Drew! Does copying the /var/lib/elasticsearch/nodes really result in all indices being copied without any issues? I couldn’t believe it’s that easy…


Does copying the /var/lib/elasticsearch/nodes really result in all indices being copied without any issues

It does. Some important points though:

  • be sure you STOP elasticsearch/opensearch before starting the copy as these files are constantly changing
  • be sure the versions are the same on the source and destination servers
  • ensure you copy ALL files/folders beneath nodes
  • ensure the owner and permissions are the same. Typically when using rsync with the -a (archive mode) permissions are preserved but owner is not.
  • when starting elasticsearch/opensearch on the new server open the log in tail to quickly catch any errors/issues
  • if you do run into issues/errors, delete the node directory on your target server and start over

I just ran through this yesterday when migrating my opensearch server from old hardware to new hardware and this worked perfectly. Where possible, i reccomend running through a test to get a feel for how it works, how long it will take, and to see if you run into any issues.

Thankfully graylog can buffer log messages for a good while (you may need/want to increase your journal size if 5GB is not enough to store your messages during the time it takes to copy your nodes folder while elaticsearch is stopped) so you won’t lose any messages while elaticsearch is offline.


Jup, that actually worked flawlessly after chowning to elasticsearch:elasticsearch.
Thanks, wish a nice weekend!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.