Due to lack of storage, we have to move our log storage to another server. Extending the Elasticsearch cluster is unfortunately not an option because our second bare metal server has a lot more storage which would result in different Elasticsearch node sizes.
I was planning on moving the following way:
Setting up Elasticsearch on new server
Stopping Graylog
Adjusting config accordingly to new Elasticsearch cluster
Starting Graylog again
Moving the indices of the old cluster, reindex or elasticdump?
Are the any errors I should expect after changing the Elasticsearch cluster? I’d like to keep all inputs etc.
Any suggestions or problems you would see with that?
If you are going to migrate i would recommend to migrate away from elasticsearch. The most recent version of elasticsearch supported in graylog is 7.10 which went end of life may 2022.
The basic idea is:
Upgrade elastic to 7.10 (if not already)
re-index data (note: this is only needed if you upgraded elasticsearch)
build new server, install OpenSearch
copy data folder from elasticsearch to opensearch
For example, from /var/lib/elasticsearch/nodes
to (on new server) /var/lib/opensearch/nodes
Set owner of folder
e.g. chown -R opensearch:opensearch /var/lib/opensearch/nodes (-R means recursive and will apply to all child files and folders)
This document covers the process in greater detail: Guide Index
IF you only want to move the data and not change versions or migrate to opensearch:
build new server, install Elasticsearch 7.10.2
copy data folder to new server
For example, from /var/lib/elasticsearch/nodes
Set owner of folder
e.g. chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/nodes (-R means recursive and will apply to all child files and folders)
Thanks Drew! Does copying the /var/lib/elasticsearch/nodes really result in all indices being copied without any issues? I couldn’t believe it’s that easy…
Does copying the /var/lib/elasticsearch/nodes really result in all indices being copied without any issues
It does. Some important points though:
be sure you STOP elasticsearch/opensearch before starting the copy as these files are constantly changing
be sure the versions are the same on the source and destination servers
ensure you copy ALL files/folders beneath nodes
ensure the owner and permissions are the same. Typically when using rsync with the -a (archive mode) permissions are preserved but owner is not.
when starting elasticsearch/opensearch on the new server open the log in tail to quickly catch any errors/issues
if you do run into issues/errors, delete the node directory on your target server and start over
I just ran through this yesterday when migrating my opensearch server from old hardware to new hardware and this worked perfectly. Where possible, i reccomend running through a test to get a feel for how it works, how long it will take, and to see if you run into any issues.
Thankfully graylog can buffer log messages for a good while (you may need/want to increase your journal size if 5GB is not enough to store your messages during the time it takes to copy your nodes folder while elaticsearch is stopped) so you won’t lose any messages while elaticsearch is offline.
