Monolithic VM to HA multinode setup migration

Graylog Central (peer support)
1

Hi All,

I’m currently running a single VM instance of graylog, ~6.2 on ubuntu server where grayog, opensearch and mongoDb are all installaed and managed seperately.

The total VM size is around 3.3TB.

I’ve been playing around with the newer graylog datanode managed deployments, and have created an environment where i have 3 graylog nodes with graylog and mongoDb installaed, and 3 datanodes; all behind an nginx proxy server for load balancing and things like that. These are all on the newly released V7.

The new cluster is currently setup, all nodes can talk to each other, does anyone know the best way to migrate from my single VM to the new cluster?

All data needs to be copied over to the new cluster, hows the best way to go about this?

Do my do all of my datanodes need to have at least the same capacity as the single VM i currently run, to facilitate copying the data from the old node to the new cluster?

Any advise is appreciated!

2

Hello @josh.7,

You would want to upgrade the older cluster versions of Graylog and Mongo to match those in the new, take a mongo dump from the old cluster and apply that to the new cluster. This will maintain all configurations and the data will be searchable in the new cluster.

One of the new data nodes would need the same storage as the old OS node, once copied and the new cluster is running the shards can then be rebalanced across the whole cluster.

3

Thanks,

Doesn’t sound too bad to me!

So inplace upgrade version wise, then copy a matter of copying across.

Is the shard rebalancing something i’d also need to deal with manually, or would setting the shard settings then doing some automated maintainance/recalculating do the trick for me?

4

I’m wondering if there might be some issues if the 3 node cluster if formed and running prior to the copy of the original data. It might be easier to copy the data across to a single node which is yet to be started, start that as a single node cluster and then add the other two nodes once that is running.

Two other options are, apply a snapshot repo to the older cluster and snapshot all indices. Add that same snapshot repo to the new cluster restore the indices there. A remote reindex could also be performed, cluster to cluster. Both of these would distribute the data across the 3 nodes at the point of ingestion.

Rebalancing would depend on the shard count of each index, OS would move shards that we part of an index with shard count > 1. You can force the movement of shards with the below api call.

POST /_cluster/reroute
{
“commands”: [
{
“move”: {
“index”: “myindex”,
“shard”: 0,
“from_node”: “node1”,
“to_node”: “node2”
}
}
]
}

5

IF you think it could be error prone, i’ve got no issues creating some datanodes, completing the migration to datanode done in the graylog GUI, and going from there?

I’d create my 3 datanodes, do the migration and any version upgrades that need doing.

My only sticking point then would be to add in some more graylog+mongodb servers as the front end servers.

That way, i would have all my data migrated into the new datanode servers, and i’d be able to replace the ‘old’ server with the new ‘front end’ graylog+mongoDb servers

I’ve been running into some problems trying to test an opensearch migration to the fresh datanode manually, so this could be easier if it all works

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.