I am having trouble linking my data back to the indexer. I have copied all of the log files from /var/lib/elasticsearch/graylog/nodes/0/indices, but now what do I do to keep this data and start a new index? I am trying to keep the indices separated by disk, is this possible? I tried mounting the disk into the above folder with no success, I even created a new index and copied it there but it doesn’t show up. Any help would be greatly appreciated. Thanks in advance.
EDIT: My old server crashed so I copied all the configs over and the new server is running just fine, just no old data.
first I guess you are using the OVA and you do not have any other backup of your data available.
You would normal create a snapshot and restore that like it is described in the elasticsearch documentation - but if you need to recover from a situation you described the only option is to get all files and directories from the elasticsearch data path, that is /var/opt/graylog/data/elasticsearch in the OVA and not the patch you had restored.
Moving that back could work, but without any guarantee. Other options might be available but not very easy to apply.
I am using a manually installed graylog with ubuntu 16.04, so my path is /var/lib/elasticsearch/graylog/nodes/0/indices/. I have the data already on another disk, what I am asking is if there is a way to point elasticsearch back to those log files so I can search them with graylog. My old server crapped out on me so I just copied the files and I would like to now migrate (import?) them into the new graylog server.
no - import them is not possible as Elasticsearch did not have the option to just move datafiles around and make them “visible” on other Systems.
As already said, your only option is to replace the data path with the old data you have or hope that it might work to add a new data path but you are way beyond something that is supported.
I mounted it in the path, new server wasn’t happy with it. It’s all good. I still have the old server with the data still on it as well, Maybe I will try to do an export. Is there a way to point to multiple elasticsearch indices? Ideally I would like to be able to reference it from the new server if at all possible. I guess what I am asking is if I properly export it from the old server via elasticsearch, can I then import it into the new elasticsearch on a different mount point other than my path to indices?
just moving files should work, of course there could be corrupted data if the server broke when writing the data, some lockfiles could prevent the elasticsearch server from starting, if you want graylog to run as before you also need mongodb data from previous server
for non-OVA setup i suggest deploying graylog through docker, updating to newer versions of graylog, data management and backup is much easier when you get the grasp of it
I have everything else copied over, the new server is pretty much a copy of the last, except the data. All of my dashboards and other settings were carried over. The old server is actually still able to run, it just has some corrupt files when the server ran out of HDD space so the OS has issues trying to update or grab packages, etc. I am not new to Linux, it was just much easier and less time consuming to build a new server. I just would like to have access to the data without starting the VM on the rare occasion I may need or want to. Plus this is a good learning opportunity with elasticsearch.
If your servers are on the same network, the easiest way would be to make a cluster of the new and old servers. Set elasticsearch replicas to 1, add all new elasticsearch nodes to the same cluster, wait that the replication is finished, then shut down the old servers one at a time. Wait that the cluster becomes green before shutting down the next node.
Then add new mongod instances to the mongo replica set. Change graylog conf to use the new servers. Then remove the old mongod servers from the replica set.
The graylog servers could be migrated the same way; the only difference is that the master node needs special care. I don’t know this, perhaps some other person knows, if it is enough to shut down the old master and just start the new one.
