Elastic Restore Help Please

I’m migrating from Graylog 3.2.x to 4.x and I have made an elastic backup from 3.x.

I have added the elastic repo on the new running and logging server. It has been collecting logs for about a week. I’m about ready to pull the trigger on restoring the old data. But I need a sanity check before I blow away a week worth of logs. I’m not exactly sure what happens now. Will I see a new index appear in Graylog after the restore? Will I accidentally erase the new stuff restoring the old? Will Graylog even be able to search the restored indices, or is something required to see them? Should graylog-server be stopped while the restore runs?

Help???

#> curl -XGET ‘http://localhost:9200/_snapshot/_all?pretty
{
“esrestore” : {
“type” : “fs”,
“settings” : {
“compress” : “true”,
“location” : “/mnt/elastic/es-backup”
}
}
}

The restore command?
curl -XPOST “http://localhost:9200/_snapshot/esrestore/linuxpoint_snapshot/_restore?wait_for_completion=true

Bump, I’m still hoping for validation I’m not about to destroy my existing logs.

Hello,
Just to touch base with you. I personally have not migrating from Graylog 3.2.x to 4.x. I’ve always upgraded my server/s.

Since our server/s are virtual machine’s we can create a checkpoint incase things go wrong.
I am in the process of migrating my graylog server to a different OS but not yet. I’m actually waiting to see what happens with your environment :slight_smile:

Not exactly what I was hoping to hear :roll_eyes:

Hmmmm…I’ve not done a restore from a snapshot before, but I think you may be able to avoid overwriting existing data by renaming the indices on restore (see Restore a snapshot | Elasticsearch Reference [7.10] | Elastic). They’d be under a different index, but should still be present and available search if I’m reading their docs correctly.

My thinking was renaming would not be required because of the GUID. But since the default index seems to have a prefix of “graylog” I may be wrong. I guess renaming is the best option since I don’t know. I suppose I’ll do something like a “7” suffix (RHEL7).

“rename_pattern”: “graylog_(.+)”,
“rename_replacement”: “graylog7_$1”,

There is also a “Restored Archives” index on the new one that says “read only” and it’s empty??

My first attempt at a restore was an unmitigated disaster. Had to restore to a 10 hour old backup. It could be I simply had a missing comma in the JSON. The rename commands were totally ignored, and everything went to hell in a Fed-Ex truck very quickly.

curl -XPOST “http://localhost:9200/snapshot/esrestore/linuxpoint_snapshot/restore?wait_for_completion=false" '{
“indices”: “graylog_16,graylog_17,graylog_18,graylog_19,graylog_20,graylog_21,graylog_22,graylog_23,graylog_24,graylog_25”
“ignore_unavailable”: true,
“include_global_state”: false,
“rename_pattern”: "graylog
(.+)",
“rename_replacement”: "old_graylog
$1”
}’

I’ll be making another attempt today. Not only was a comma missing, but more importantly I was missing the -d switch in curl. #eyeroll

I have managed a successful data restore into Elastic on the new server. And finally managed to get the indices to be renamed. So at this point the restored indices “old_graylog_x” do NOT appear in Graylog at all. The question is how do I go about adding them in Graylog? It has a wizard to create a new index set. But these indices already exist. How do you add them if they are already there? And can they be added read-only?

# Restore indices
#!/bin/bash

curl  -X POST "http://localhost:9200/_snapshot/esrestore/linuxpoint_snapshot/_restore?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'
{
	 "indices": "graylog_16,graylog_17,graylog_18,graylog_19,graylog_20,graylog_21,graylog_22,graylog_23,graylog_24,graylog_25",
	 "ignore_unavailable": "true",
	 "include_global_state": "false",
	 "rename_pattern": "graylog_(.+)",
	 "rename_replacement": "old_graylog_$1",
	 "include_aliases": false
}
'
1 Like

BTW, in Googling getting at other indices in Elastic (without the graylog_ prefix), other Graylog staff have said this is impossible. So now I’m really intrigued what the hell I do. @aaronsachs said they should be available under a different index. But I found other staff members saying you can only have the one prefix. So having old_graylog as a prefix makes them unreachable. Or is this a capability that has been added since they said that?

Hi @mntbighker. Thanks for your patience here–I’ve been out since one of my kiddos had surgery and haven’t been able to reply. Let me test in the lab, and I’ll check back in. AFAIK, this should be doable, but maybe someone like @konrad or @mpfz0r might have a better idea in the intervening time.

Thanks, the graylog_xx indices are restored, but I have my concerns that the gl-* indices that were not restored contain essential metadata for the logs to be added in Graylog. It’s beginning to look like the only supported restore method is supposed to entirely replace the fresh installed indices with identically named ones. Complete with all related indices. It seems like (at least in the past) there was simply no way to add the indices from another Graylog instance to an existing one. Or if it’s possible, you don’t have a method documented anywhere.

Any ideas yet @aaronsachs , @konrad or @mpfz0r ?? What we are talking about here is really the ability to combine multiple Graylog instances. This seems to me like a use case that should be documented somewhere. Recovering from a disaster is a process that, to me anyway, looks markedly different from building a fresh instance to migrate your data to. One is planned, and the other is not.

As you can see, we have another thread started on this same topic.

@mntbighker I’ve not forgotten about this–it’s just taken longer than I expect because my normal day-to-day duties have kept me pretty busy. I want to ensure I understand what you’re expecting to see. My read is that you’re expecting to see the renamed Graylog indices (both those prefixed w/ graylog- and gl_) just show show up in Graylog–is accurate? If so, there’s additional steps that have to happen for an index to show up in Graylog, and those additional steps are what I was unaware of earlier in the post.

I’ll start this this: while an index can be snapshotted and restored to Graylog, Graylog on its own has no concept that the index exists, effectively resulting in it not being visible in the UI. That’s because an index set is required to inform Graylog that the index exists and should be read by Graylog. Graylog’s a bit dumb in this respect.

So let me walk through the test that I set up. I started by doing a pretty standard snapshot:

curl -X PUT "localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d '
{
    "indices": "graylog_1",
    "ignore_unavailable": true,
    "include_global_state": false,
    "metadata": {
        "taken_by": "aaron",
        "taken_because": "testing for community issue",
        "date": "2021-04-09"
    }
}
'

No issue there.

I then restored the snapshot to another Graylog deployment:

curl -X POST "192.168.156.181:9200/_snapshot/my_backup/snapshot_1/_restore?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": "graylog_1",
  "ignore_unavailable": true,
  "include_global_state": "false",
  "rename_pattern": "graylog_(.+)",
  "rename_replacement": "test_graylog_$1",
  "include_aliases": false
}
'

Again, no issue. I double checked to see that the indices existed in the cluster:

curl "localhost:9200/_cat/indices?pretty"                                                                                                                                      
green open restored_graylog_1 _5KCzSLnSG6HWVq1KOz8IA 4 0 20000601 0 7gb 7gb
green open test_graylog_1     VKDk6ZQ3Qq6R0Ytnp0G0Ew 4 0 20000601 0 7gb 7gb

Excellent–the indices exist, but they’re not showing up in Graylog:

So I created a new index set:

I’ll note that after saving, I could see the index set, but it didn’t seem to indicate that there was anything there until I clicked on it:

So the missing piece is indeed the Index Set–did you happen to try adding one for the graylog_ prefixed indices that you restored to see if that enabled the indices to show up?

2 Likes

Ok @aaronsachs, I ceated a new index as you showed, with my restore prefix. For the record I did NOT restore any of the “gl_” indices. I then looked at Overview with no sign of trouble. Then back to Indices, and bingo, it showed the 10 indices with millions of events. The data was entirely useless, but it kindly informs you to recalculate index ranges. That happened shockingly quickly, and after that all looks good to go. I was able to search from last May to August and up popped the results. I’m still not sure I have a 100% healthy Graylog. There are a few niggling issues remaining as a result of doing the mongo restore presumably. But I would say it’s 98% operational, and collecting logs. And the old data is available to search. In general I think Graylog needs a well documented process for combining Elastic data, which is effectively the same thing as a disaster recovery. That being said, I appreciate the help. I can finally move on to bigger fish waiting to be fried :wink: