I have tried to migrate data to fresh new Elasticsearch cluster/Graylog (from all-in-one setup to distributing setup). Version are the same for Elasticsearch (6.8.10) and Graylog (3.3.1). Used the following steps:
Created snapshot on the old Elasticsearch via API
Restored snapshot on the new Elasticsearch via API (can see indices on ES)
Created Index Set using indices prefix in new Graylog (can see indices count, size and also documents count)
Recalculated index ranges on new Graylog for Index Set
After these steps, still can’t view any data using search on new Graylog. Trying to search for “all in messages”. Can see fields inside restored index on new Graylog (fields filter) and also can view data by using Elasticsearch API (/_search?q=) on new Elasticsearch cluster.
Do you have any thoughts what I’m missing or what need to do to use Graylog search to access restored data ?
No errors in Graylog server.log at all - there is INFO about Rebuild.
INFO [RebuildIndexRangesJob] Done calculating index ranges for 184 indices. Took 15542ms.
Size of restored indexes was about 400 GB. Created a stream and rule to route messages to index I have restored. New events are searchable, but its different index ID - autoincremented - but same prefix. Did double check of permissions, settings, etc. Also closed and reopened few old indices to see if it can help, but not. Performed a search from one of Graylog node to Elasticsearch.
curl -XGET 'http://IP:9200/_search?q=test&pretty'
Result contains events in past, but same time window is not searchable in Graylog.
@nirgil, I’m not sure if it will be valuable, but if you enable developer mode in your browser and capture the API call can you see if it’s doing something strange with filtering? Maybe it’s constraining by index id?
@ttsandrew, nothing weird inside execution API call, tried to compare search output with data contra search output without data and nothing additional or pointless. When searching last 6 months data, only new data visible.
what you have done is not really supported any many items could have gone wrong here. Without looking into very specific sections to find the real reason it is hard to tell what the problem is.
A better solution would have been to “extend” the given Graylog with new servers and after that decommision the old ones. That way you would have preserved all IDs and configuration. Because that would have been shifted from one to the other server.
Understand, that there is a “deeper” relation in Graylog to Elasticseach indices. But I don’t want to do anything special - I hope. Just grab a data via Elastic’s snapshot and transfer data to different Elastic cluster, where is also different Graylog. I can work with data via Elastic API, or Kibana, but not with Graylog search (what is using same way to access data). This is not only about data migration, but also about data recovery.
If I understand correctly, there is no supported way how to restore data (indices) except Archive/Restore in Enterprise version ? So any restore of any snapshot made by Curator/Elastic is not possible and can end with an unexpected condition, right ? If so, it is quite limiting o/
Everything that is done outside of Graylog with the data is something that is not supported at all.
When you manipulate or move data you have multiple pitfalls and problems you can run into, that is the reason non of them are called supported.
You need to have the index sets with the right prefix and numbering, you need/should have the same IDs for Streams to maintain the authorization settings. You might have in the original.
Moving data from one Graylog to another (or restore archived data) is the official way the only supported.
I know some people in the community have build some kind of archiving themself - but they might not like to share that at all (or are no longer active here).
Well, index set existed and also right prefix (Graylog was able to see indexes) and continued incrementing numbering. It might be possible, that an unknown Stream ID caused the empty search result. I’ll try experiment with it bit more. However, I’d expected, that search all messages for all the time should return all data ;] But you gave me some points, thanks.
