Restore data on different cluster - no data in search


I have tried to migrate data to fresh new Elasticsearch cluster/Graylog (from all-in-one setup to distributing setup). Version are the same for Elasticsearch (6.8.10) and Graylog (3.3.1). Used the following steps:

  1. Created snapshot on the old Elasticsearch via API
  2. Restored snapshot on the new Elasticsearch via API (can see indices on ES)
  3. Created Index Set using indices prefix in new Graylog (can see indices count, size and also documents count)
  4. Recalculated index ranges on new Graylog for Index Set

After these steps, still can’t view any data using search on new Graylog. Trying to search for “all in messages”. Can see fields inside restored index on new Graylog (fields filter) and also can view data by using Elasticsearch API (/_search?q=) on new Elasticsearch cluster.

Do you have any thoughts what I’m missing or what need to do to use Graylog search to access restored data ?

Thank you

Hello @nirgil, welcome!

Are there any errors in your Graylog server.log indicating a problem? Permissions errors, etc?

Hello @Andrew,

thanks for response.

No errors in Graylog server.log at all - there is INFO about Rebuild.

INFO [RebuildIndexRangesJob] Done calculating index ranges for 184 indices. Took 15542ms.

Size of restored indexes was about 400 GB. Created a stream and rule to route messages to index I have restored. New events are searchable, but its different index ID - autoincremented - but same prefix. Did double check of permissions, settings, etc. Also closed and reopened few old indices to see if it can help, but not. Performed a search from one of Graylog node to Elasticsearch.

curl -XGET 'http://IP:9200/_search?q=test&pretty'

Result contains events in past, but same time window is not searchable in Graylog.

Bit confused … o/

@nirgil, I’m not sure if it will be valuable, but if you enable developer mode in your browser and capture the API call can you see if it’s doing something strange with filtering? Maybe it’s constraining by index id?

@ttsandrew, nothing weird inside execution API call, tried to compare search output with data contra search output without data and nothing additional or pointless. When searching last 6 months data, only new data visible.