After data recovery not seeing data in Search

Graylog Central (peer support)
1

Hi,
I am working on recovery scenario in case our Graylog server would fail. I am doing snapshots of data and configuration every day. I made a test:

  1. created new Graylog server as appliance
  2. mounted remote directory with snapshots and configuration
  3. copy key (password_secret) from /etc/graylog/server/server.conf
  4. restored configuration with mongorestore
  5. restored snapshot of the data from last day - for backup/restore I use my own scripts, but they are basically the same like https://support.cloudbees.com/hc/en-us/articles/115000592472-Managing-snapshots-of-your-Elasticsearch-indices-
  6. reload server
  7. can’t see any data in Search

Any advices, what I am doing wrong?

I should have backup of data for last two months so I am not very happy about idea, that I would need to do the same 60-times (meaning two monts). Advice, how to do better are welcome.

Best Regards
B.

2

If you have the restored elasticsearch and Graylog connected to that - did you run “recalculate index range”? Because Graylog will check with that command what data is inside of Elasticsearch available.

3

Thanks for advice, I’ve just did that, but all the same, no data in Search.

4

did you checked if Elasticsearch is able to see the data? So you need to look step-by-step where the problem is.

I would first check if the Elasticsearch cluster is having the data. If yes check if Graylog can speak to elasticsearch and see the data (because the index set configuration is pointing to the right indices) … and so on.

5

did you checked if Elasticsearch is able to see the data

I am not sure, whether I check that right. I’ve tried to do the restore of the last day and I checked, that the ES see restored indeces from the snapshot:

root@graylog:/nas/konfigurace# curl -XGET 'http://localhost:9200/_cat/indices'
green open graylog_0          Dfo_baa6QjeWvngl05uwnA 4 0       0 0     1kb     1kb
green open restored-graylog_0 7TcrdkKxRRyzpHpVnmUNJg 4 0 5304870 0   2.7gb   2.7gb
green open restored-graylog_1 EldS8TDjR8CPKZVDGPsqrg 4 0   17895 0   9.8mb   9.8mb
green open restored-graylog_5 7pw-xQbGTJWPtFUVoYMXcA 4 0   94940 0  52.1mb  52.1mb
green open restored-graylog_3 bdUwUxw6Q6axMDj6FH3pPg 4 0  284511 0 154.6mb 154.6mb
green open restored-graylog_2 wVVTNTJOQuOsesrlHNGjyw 4 0  250930 0 136.7mb 136.7mb
green open restored-graylog_4 4vqaSK5cTyiVWbQ8WL_e5Q 4 0  273109 0 148.3mb 148.3mb

but I can’t see these indices in the Graylog (GUI > System > Indices), there is only " Default index set" with index graylog_0. I did “recalculate index range”.

6

to be honest - you have restored the indices with the name restored-graylog_N and you index pattern is graylog_N.

If your indices are not named correctly Graylog will not be able to see them.

7

That was it, the index pattern! Thank you very much @jan!

Rather silly question: I do snapshots every day, restored snapshot from the last day, there is retention strategy (rotate index every week, keep 9 indeces, delete older), but I see even older data. No big deal, because it doesn’t take much data, but I would like to understand it better. I have to keep at least two months of history.

8

ok - if you see older data, but the indices are rotated out, it is very likely that you have devices with wrong timestamps.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.