Index Corrupt after server shutdown

Graylog Central (peer support)
1

I gracefully shutdown my graylog server > opensearch > mongodb and reboot. After the reboot, all indexed segment gone and from the search page of graylog. Index is still here but all historical indexed segment disappear. However, the index shard is still in the harddisk but somehow Opensearch cannot link them together,

I ran [ wget -q ‘http://localhost:9200/_cat/indices?v’ -O- ] also cannot return old archived index segment . latest segment of all indexes reset to 0, but no red index.

Could I still recover my previous indexes ?

Env,
Graylog 5.1.4+6fa2de3 on xxx (Eclipse Adoptium 17.0.8 on Linux 5.14.0-284.18.1.el9_2.x86_64)

2

Hey @nkmfree

What do you see when executing this on your GL node?

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
3

image
image1069×533 19.8 KB

Hi , thanks for your help. attached is the output of command

4

Oh… Is this a cluster?

For the message above

a copy of this shard is aready allocated to this node

Opensearch never assigns the replica of the same primary shard on the same node for high availability reasons.

By chance you see anything in the logs?

1 Like
5

no, just use single server for graylog server/opensearch and mongo DB

there is no special error but the following

image

1 Like
6

Hey,

So the server shutdown and Opensearch can not find your index sets, hence why there set to 0.

Check your data path that is configured in Opensearch YAMl file. Should be something like this path.data: /var/lib/opensearch

If the data path moved or was on a different volume and the volume detached this could happen. They should still be there unless there were deleted. But now if the new index set were create it probably wont attach to those original ones.

When you run this command what do you see?

curl -XGET http://localhost:9200/_cat/shards
1 Like
7

image
image818×724 54.8 KB

Attached is the output, seem all are newly created shard. prev shard disappeared

data path have not changed, can I import it manually ?

8

I see, now thats wierd.

I have no idea why that happened to you. Like I mentioned , the only way I know of why you instance created a new Index set would be the Data Path, or you data was on a different volume and was detached. I never seen a index vanish from a reboot unless a configuration was made but not save and reboot set configuration back prior.

9

I finally figured out why the index disappeared. after the server reboot , I found the previous indices in /var/lib/opensearch/nodes/nodes/0 !! ( 1 more nodes in the path depth )
I think Opensearch found that no indices could be found in original path in /var/lib/opensearch/nodes/0, it automatically recreated another set of index. Can I simply copy the index shard to original path and let Opensearch import it automatically ?

10

Ah that makes sense, so it was the data path location.

Not that I know of, once in a index set the only way is through snapshots/Restore

Couple options:

1.you can either reset the data path to your orginal settings and restart Opensearch or leave it as is.

2.Next option is alittle harder. create a snapshot of the new index set that opensearch created then adjust you data path back to the original settins and restart opensearch. you can restore the new snapshot to a different index set.

Here is some info on that.

EDIT: One of the reason I suggested to use the snapshot is beacue now you have dups of thos eindex sets, but I havent tried this you might be able to do something like this, it may not work. your find it in this documentation

11

ok, thx for your help!

1 Like
12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.