Datanode startup error

[workinonit]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident: two of my datanodes are refusing to start with the following errors:

Startup error:
org.graylog.datanode.filesystem.index.IndexerInformationParserException: Failed to open index for read

Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in MMapDirectory@/var/lib/graylog-datanode/opensearch/data/nodes/0/indices/YxW2JGwvSLKhPrzV66NUGg/1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@273fa9e: files: [write.lock]

2. Describe your environment:

• OS Information: Ubuntu

• Package Version: 6.3.4

• Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem? Referred to my notes about how to view Opensearch cluster but with datanode unable to start that isn’t going to work

4. How can the community help? Tell me how to remove the write.lock? Tell me how to fix this issue.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[drewmiranda-gl (Drew Miranda)]

Howdy! Can you provide some more context and informaiton?

• How long have you been running datanode?
• When did this issue start?
• where any changes made to the systems leading up to this issue?
• were there any hardware issues or other errors?

Based on what you have provided, data node is saying it cannot start because its child process, OpenSearch, is unable to write to path /var/lib/graylog-datanode/opensearch/data/nodes/0/indices/YxW2JGwvSLKhPrzV66NUGg/1/index .

Can you verify:

• this path exists
• you can read/write to it
• there was no disk failure
• there was no file corruption

[workinonit]

Thanks for the reply. I ended up just nuking the entire directory and things seem to be working on the 2 nodes that were having the issue. Clearly I lost data but we are in the fortunate position to not have any regulatory compliance that we must maintain the data.