Index file failures

Graylog Central (peer support)
1

Hi,

I’m an running Greylog 3.0.2 as an appliance. In the last few days I noticed there are no messages that are being captured as a result of index failures.

The error messages are:

There were 203,523 failed indexing attempts in the last 24 hours.

|a few seconds ago|graylog_0|cb854853-0417-11ea-890a-000c299c48cb|{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}|
|a few seconds ago|graylog_0|cb854850-0417-11ea-890a-000c299c48cb|{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}|
|a few seconds ago|graylog_0|cb854852-0417-11ea-890a-000c299c48cb|{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}|
|a few seconds ago|graylog_0|cb854851-0417-11ea-890a-000c299c48cb|{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}|

I don’t know why these errors have appeared. It is possible the index files are full, although I would have thought graylog would handle log rotation.

Any tips or assistance would be great

2

… I may have posted the question prematurely. I found another post where the following command resolved my issue:

curl -XPUT -H “Content-Type: application/json” http://localhost:9200/_all/_settings -d ‘{“index.blocks.read_only_allow_delete”: null}’{“acknowledged”:true}

  • It is likely the index files hit a watermark and didn’t rotate correctly ?
  • Should I increase the filesystem on which the index files reside or adjust the watermark threshold ?
3

he @jgiovann

the calculation of index rotation might not be right for the available disk space. That is not done automatically. You need to calculate and configure on your own.

After you have released disk space the posted command is the right that elasticsearch accept messages again.

What the right solution for you is - add more disk space or change watermark settings highly depends on your needs. That can’t someone without your knowledge answer.

4

Hi Jan,

Thanks for the tip. I don’t have the option of increasing the disk space but I’ve re-configured the default index file to rotate after a certain size.

I’ll then look at re-configuring the watermark. Do I do this from the web interface or do I need to configure from the command line ?

5

with the current version you need to adjust the watermark configuration from the command line - or set this is in the elasticsearch configuration file.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.