There were 204,800 failed indexing attempts

Graylog Central (peer support)
1

Hi

We run Graylog 3.2.4 (Private Build 1.8.0_242 on Linux 4.15.0-91-generic)

we have 1 node

in the web interface I read there were 204,800 failed indexing attempts in the last 24 hours.

looks like 6 GB free on the disk

I see a lot of messages saying: Graylog deflector is pointing to not the newest one

How to solve the issue ?

Many thanks for your time

Kind regards

Olivier

2

he @servicedesk

check your Graylog server log - the rotation of indices was not done proper. You need to find out why that did not happen.

3

where to begin checking ?

4

https://docs.graylog.org/en/3.2/pages/configuration/file_location.html

the location depends on your OS

5

2020-06-05T07:11:01.849+02:00 WARN [Messages] Failed to index message: index=<graylog_0> id= error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}

What is the command to add the permissions ?

6

please use the search in this community.

Your Elasticsearch is running in high/low/flood watermark what means you have not enough space anymore. This is why it is read only. Add more space, delete data and make it read/write again.

1 Like
7

I have tried some commands to make it read write again and they failed…

8

curl -XPUT -H “Content-Type: application/json” https://localhost:9200/_all/_settings -d ‘{“index.blocks.read_only_allow_delete”: null}’

gives

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

9

I tried
curl -X PUT “localhost:9200/_all/_settings” -H ‘Content-Type: application/json’ -d’{ “index.blocks.read_only” : false } }’

command is accepted but I don’t see any change

no more deflector error. but nothing to show in the gui

maybe I need to wait a certain time for it to show again in the gui ?

I cant find any errors

any help is appreciated

thanks

10

Waitng paid off. I still don’t know the command. In the end I restored the vm and resized the disk. Waited for the next rotation and the messages reappeared in the gui.

I still would like to know if we can do one of these

curl -X PUT “localhost:9200/_all/_settings” -H ‘Content-Type: application/json’ -d’{ “index.blocks.read_only” : false } }’

or

curl -X PUT “localhost:9200/_all/_settings” -H ‘Content-Type: application/json’ -d’{ “index.blocks.read_only” : null } }’

11

after you have enough disk space you can use the following command.

curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d'{ "index.blocks.read_only" : false } }'
1 Like
12

I suggest use a monitoring system, to monitor the numbers of index failures. In this case, you will get notification after a few problems.

13

That is a good plan! I will Try to have Nagios Read the server log and look for these lines !

Thanks macko003

14

he @servicedesk

you can query the Graylog server API for that information … check the API-Browser for the endpoints you want to monitor.

15

Make sure to check out the Nagios plugin, it also works on Librenms.

You can use the warn and critical flags I added a while ago. I usually set them a few thousand higher than the current levels.

17

Thanks rfinney… installing it now !

18

how to install it ? I have installed go and can do a version check. When running the check command it gives
./check_graylog2: line 6: syntax error near unexpected token newline' ./check_graylog2: line 6: ’

19

Should be

$ go get github.com/catinello/nagios-check-graylog2
$ mv $GOPATH/bin/nagios-check-graylog2 check_graylog2

I’m not familiar with the Nagios side after that. I use it in Librenms.

20

what is meant by : build it yourself using the go-tools ? Do I need to use the build command ?

21

If I recall correctly go get is all you need. But it’s been a while since I’ve used it.