There were 204,800 failed indexing attempts

Graylog Central (peer support)
1

Hi

We run Graylog 3.2.4 (Private Build 1.8.0_242 on Linux 4.15.0-91-generic)

we have 1 node

in the web interface I read there were 204,800 failed indexing attempts in the last 24 hours.

looks like 6 GB free on the disk

I see a lot of messages saying: Graylog deflector is pointing to not the newest one

How to solve the issue ?

Many thanks for your time

Kind regards

Olivier

2

he @servicedesk

check your Graylog server log - the rotation of indices was not done proper. You need to find out why that did not happen.

3

where to begin checking ?

4

https://docs.graylog.org/en/3.2/pages/configuration/file_location.html

the location depends on your OS

5

2020-06-05T07:11:01.849+02:00 WARN [Messages] Failed to index message: index=<graylog_0> id= error=<{ā€œtypeā€:ā€œcluster_block_exceptionā€,ā€œreasonā€:ā€œblocked by: [FORBIDDEN/12/index read-only / allow delete (api)];ā€}

What is the command to add the permissions ?

6

please use the search in this community.

Your Elasticsearch is running in high/low/flood watermark what means you have not enough space anymore. This is why it is read only. Add more space, delete data and make it read/write again.

1 Like
7

I have tried some commands to make it read write again and they failedā€¦

8

curl -XPUT -H ā€œContent-Type: application/jsonā€ https://localhost:9200/_all/_settings -d ā€˜{ā€œindex.blocks.read_only_allow_deleteā€: null}ā€™

gives

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

9

I tried
curl -X PUT ā€œlocalhost:9200/_all/_settingsā€ -H ā€˜Content-Type: application/jsonā€™ -dā€™{ ā€œindex.blocks.read_onlyā€ : false } }ā€™

command is accepted but I donā€™t see any change

no more deflector error. but nothing to show in the gui

maybe I need to wait a certain time for it to show again in the gui ?

I cant find any errors

any help is appreciated

thanks

10

Waitng paid off. I still donā€™t know the command. In the end I restored the vm and resized the disk. Waited for the next rotation and the messages reappeared in the gui.

I still would like to know if we can do one of these

curl -X PUT ā€œlocalhost:9200/_all/_settingsā€ -H ā€˜Content-Type: application/jsonā€™ -dā€™{ ā€œindex.blocks.read_onlyā€ : false } }ā€™

or

curl -X PUT ā€œlocalhost:9200/_all/_settingsā€ -H ā€˜Content-Type: application/jsonā€™ -dā€™{ ā€œindex.blocks.read_onlyā€ : null } }ā€™

11

after you have enough disk space you can use the following command.

curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d'{ "index.blocks.read_only" : false } }'
1 Like
12

I suggest use a monitoring system, to monitor the numbers of index failures. In this case, you will get notification after a few problems.

13

That is a good plan! I will Try to have Nagios Read the server log and look for these lines !

Thanks macko003

14

he @servicedesk

you can query the Graylog server API for that information ā€¦ check the API-Browser for the endpoints you want to monitor.

15

Make sure to check out the Nagios plugin, it also works on Librenms.

You can use the warn and critical flags I added a while ago. I usually set them a few thousand higher than the current levels.

17

Thanks rfinneyā€¦ installing it now !

18

how to install it ? I have installed go and can do a version check. When running the check command it gives
./check_graylog2: line 6: syntax error near unexpected token newline' ./check_graylog2: line 6: ā€™

19

Should be

$ go get github.com/catinello/nagios-check-graylog2
$ mv $GOPATH/bin/nagios-check-graylog2 check_graylog2

Iā€™m not familiar with the Nagios side after that. I use it in Librenms.

20

what is meant by : build it yourself using the go-tools ? Do I need to use the build command ?

21

If I recall correctly go get is all you need. But itā€™s been a while since Iā€™ve used it.