Graylog not indexing


#1

Hi,

The server ran out of space over night but gave it more space but now graylog wont index how do I clear the below?

There were 198,468 failed indexing attempts in the last 24 hours.

srv_std_syslog__42	a2da2d21-155d-11e9-a927-0242ac120002	{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

Thanks


(Jan Doberstein) #2

fix elasticsearch that it accept messages again


#3

You would not have that handy for a elastic noob:) am using docker.


#4

for noobs? say hello for your data, and start a new empty server.
for users? say hello for your data. restart elastic with empty database.
for pros? start a new empty server, wait for replication or restore the database :slight_smile:

If you ran out of space the elastic database will have problem.
The elasticsearch designed for distributed data sorage, so if you have a cluster a node loos won’t be a problem. But if you store all data on one node…
Maybe you can do some research about how can you fix, but I administrate only HA systems, so I didn’t do it. I just drop out the wrong server, and wait for recover.


(Tess) #5

Wait, are you saying Elastic simply cannot be saved after it has run out of disk space? Not even once? that would be very odd… Then again, you absolutely know a lot more about ES than I do :slight_smile:


#6

Onece I ran out of disk. After a restart elastic report a broken database, so I drop the elastic directory and wait for the replication, that was a test System. Now I monitor my systems better.


#8

Ok here is a solution. But it’s also mention data loose.
http://www.flax.co.uk/blog/2016/04/21/running-disk-space-elasticsearch-solr/

  1. jan. 11., P 14:32 dátummal Tess via Graylog Community graylog@discoursemail.com ezt írta:

#9

hi,

you can just use CURL commands for the ES to check which indices are broken, and delete those. If that works out, you can get away with only partial data loss.


(Tess) #10

Yikes, that’s a pretty scary prospect though. One would hope there are ways of restoring broken indices. Some people rely upon the data in ElasticSearch for auditing etc.


(system) closed #11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.