Indices issue, Graylog no longer showing logs

[martineznet]

I am running 3.0.2. My server lvm became full and so my graylog server stopped working. The web page would not even come up because it was on the same lvm that was full. I deleted one really large elasticsearch log file to free up space. Now I can access the webpage but no logs are being shown despite graylog reporting in the GUI that it is receiving messages and the inputs are running.

I want to delete some of my indices to free up space but I don’t understand how to do that or even which ones to delete graylog_0, graylog_1, … graylog_6?

I looked at the documentation at this link:

http://docs.graylog.org/en/3.0/pages/configuration/index_model.html

I don’t see the same options to work with indices though. The link shows these options:

But this is what I see:

No logs being shown…

I know there is a command line way to delete the indices but I just don’t know which ones to delete.

graylog-server]# curl http://localhost:9200/_cat/indices
green open graylog_6 c08EFCCQSfCOUOHlI-FMGQ 1 0 15351294 0 22.7gb 22.7gb
green open graylog_1 w8YmAdDNQ6e8-ua2XGhOmQ 1 0 178 0 143.6kb 143.6kb
green open graylog_3 cackD1YeR46BN0ixyN3Hgg 1 0 20000769 0 31.9gb 31.9gb
green open graylog_4 yXfNafLjT4WLKBS2zGuH1A 1 0 20000030 0 31.1gb 31.1gb
green open graylog_5 cjh0Yy36TE6NvH0ykMG31g 1 0 20000015 0 30.7gb 30.7gb
green open graylog_0 bUU9D9HiSWmvT4uPugKJkw 1 0 1229 0 1.2mb 1.2mb
green open graylog_2 11DsiTZLQ5eGMwy6kc5X6w 1 0 20000043 0 26.9gb 26.9gb

Any idea where I should look to see why logs are no longer showing?

[Ponet (Jesse Hills)]

If you’re running ES on the same host that ran out of disk space, your ES index is likely still set to read-only.

[martineznet]

Ok do you know how I can change elasticsearch to being able to write again?

[martineznet]

Also how can I delete old indices to free up more space?

[jan (Jan Doberstein)]

you should first resolve your unassigned shards:

How to resolve unassigned shards in Elasticsearch

after that free up space, make the indices writable again and you are up and running again

Jan

[Ponet (Jesse Hills)]

How to undo setting Elasticsearch Index to readonly?

[martineznet]

Actually now I can see the logs again. Not sure why. Also why do you say I have unassigned shards. I tried the below command in the link you sent above and I received the below. Not sure if this is saying anything is unassigned though.

graylog ~]# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 161 100 161 0 0 1354 0 --:–:-- --:–:-- --:–:-- 1364

Also how do I free up space? Can I delete old indices and if so, how do I know which ones are the old ones?

[jan (Jan Doberstein)]

your first screenshot show unassigned shards - that is why I have written that.

You can delete indices from the Graylog UI or via the Elasticsearch API - whatever serves you better.

[martineznet]

Ok, where in the screenshot does it say a shard is unassigned?

Also my issue is that

1. I don’t know which indices I should be deleting. Does it not matter what I delete?
2. I don’t see where in the GUI the indices can be deleted (see screenshot above, nothing there shows option to delete an indice)

[jan (Jan Doberstein)]

the yellow bar under the green bar is telling me that.

Expand the first index in the list - you will have the option to delete that. You can delete all indices, but not the active write index.

[martineznet]

Ahhhh… I see. The image you were looking at I took from the document link in my first post. That isn’t my server. I was using that to compare with what I was seeing on my own server GUI which has far less detail (see below)

So I don’t see anyway to delete indices from the view above.

[jan (Jan Doberstein)]

you need to click on the link name “default index set” what will give you a different view.

[martineznet]

Nice… This is what I see now. I can see the option to delete. Great!

I know not to delete the active indice but does it matter which of the others I delete? Is there a way to know which one is the oldest or would that just be graylog_0, graylog_1, etc?

[jan (Jan Doberstein)]

Graylog starts the sequence from 0 - as very first index and count up from that point.

So ig you have the index graylog_0 that is the oldest one in your environment.