I mistakenly let the default index run out of space which caused all the streams to stop working.
I’ve been able to create new indices for the affected streams, but the default index remains in a state that isn’t taking in new messages.
Since I’ve added space, I expected the index to start taking in new data, however it isn’t.
I’ve adjusted the default index, reducing the Rotation Period (Index Time) down and ‘Recalculated index ranges’ as well as ‘Rotated active write index’ expecting that to trim the index but nothing seems to have changed. The size of the index and document count on the index remain the same.
I’d appreciate any guidance that might be offered in order to resolve the situation.
Right - I think I’d found that exact post and I’d tried that earlier, the response was “{“acknowledged”:true}”. Sadly that hasn’t seem to done anything with respect to getting the index to start taking in any data.
I’ve also tried reducing the number of shards and the retention time, then done the “Recalculate index ranges” and "Rotate active write index " trying to age off old data, but nothing has changed on the index.
Am I missing another step in trying to the index active again?
Here is some more information on watermarks and clearing them.
If you want you can post the commands and their results for better clarification of what you are seeing - use the forum tools ( </> ) for formatting clarity…
Hello and thanks again for your help. I tried the curl command from the article you pointed me to but that didn’t work
[curl -XDELETE --header ‘Content-Type: application/json’ http://localhost:9200/graylog_0 /_search -d ‘{ “query”: { “range”: { “createdtime”: { “gte”: “1y” } } } }’]
I eventually ended up running this, hoping I’d only remove a subsection of older data from the index:
[curl -XDELETE --header ‘Content-Type: application/json’ http://localhost:9200/graylog_0 /_search -d ‘{ “query”: { “range”: { “createdtime”: { “gte”: “1y” } } } }’]
Sadly that removed everything from the index and I didn’t see any data coming into the index until I restarted the system. At this point I won’t be trying to recover the data and will just move on from here. Lesson learned - DO NOT let space get low!
There were plenty of warnings so I don’t fault Graylog at all - just ME. However when I get time I’ll be trying to make sure I CAN recover from something like this in the future.
