Unable to get any data on streams

Graylog Central (peer support)
1

Hi there,

I am facing an issue with Graylog indices, my server HD got full hence Graylog stopped and I could not access graylog console. Since I first had to free up the disk space I had delete few old indices from Curl command line that is

curl -XDELEET http://127.0.0.1/graylog_1 and so on

After this I see the graylog has started however messages were not dumping in any of the indices however I see on the top that messages are being accepted but not sure where are those being stored.

Any clue what could have gone wrong?

2

I recall another recent discussion that mentioned ElasticSearch having lots of trouble after its storage space had filled up. Could be that your Elastic instance has gone tits-up.

3

Well I tried that but nothing worked. I guess reinstalling ES should do the trick? or any other measure you can recommend?

4

what is in your Graylog server.log ? If the disk where the Graylog journal is located was full before the journal actually has grown to its max size, the journal is corrupt and you need to delete the message journal.

1 Like
5

Reinstalling / wiping ElasticSearch is also a bit dramatic :slight_smile: Is this your DEV/TEST environment? Or worse, production data?

6

@Totally_Not_A_Robot That was prod data

@jan - How do I delete the journal data then? to make it work? and my version was 2.4.

Please suggest

7

The question is: WAS it the Graylog journal disk that filled up? Or was it the ElasticSearch disk? The solution differs wildly.

8

Ooops. My bad, didnt read the question correctly.

Yes it was elasticsearch which filled up my disk not journal.

9

Did you try to manually rotate write indexes?
(Indices - click on name - maintenance)

1 Like
10

How do I delete the journal data then? to make it work? and my version was 2.4.

stop graylog delete all content from the folder that is configured as your journal, start Graylog

11

In that case @jan’s suggestion doesn’t apply. But then I’m also not sure on how to proceed, because as I’ve mentioned: I’ve read threads where ElasticSearch itself gets broken from running out of disk space. I reckon you’d do well to also investigate through ElasticSearch documentation and forums.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.