Elasticsearch will not allow me to search anymore

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I ran into an issue with the storage on my server filling up, this caused a backlog of 6.5Million messages unprocessed until I realized there was a problem (not actively monitoring this daily). Expanded storage and it cleared the messages up in a matter of minutes. When I clicked on the “Search” tab again to view new messages coming in I got this message:

While retrieving data for this widget, the following error(s) occurred:
Elasticsearch exception [type=index_not_found_exception, reason=no such index ].

2. Describe your environment:

  • OS Information:
  • Redhat 17.0.2 on Linux 4.18.0
  • Graylog - 4.2.6
  • Package Version:
    N\A
  • Service logs, configurations, and environment variables:

Caused by: org.graylog.shaded.elasticsearch?.org.elasitcsearch.elasticsearchstatusexception: Elasticsearch exception [type=cluster_block_exception, reason=index [graylog_7] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

3. What steps have you already taken to try and solve the problem?
Tried restarting the elasticsearch service, tried restarting the server. Researched the error code but come across a lot of saying to re-index but instructions aren’t clear and I’m not sure if that’s what I need…any help is very appreciated.

4. How can the community help?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hello @jaydons

So after you expanded the volume this create more then one issue, it looks like Elasticsearch went into read mode. What needs to happen is take it out of read mode or you journal will fill up again.

First, I would check the cluster, if ES/OS setting are 127.0.0.1 or localhost use this

curl -X GET "localhost:9200/_all/_settings?pretty"

You should see something like this on you index used.

  "gl-failures_323" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "ignore_malformed" : "true"
        },
        "number_of_shards" : "2",
        "blocks" : {
          "write" : "true",
          "metadata" : "false",
          "read" : "false"   
        },
        "provided_name" : "gl-failures_323",
        "creation_date" : "1661299205649",
        "number_of_replicas" : "0",
        "uuid" : "VcPuCfD0QauIPOWLaEHrkg",
        "version" : {
          "created" : "7100299",
          "upgraded" : "135248527"
        }

If set to “true” then use curl with the PUT commmand.

The curl command would be something like this.

curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d'{ "index.blocks.read_only" : false } }'

Hope that helps

3

Hello,

Thank you for the response!
I ran the command you mentioned and Read is set to false already.

Is there anything else you know of that I could check?

4

Hello,

Have you tried to rotate you index set manually?

Navigate to…

System --> Indices

Click on Default index, there should be a drop down for Maintenance in the upper right side.
Maybe that would clear things up.

Whenyou see this ,Elasticsearch is tell you it is full and read-only-allow-delete block

Somethinglike this.

1 Like
5

Hello,

Thank you very much for this. I’m extremely new to graylog so I don’t know all of these little ins and outs. Rotating the index seems to have done the trick!

If you’re able to answer just so I know better, when rotating the index it appears to have created a new index. Does this make so the old messages are now unreadable? For example I had index(s) Graylog_1-8 and it created Graylog_9. Does this make so 9 is all that’s being read or does the search function show messages from every index?

6

Hey @jaydons

You are correct, sort answer is the default index set can be configured different ways.
The index will rotate as it was configured, this is called rotation strategy. three settings ( size, time, count).Then below that setting you have Rotation period. Tthis setting would be for hour, day, week, etc…

Remember the more indices you the more volume will be used. TBH you may want to read over the documention.

hope that helps.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.