Graylog index rotation not working

Graylog Central (peer support)
1

Hello,

Previously my index was set to store logs for about 4 years. That was my first, initial, and not quite a wise setting. But it was, and it worked.
Unfortunatelly partition where folder /var is mounted (where I think logs from Graylog are stored) is occupied in 55%. I think this causes that index in Graylog won’t accept new logs - it shows " There were 204,800 failed indexing attempts in the last 24 hours."
Few days ago I changed index roration period from 4 years to P1W (7d, 7 days) but It does not delete old logs. Previously index has 20.7GB and today also it have exact the same size.

What should I change to delete old logs?

Best Regards

graylog_index
graylog_index.png1176×179 16.3 KB

2

you have your elasticsearch on the same host?

Did you checked the elasticsearch logfiles? Maybe that is read-only. But without the log that is guessing only.

3

Yes, elasticsearch is on the same host, and it is in read only state. I know that and I thought this is due to the decreasing amount of free space. I tried to fix this by reducing time for index rotation.
Am I wrong?

Below are last entries from server.log

WARN [Messages] Failed to index message: index=<graylog_0> id=<052224a0-b375-11e9-bcfb-00155d034305> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>

ERROR [Messages] Failed to index [19] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

4

you need to make elasticsearch writable again that Graylog is able to do any index manipulation.

5

Yes Jan - I know that cause of this problem is elasticsearch in read only state. But I don’t know how can I change that state … :confused:
Any suggestions how to solve this problem?

6
1 Like
7

@Karlis thank you for reply.
After use that command I got:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "resource_already_exists_exception",
        "reason" : "index [settings/5GJ9lZHLTuGvqBfZjW6PLw] already exists",
        "index_uuid" : "5GJ9lZHLTuGvqBfZjW6PLw",
        "index" : "settings"
      }
    ],
    "type" : "resource_already_exists_exception",
    "reason" : "index [settings/5GJ9lZHLTuGvqBfZjW6PLw] already exists",
    "index_uuid" : "5GJ9lZHLTuGvqBfZjW6PLw",
    "index" : "settings"
  },
  "status" : 400
}

Hm.
I think this is job for elasticsearch developer, not for user who want to log simple data do GrayLog.

8
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.