Rotate active write index not working

Graylog Central (peer support)
1

incident: I am done with the upgrade from Graylog 5.2 to 6.0. And i realized that my indices are not rotating. If I click on the Rotate active write index, nothing happens. What could be the problem? This could be a permission problem. How can i set the right permissions on my incides folder or do I have to do it manually?

  1. Describe your environment:
    OS Information: ubuntu 22.04

Package Version:
graylog-5.2-repository/now 1-2 all [telepítve, helyi]
graylog-server/stable,now 6.0.1-1 amd64 [telepítve]
mongodb-mongosh/focal/mongodb-org/7.0,now 2.2.5 amd64 [telepítve]
mongodb-org-database-tools-extra/focal/mongodb-org/7.0,now 7.0.9 amd64 [telepítve]
mongodb-org-database/focal/mongodb-org/7.0,now 7.0.9 amd64 [telepítve]
mongodb-org-mongos/focal/mongodb-org/7.0,now 7.0.9 amd64 [telepítve, automatikus]
mongodb-org-server/focal/mongodb-org/7.0,now 7.0.9 amd64 [telepítve]
mongodb-org-shell/focal/mongodb-org/7.0,now 7.0.9 amd64 [telepítve]
opensearch/stable,now 2.13.0 amd64 [telepítve]

Service logs, configurations, and environment variables:
Not sure what you need here.

  1. What steps have you already taken to try and solve the problem?
    Tried to restart opensearch and graylog services and check the graylog.log file but no success. I had one error log early this morning but do not have any idea what is the problem:
    grep -i error /var/log/opensearch/graylog.log
    [2024-05-14T05:38:54,926][ERROR][o.o.a.a.AlertIndices ] [prodlog0001l] info deleteOldIndices
    [2024-05-14T05:38:54,926][ERROR][o.o.a.a.AlertIndices ] [prodlog0001l] info deleteOldIndices

  2. How can the community help?
    I would really appreciate any help whit this topic. Thanks!

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hey @Zoltan

What do you see in Graylog’s log file or MongoDb? Maybe something in there might help.

3

Hey @gsmith,

Thanks for you reply!
I am not sure what to lookup but found these error’s:

In graylog’s log:
sudo grep -i error /var/log/opensearch/graylog.log
[2024-05-16T06:51:05,074][ERROR][o.o.a.a.AlertIndices ] [prodlog0001l] info deleteOldIndices
[2024-05-16T06:51:05,074][ERROR][o.o.a.a.AlertIndices ] [prodlog0001l] info deleteOldIndices

In mongod.log:
sudo grep -i error /var/log/mongodb/mongod.log
{“t”:{“$date”:“2024-05-12T07:37:45.129+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=7509M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],”}}
{“t”:{“$date”:“2024-05-13T16:26:50.638+02:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2024-05-13T16:30:17.104+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=7477M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],”}}
{“t”:{“$date”:“2024-05-13T16:44:33.673+02:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2024-05-13T16:49:46.889+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=7477M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],”}}
{“t”:{“$date”:“2024-05-13T17:18:39.461+02:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2024-05-13T17:18:51.572+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=7477M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],”}}
{“t”:{“$date”:“2024-05-13T17:31:09.110+02:00”},“s”:“I”, “c”:“CONTROL”, “id”:23377, “ctx”:“SignalHandler”,“msg”:“Received signal”,“attr”:{“signal”:15,“error”:“Terminated”}}
{“t”:{“$date”:“2024-05-13T17:31:20.166+02:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=7477M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],”}}

In graylog_server.json:
sudo grep -i error /var/log/opensearch/graylog_server.json
{“type”: “server”, “timestamp”: “2024-05-16T06:51:05,074+02:00”, “level”: “ERROR”, “component”: “o.o.a.a.AlertIndices”, “cluster.name”: “graylog”, “node.name”: “prodlog0001l”, “message”: “info deleteOldIndices”, “cluster.uuid”: “-mbGzqiZRmuKROpAzWr9Ew”, “node.id”: “SepbWI_kQJ-3hm1XGzRVQA” }
{“type”: “server”, “timestamp”: “2024-05-16T06:51:05,074+02:00”, “level”: “ERROR”, “component”: “o.o.a.a.AlertIndices”, “cluster.name”: “graylog”, “node.name”: “prodlog0001l”, “message”: “info deleteOldIndices”, “cluster.uuid”: “-mbGzqiZRmuKROpAzWr9Ew”, “node.id”: “SepbWI_kQJ-3hm1XGzRVQA” }

Thanks in advance!
Regards,
Zoltan

4

Hey @Zoltan

I’m unsure about your index rotation. I assume this was working correct before the upgrade? If so, I don’t think it is a permission issue.

NOTE: OpenSearch 2.13. has some bugs in it we stayed at version 2.10.

For troubleshooting you can check your indices as show in this doc below.

check if you have enough drive space.

5

Thanks for the suggestions @gsmith!

Finally I fixed the problem. When I reinstalled opensearch last time to solve the huge memory consumption I gave ${HOSTNAME} for the nodename and it was different than the original nodename. Anyway I found the old nodename and correct it in opensearch.yml and It’s working now.

Again, thanks for your efforts and have a nice day!
Regards,
Zoltan