Configuring default index set

[groovyghost (Rahul)]

Incident:
I have 2 graylog setups one as prod and another as staging.As per a request i needed to configure default index set rotation period from message count to time period.

When tried in staging it automatically created new index and there was no issues.But when done in Prod,elasticsearch went red with unassigned shards.

After edit of default index set,elasticsearch cluster is red

As per a request i created a new index set (2yrs) with 1 week per index max of 120 indicies,and changed exisiting index set (3months)to 1 week per index max of 14 index.after 12 hrs found that 14 unassigned shards.I wanna recover this without data loss.there are nor space or resources constrain Pls help Allocation explaination - “cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster” OS Information:centos Package Version:G…

I think the mistake was i didnt mannually rotate active write index.

To verify that in staging i tried to recreate the issue i.e change the default index set rotation period from message count to time period.But weirdly the the active write index is not rotated and no crashes.it just keeps on using the same active write index.

Where is the problem? What are the instructions to edit a default index?

Kindly help

Environment:

• OS Centos 7

Package verison

• Graylog 2.4.3
• Elasticsearch 5.6.0
• Mongo 4.2.12

Configuration:
Production :
3 Severs all have Graylog,Elasticsearch and Mongod (Replica set) installed

• Elasticsearch JVM - 4GB per node
• Approx 500GB Disk space available in each node

Staging :
3 Severs all have Graylog,Elasticsearch and Mongod(Replica set) installed

• Elasticsearch JVM - 2GB per node

[gsmith]

Hello,

If you changed the your active index Rotation period I would execute maintenance for recalculating Index ranges first.

Once you seen this completed here Under System/Overview

I would try to rotate the index manually as show in the picture above using “Rotate Active write Index” .
Any time I do this in production I always TAIL my graylog log file for errors.

tail -f /var/log/graylog-server/server.log

Hope that helps

EDIT:

@groovyghost

What are the instructions to edit a default index?

Marked in red box.

EDIT2:

@groovyghost

But weirdly the the active write index is not rotated and no crashes.it just keeps on using the same active write index.

If you look at the two red boxes below in the picture, the one red box in the upper right shows 10 Max number of indices and the one lower left show you only have 8 is created. So unless you manually rotate it will stay that way till you reach 10 Index sets.