Create/Cycle index set to specific ID?

[nisow95612]

Hello graylog community,
this time I have tricky question.

I am running several index templates like “logs_r01m”, “logs_r06m”, “logs_r12m”.
All of these index templates use time-based rotation and were created at the same time, so their current write indices points to logs_r01m_736, logs_r06m_736 and logs_r12m_736 respectively.

Now I need to create a new index set “logs_r03m” for a 3M retention period, but I’d like to keep its numbering synchronized. Is there a comfortable way to get its current write index to point to logs_r03m_736?

I know I can press “index maintenance > rotate active write index” 736 times, but that is hardly comfortable and will get only worse.

[tmacgbay]

This will start you down the path:

Reindex API | Elasticsearch Guide [7.15] | Elastic

(Note Elasticsearch for Graylog should stay at version 7.10 for now…)

On the Elastic side you can create/re-index to the name you want - the trick is getting Graylog to see it. I think you can use the index maintenance to recalculate the index range but I haven’t specifically done that…

[nisow95612]

@tmacgbay Thank you for trying to help.
I want to start a new empty index set and I want Graylog to skip creating “logs_r03m_0”," logs_r03m_1", etc. all the way to “logs_r03m_735” and start the index set at logs_r03m_736.
I don’t have any data that would go into this index yet, so I don’t see how reindexing could help. “Recalculating index ranges” works on individual indices and recalculates their timestamp ranges, not index names.

[tmacgbay]

The re-index is essentially a copy command. Create the new index, the copy (or as elastic search calls it “re-index”) from your initial name to the one you want. Delete the old index via elastic or via Graylog.

[nisow95612]

Sure, I can copy (empty) “logs_r03m_0” to “logs_r03m_736” and delete the original. That is the easy part.
But Graylog will not know about this and complain “logs_r03m_0” does not exist. Then I could click “index maintenance > rotate active write index”, but that will create “logs_r03m_1” and point write index there, right?

I don’t want to spend whole day clicking “index maintenance > rotate active write index” 735 times.

[tmacgbay]

To get the active index right, you could re-index to one lower than you want, delete all others, then once you restart graylog, rotate the index once….

You are working with a blank slate, give it a try!

[gsmith]

I was curious and I wanted to give this a try in my lab

@tmacgbay

the trick is getting Graylog to see it.

Agree

I took me a minute but I execute

curl -X PUT "localhost:9200/logs_r03m_736?pretty"

Did a quick check, This is about as far as I got but its interesting.

[root@graylog graylog_user]# curl -X GET http://localhost:9200/_cluster/allocation/explain?pretty
{
  "index" : "logs_r03m_736",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2021-10-07T21:53:27.917Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "srgsYBshRb-ue3IQzIK-RQ",
      "node_name" : "graylog.domain.com",
      "transport_address" : "localhost:9300",
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "enable",
          "decision" : "NO",
          "explanation" : "replica allocations are forbidden due to cluster setting [cluster.routing.allocation.enable=primaries]"
        },
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[logs_r03m_736][0], node[srgsYBshRb-ue3IQzIK-RQ], [P], s[STARTED], a[id=H
        }
      ]
    }
  ]
}

@nisow95612
I just wanted to give this a try but this is far as I got.

[tmacgbay]

I am nearly off grid until Wednesday but I think it would follow along the lines of:

• in graylog create index logs_r03m so elastic would have logs_r03m_0
• in elastic, re-index logs_r03m_0 copied to logs_r03m_736 and change it to not be marked as a replica
• remove the _0 version if so desired… or let it roll off
• restart graylog or run index maint to get Graylog to check and pull in elastic data. (May not be needed!)

I can play with it more next Wednesday. (or later today)

[tmacgbay]

or… the internet works better here than originally thought… here are the commands I tested and it worked for me…

 curl -X POST --netrc "elast-server:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
 {
   "source": {
      "index": "test_0"
    },
    "dest": {
      "index": "test_736",
      "version_type": "internal"
    }
 }
 '
 
 curl -XPUT --netrc -H 'Content-Type: application/json' http://elast-server:9200/test_736/_settings?pretty -d '{"number_of_replicas":0}'

interestingly I didn’t restart Graylog or mess with indexes at all…

[gsmith]

Ah that was my missing piece, Thank @tmacgbay

EDIT: new GL site I noticed this section.

https://docs.graylog.org/docs/elasticsearch-reindexing

[nisow95612]

@tmacgbay Thank you for the big push for just giving it a try.

Based on your and @gmith’s experiments I tried to just create the desired name like this:

curl -X PUT 'elast-server:9200/logs_r03m_376' -H 'Content-Type:application/json' -d '
{
	"settings":{
		"number_of_shards":"4",
		"number_of_replicas":"0"
	}
}
'

and voila, Graylog switched to the new name automatically. Cool!

-----------------------------------------------------------------

So the revised how-to is:

• in Graylog create index set logs_r03m so elastic would have logs_r03m_0
• in Elastic create index set logs_r03m_736 - see curl call above
• Graylog automatically switches to the new index and starts writing to it
• Finished! You can remove the _0 version if you want… or you can let retention policy do it.

Or, just to be on the safe side, create logs_r03m_735 and then rotate active write index.
That way logs_r03m_736 will be guaranteed to be created exactly as Graylog wants it.

-----------------------------------------------------------------

PS: It even works the other way. If I delete logs_r03m_736, graylog automatically switches to logs_r03m_0 and starts writing there (NOT RECOMMENDED!). That explain why simply rotating write index was enough for fixing Need to clear elastic search after upgrade to version 4