System Messages "There is no index target to point to. Creating one now"

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Getting re-occurring messages about “There is no index target to point to. Creating one now”
Logs seem to be getting processed though.

2. Describe your environment:

  • OS Information: Ubuntu 20.04

  • Package Version: Graylog 4.2.3, elasticsearch 7.10.2

3. What steps have you already taken to try and solve the problem?
Verified elasticsearch is working and running. Looked through the community posts and found this one There is no index target to point to. Creating one now but it only seemed to be about elastic version which I’ve confirmed on the setup guide is compatible on my instance.

4. How can the community help?
At a loss right now about what to try and do about this.

This happens where? In the GUI? In Graylog logs?

Adding on to @tmacgbay question.

What were you trying to do when this occurs? Is this a fresh install or did this issue just occur?
Can you show your full log message?

After digging through more on the Elasticsearch logs and forums I ran the check health
curl -X GET "localhost:9200/_cluster/health"

and got the following

{"cluster_name":"graylog","status":"red","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":996,"active_shards":996,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":4,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":99.6}

Which says the status was red so I searched around for what that meant and I ran the following

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 36068  100 36068    0     0  45540graylog_87          1 p UNASSIGNED ALLOCATION_FAILED
 graylog_87          2 p UNASSIGNED ALLOCATION_FAILED
  graylog_87          3 p UNASSIGNED ALLOCATION_FAILED
 graylog_87          0 p UNASSIGNED ALLOCATION_FAILED
  0 --:--:-- --:--:-- --:--:-- 45540

and this as well

curl -XGET localhost:9200/_cluster/allocation/explain?pretty
{
  "index" : "graylog_87",
  "shard" : 1,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2021-12-20T20:47:46.055Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed shard on node [O6_sxqoVR3CDeTBoZoG2uw]: failed recovery, failure RecoveryFailedException[[graylog_87][1]: Recovery failed on {graylog}{O6_sxqoVR3CDeTBoZoG2uw}{-En-X4ftSOOMMZ6Q0impDA}{127.0.0.1}{127.0.0.1:9300}{dimr}]; nested: IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/media/data/elasticsearch/nodes/0/indices/D6EoAlY0S5-GpdU30weTrg/1/translog] is corrupted]; nested: IllegalStateException[pre-1.4 translog found [/media/data/elasticsearch/nodes/0/indices/D6EoAlY0S5-GpdU30weTrg/1/translog/translog-216.tlog]]; ",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy",
  "node_allocation_decisions" : [
    {
      "node_id" : "O6_sxqoVR3CDeTBoZoG2uw",
      "node_name" : "graylog",
      "transport_address" : "127.0.0.1:9300",
      "node_decision" : "no",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "g8Hq5B3KSL6prD2LooaZVg"
      },
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-12-20T20:47:46.055Z], failed_attempts[5], failed_nodes[[O6_sxqoVR3CDeTBoZoG2uw]], delayed=false, details[failed shard on node [O6_sxqoVR3CDeTBoZoG2uw]: failed recovery, failure RecoveryFailedException[[graylog_87][1]: Recovery failed on {graylog}{O6_sxqoVR3CDeTBoZoG2uw}{-En-X4ftSOOMMZ6Q0impDA}{127.0.0.1}{127.0.0.1:9300}{dimr}]; nested: IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/media/data/elasticsearch/nodes/0/indices/D6EoAlY0S5-GpdU30weTrg/1/translog] is corrupted]; nested: IllegalStateException[pre-1.4 translog found [/media/data/elasticsearch/nodes/0/indices/D6EoAlY0S5-GpdU30weTrg/1/translog/translog-216.tlog]]; ], allocation_status[deciders_no]]]"
        }
      ]
    }
  ]
}

Found online that the solution was to delete the index so I ran

curl -XDELETE 'localhost:9200/graylog_87/'

Ran the health check again and it was green and in the Graylog web interface system messages it stopped showing the messages continuously now.

Hello,

Awesome troubleshooting, and its much appreciated that you showed the resolution to this issue in detail.
Thanks :smiley:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.