Elasticsearch unassigned shards

[Matthias]

Hi everyone,
I’m in a real pain with unassigned log that failed to rotate, overwriting my custom mapping.
I’ve got 2 elasticsearch nodes (with a hot-warm conf, Using a Hot/Warm Elasticsearch Cluster | Graylog) and graylog-server installed on the master node.

CONFIGURATION:

/etc/elasticsearch/elasticsearch.yml

cluster.name: graylog
node.master: true
node.name: ${HOSTNAME}
node.attr.box_type: hot
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.1.100
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.100", "192.168.1.99"]
discovery.zen.minimum_master_nodes: 1
action.auto_create_index: false

/etc/graylog/action/hot_to_warm_action.yml

actions:
  # regle de retention des logs EXP
  1:
    action: allocation
    description: "Applique les règles de filtrage de l'allocation des shards aux index concernés"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: true
      timeout_override:
      continue_if_exception: false
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: graylog_
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 14
  2:
    action: forcemerge
    description: "Effectue un forceMerge sur les index affectés jusqu'à la valeur de 'max_num_segments' par shard"
    options:
      max_num_segments: 1
      delay:
      timeout_override: 21600
      continue_if_exception: false
      ignore_empty_list: true
      disable_action: false
    filters:
    - filtertype: pattern
      kind: prefix
      value: graylog_
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 15

I created more actions in this file (exactly the same, duplicates) 3, 4, 5, et 6 for prefix “dev_” and “int_”.
These two will match with streams that extract from “all messages” server’s log and set them the previous prefix “dev” and “int”.
Next, three Index set are up to store logs with each prefix graylog, dev and int.

As the blog-post said (link at the beginning of the topic), I configured a custom mapping, more like three to be exact with different “template”.

PROBLEM:

Everything is fine at the startup of graylog, the logs come to the each index set, they are all in my master node (hot node).
BUT, at the moment of the rotation, the new shards appear UNASSIGNED, the templates are overwritten without hot/warm conf.

[Matthias]

(can’t add the full text more than 2 links)

RESULTS:

“_cluster/allocation/explain?pretty”:

{
  "index" : "dev_0",
  "shard" : 3,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2021-04-16T11:33:42.565Z",
    "last_allocation_status" : "no"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "9p358b3rT6C64VAJJRPZ6w",
      "node_name" : "srvlog3",
      "transport_address" : "192.168.1.99:9300",
      "node_attributes" : {
        "box_type" : "warm"
      },
      "node_decision" : "no",
      "weight_ranking" : 1,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : "node does not match index setting [index.routing.allocation.require] filters [boxtype:\"hot\"]"
        }
      ]
    },
    {
      "node_id" : "CUpwX683QKyQLsDFGaf8Qw",
      "node_name" : "srvlog2",
      "transport_address" : "192.168.1.100:9300",
      "node_attributes" : {
        "box_type" : "hot"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "filter",
          "decision" : "NO",
          "explanation" : "node does not match index setting [index.routing.allocation.require] filters [boxtype:\"hot\"]"
        }
      ]
    }
  ]
}

GRAYLOG logs:

ERROR [IndexRotationThread] Couldn't point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn't read health status for index dev_1


        at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:107) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:60) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.waitForStatus(Indices.java:689) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.waitForRecovery(Indices.java:680) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:163) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_282]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_282]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
2021-04-19T11:06:42.316+02:00 WARN  [IndexRotationThread] Deflector is pointing to [graylog_0], not the newest one: [graylog_1]. Re-pointing.

ELASTICSEARCH logs:

[2021-04-19T11:32:07,999][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:11,989][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:15,996][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:19,863][DEBUG][o.e.a.a.c.a.TransportClusterAllocationExplainAction] [srvlog2] explaining the allocation for [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDe
cisions?=false], found shard [[dev_0][3], node[null], [P], recovery_source[new shard recovery], s[UNASSIGNED], unassigned_info[[reason=INDEX_CREATED], at[2021-04-16T11:33:42.565Z], delayed=false, allocation_status[deciders_no]]]
[2021-04-19T11:32:20,013][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:23,992][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:27,992][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:31,990][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]
[2021-04-19T11:32:35,994][DEBUG][o.e.a.s.TransportSearchAction] [srvlog2] All shards failed for phase: [query]

template overwritten:

{
  "dev-template" : {
    "order" : -1,
    "index_patterns" : [
      "dev_*"
    ],
    "settings" : {
      "index" : {
        "analysis" : {
          "analyzer" : {
            "analyzer_keyword" : {
              "filter" : "lowercase",
              "tokenizer" : "keyword"
            }
          }
        }
      }
    },
    "mappings" : {
      "message" : {
        "_source" : {
          "enabled" : true
        },
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string",
              "match" : "gl2_*"
            }
          },
          {
            "store_generic" : {
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string"
            }
          }
        ],
        "properties" : {
          "gl2_processing_timestamp" : {
            "format" : "yyyy-MM-dd HH:mm:ss.SSS",
            "type" : "date"
          },
          "gl2_accounted_message_size" : {
            "type" : "long"
          },
          "gl2_receive_timestamp" : {
            "format" : "yyyy-MM-dd HH:mm:ss.SSS",
            "type" : "date"
          },
          "full_message" : {
            "fielddata" : false,
            "analyzer" : "standard",
            "type" : "text"
          },
          "streams" : {
            "type" : "keyword"
          },
          "source" : {
            "fielddata" : true,
            "analyzer" : "analyzer_keyword",
            "type" : "text"
          },
          "message" : {
            "fielddata" : false,
            "analyzer" : "standard",
            "type" : "text"
          },
          "timestamp" : {
            "format" : "yyyy-MM-dd HH:mm:ss.SSS",
            "type" : "date"
          }
        }
      }
    },
    "aliases" : { }
  }
}

If there is other error log or configuration that I forgot to mention, ask me.
Thank you all

[reimlima (Dummy)]

Hi @Matthias

I had a similar problem a while ago, here the whole thread and what I did to solve it.

Graylog rewriting "index template" in Elasticsearch

Hi Everyone! I’m new using Graylog, as well as use Elastichsearch in cluster and MongoDB, it’s been a long learning curve to me but I’m enjoying everything. My environment is: Graylog 4, 2 nodes running in cluster Elasticsearch 7, 6 nodes (2 hot, 2 warm, 2 cold) MongoDB (4.4.4) running in Graylogs master node elasticsearch-curator (5.8.3) running in the Elastichsearch master nodes I was trying to configure Elasticsearch ILM in my end, but I saw here (improved elasticsearch index managemen…

Hope it helps.

[Matthias]

Hi @reimlima, thank you for your reply.
I’ll see if it can help !