Hot/Warm Architecture and wrongly assigned shards on Elasticsearch 7.10

Graylog Central (peer support)
1

1. Describe your incident:
I’m trying to setup a Hot/Warm Architecture with Elasticsearch 7.10.2. via node attr box_type, but shards are still assigned to nodes which are set to “warm”

curl http://localhost:9200/_cat/nodeattrs

foo0083 1.2.3.42 1.2.3.42 box_type hot
foo1080 1.2.3.51 1.2.3.51 box_type hot
foo1081 1.2.3.52 1.2.3.52 box_type hot
foo1082 1.2.3.53 1.2.3.53 box_type hot
foo0084 1.2.3.43 1.2.3.43 box_type hot
foo0086 1.2.3.45 1.2.3.45 box_type hot
foo0085 1.2.3.44 1.2.3.44 box_type hot
foo0081 1.2.3.40 1.2.3.40 box_type hot
foo0082 1.2.3.41 1.2.3.41 box_type hot
foo1087 1.2.3.57 1.2.3.57 box_type warm
foo1086 1.2.3.56 1.2.3.56 box_type warm
foo1088 1.2.3.58 1.2.3.58 box_type warm
foo1085 1.2.3.55 1.2.3.55 box_type warm

Custom mapping

{
  "template": "fw_audit_*",
   "settings": {
    "index": {
      "routing": {
        "allocation": {
          "require": {
            "box_type": "hot"
          }
        }
      },
      "refresh_interval": "30s"
    }
  },
  "mappings": {
    "properties": {
      "client_ip": {
        "type": "ip"
      },
      "target_ip": {
        "type": "ip"
      }
    }
    }
  }
}

Shard output

fw_audit_463  1     r      STARTED    296209230  45.9gb 1.2.3.55 foo1085
fw_audit_502  1     r      STARTED    246735688    77gb 1.2.3.55 foo1085
fw_audit_484  2     p      STARTED    172817454  51.7gb 1.2.3.55 foo1085
fw_audit_470  1     p      STARTED    270558453  81.7gb 1.2.3.55 foo1085
fw_audit_508  1     p      STARTED    284684002  86.9gb 1.2.3.55 foo1085
fw_audit_450  0     r      STARTED    271875377  37.7gb 1.2.3.55 foo1085
fw_audit_439  0     p      STARTED    294420183    46gb 1.2.3.55 foo1085
fw_audit_507  2     r      STARTED    261709599  74.7gb 1.2.3.55 foo1085
fw_audit_449  0     p      STARTED    281349238  43.4gb 1.2.3.55 foo1085
fw_audit_457  0     r      STARTED    253075699  35.6gb 1.2.3.55 foo1085
fw_audit_476  0     p      STARTED    212195528    65gb 1.2.3.55 foo1085
fw_audit_483  1     r      STARTED    186939149  57.6gb 1.2.3.55 foo1085
fw_audit_486  2     p      STARTED    166063109  49.1gb 1.2.3.55 foo1085
fw_audit_482  2     p      STARTED    188272068  58.2gb 1.2.3.55 foo1085
fw_audit_447  1     p      STARTED    263052054    43gb 1.2.3.55 foo1085
fw_audit_445  2     r      STARTED    278579183  43.7gb 1.2.3.55 foo1085
fw_audit_517  1     p      STARTED    283368483  88.6gb 1.2.3.55 foo1085
fw_audit_520  2     r      STARTED    240099732  70.1gb 1.2.3.55 foo1085
fw_audit_503  1     r      STARTED    231039879  73.6gb 1.2.3.55 foo1085

From my expectation, the warm nodes shouldnt receive any shards with this configuration…

2. Describe your environment:

  • OS Information: Ubuntu 20.04

  • Package Version:
    ii elasticsearch-oss 7.10.2 amd64 Distributed RESTful search engine built for the cloud
    ii graylog-server 4.2.5-1 all Graylog server

Any help/hint is very appreciated.

Regards
Oliver

2

Hello @riskersen

Only time I execute a Hot/Warm/Cold Architecture with indices was in ELK, and I could do it from the Web UI. But as for executing it on Graylog, I have not BUT I did run into this perhaps it may help if you haven’t seen it already

Also have you seen this?

3

Hi,

thanks for the link. It seems, that you are right. ES7 introduced a lot of new features related to data tiering.

I will have a look on that.

Thanks!

Regards
Oliver

1 Like
4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.