Graylog running but elasticsearch cluster health is red

[vubv@nimhlogs ~]$ curl -s -XGET http://127.0.0.1:9200/_cat/shards
graylog_139 3 p STARTED    13625117   9.9gb 137.187.63.155 graylog-master-node
graylog_139 1 p UNASSIGNED
graylog_139 2 p STARTED    13630438   9.9gb 137.187.63.155 graylog-master-node
graylog_139 0 p UNASSIGNED
graylog_158 3 p STARTED    10729900   9.2gb 137.187.63.155 graylog-master-node
graylog_158 1 p STARTED    10722570   9.2gb 137.187.63.155 graylog-master-node
graylog_158 2 p STARTED    10724075   9.2gb 137.187.63.155 graylog-master-node
graylog_158 0 p STARTED    10731803   9.2gb 137.187.63.155 graylog-master-node
graylog_130 3 p STARTED     9717149   8.1gb 137.187.63.155 graylog-master-node
graylog_130 1 p UNASSIGNED
graylog_130 2 p STARTED     9724887   8.1gb 137.187.63.155 graylog-master-node
graylog_130 0 p UNASSIGNED
graylog_144 3 p STARTED    12581794  11.7gb 137.187.63.155 graylog-master-node
graylog_144 1 p UNASSIGNED
graylog_144 2 p STARTED    12590860  11.7gb 137.187.63.155 graylog-master-node
graylog_144 0 p UNASSIGNED
graylog_133 3 p STARTED     2217594     2gb 137.187.63.155 graylog-master-node
graylog_133 1 p UNASSIGNED
graylog_133 2 p STARTED     2217861     2gb 137.187.63.155 graylog-master-node
graylog_133 0 p UNASSIGNED
graylog_140 3 p STARTED    12319847   9.2gb 137.187.63.155 graylog-master-node
graylog_140 1 p UNASSIGNED
graylog_140 2 p STARTED    12326557   9.2gb 137.187.63.155 graylog-master-node
graylog_140 0 p UNASSIGNED
graylog_160 3 p STARTED     2324196   2.3gb 137.187.63.155 graylog-master-node
graylog_160 1 p STARTED     2322864   2.3gb 137.187.63.155 graylog-master-node
graylog_160 2 p STARTED     2324044   2.3gb 137.187.63.155 graylog-master-node
graylog_160 0 p STARTED     2322780   2.3gb 137.187.63.155 graylog-master-node
graylog_131 3 p STARTED      925219     1gb 137.187.63.155 graylog-master-node
graylog_131 1 p UNASSIGNED
graylog_131 2 p STARTED      924629     1gb 137.187.63.155 graylog-master-node
graylog_131 0 p UNASSIGNED
graylog_137 3 p STARTED    17004111  13.6gb 137.187.63.155 graylog-master-node
graylog_137 1 p UNASSIGNED
graylog_137 2 p STARTED    16999162  13.6gb 137.187.63.155 graylog-master-node
graylog_137 0 p UNASSIGNED
graylog_165 3 p STARTED     1115730   1.3gb 137.187.63.155 graylog-master-node
graylog_165 1 p STARTED     1118040   1.4gb 137.187.63.155 graylog-master-node
graylog_165 2 p STARTED     1115565   1.4gb 137.187.63.155 graylog-master-node
graylog_165 0 p STARTED     1117694   1.4gb 137.187.63.155 graylog-master-node
graylog_169 3 p STARTED     1101462 963.9mb 137.187.63.155 graylog-master-node
graylog_169 1 p STARTED     1100256 962.8mb 137.187.63.155 graylog-master-node
graylog_169 2 p STARTED     1101492 964.6mb 137.187.63.155 graylog-master-node
graylog_169 0 p STARTED     1103342 965.3mb 137.187.63.155 graylog-master-node
graylog_154 3 p STARTED     2296185   2.3gb 137.187.63.155 graylog-master-node
graylog_154 1 p STARTED     2291285   2.3gb 137.187.63.155 graylog-master-node
graylog_154 2 p UNASSIGNED
graylog_154 0 p UNASSIGNED
graylog_141 3 p STARTED    13444881    10gb 137.187.63.155 graylog-master-node
graylog_141 1 p UNASSIGNED
graylog_141 2 p STARTED    13441770    10gb 137.187.63.155 graylog-master-node
graylog_141 0 p UNASSIGNED
graylog_142 3 p STARTED    16501475  12.2gb 137.187.63.155 graylog-master-node
graylog_142 1 p UNASSIGNED
graylog_142 2 p STARTED    16511232  12.2gb 137.187.63.155 graylog-master-node
graylog_142 0 p UNASSIGNED
graylog_151 3 p STARTED    12073487  10.2gb 137.187.63.155 graylog-master-node
graylog_151 1 p UNASSIGNED
graylog_151 2 p STARTED    12074020  10.2gb 137.187.63.155 graylog-master-node
graylog_151 0 p UNASSIGNED
graylog_164 3 p STARTED     1519501   1.7gb 137.187.63.155 graylog-master-node
graylog_164 1 p STARTED     1517652   1.7gb 137.187.63.155 graylog-master-node
graylog_164 2 p STARTED     1517250   1.7gb 137.187.63.155 graylog-master-node
graylog_164 0 p STARTED     1519092   1.7gb 137.187.63.155 graylog-master-node
graylog_128 3 p STARTED     2087999   1.8gb 137.187.63.155 graylog-master-node
graylog_128 1 p UNASSIGNED
graylog_128 2 p STARTED     2089309   1.8gb 137.187.63.155 graylog-master-node
graylog_128 0 p UNASSIGNED
graylog_138 3 p STARTED    10573340     8gb 137.187.63.155 graylog-master-node
graylog_138 1 p UNASSIGNED
graylog_138 2 p STARTED    10574497     8gb 137.187.63.155 graylog-master-node
graylog_138 0 p UNASSIGNED
graylog_167 3 p STARTED     4264036     4gb 137.187.63.155 graylog-master-node
graylog_167 1 p STARTED     4259222     4gb 137.187.63.155 graylog-master-node
graylog_167 2 p STARTED     4258596     4gb 137.187.63.155 graylog-master-node
graylog_167 0 p STARTED     4258986     4gb 137.187.63.155 graylog-master-node
graylog_148 3 p STARTED     3352209   4.2gb 137.187.63.155 graylog-master-node
graylog_148 1 p UNASSIGNED
graylog_148 2 p STARTED     3355684   4.2gb 137.187.63.155 graylog-master-node
graylog_148 0 p UNASSIGNED
graylog_162 3 p STARTED     3262307   3.1gb 137.187.63.155 graylog-master-node
graylog_162 1 p STARTED     3259799   3.1gb 137.187.63.155 graylog-master-node
graylog_162 2 p STARTED     3263068   3.1gb 137.187.63.155 graylog-master-node
graylog_162 0 p STARTED     3261792   3.1gb 137.187.63.155 graylog-master-node
graylog_157 3 p STARTED     3683725   3.6gb 137.187.63.155 graylog-master-node
graylog_157 1 p STARTED     3686542   3.6gb 137.187.63.155 graylog-master-node
graylog_157 2 p STARTED     3684416   3.6gb 137.187.63.155 graylog-master-node
graylog_157 0 p STARTED     3685160   3.6gb 137.187.63.155 graylog-master-node
graylog_134 3 p STARTED     2703949   2.2gb 137.187.63.155 graylog-master-node
graylog_134 1 p UNASSIGNED
graylog_134 2 p STARTED     2702812   2.2gb 137.187.63.155 graylog-master-node
graylog_134 0 p UNASSIGNED
graylog_143 3 p STARTED    10547255   8.9gb 137.187.63.155 graylog-master-node
graylog_143 1 p UNASSIGNED
graylog_143 2 p STARTED    10547614   8.9gb 137.187.63.155 graylog-master-node
graylog_143 0 p UNASSIGNED
graylog_153 3 p STARTED     2074770   2.1gb 137.187.63.155 graylog-master-node
graylog_153 1 p UNASSIGNED
graylog_153 2 p STARTED     2073780   2.1gb 137.187.63.155 graylog-master-node
graylog_153 0 p UNASSIGNED
graylog_136 3 p STARTED     2388091     2gb 137.187.63.155 graylog-master-node
graylog_136 1 p UNASSIGNED
graylog_136 2 p STARTED     2386369   2.1gb 137.187.63.155 graylog-master-node
graylog_136 0 p UNASSIGNED
graylog_161 3 p STARTED     2822716   2.8gb 137.187.63.155 graylog-master-node
graylog_161 1 p STARTED     2819310   2.8gb 137.187.63.155 graylog-master-node
graylog_161 2 p STARTED     2818809   2.8gb 137.187.63.155 graylog-master-node
graylog_161 0 p STARTED     2819468   2.8gb 137.187.63.155 graylog-master-node
graylog_135 3 p STARTED     3543223   2.9gb 137.187.63.155 graylog-master-node
graylog_135 1 p UNASSIGNED
graylog_135 2 p STARTED     3545060   2.9gb 137.187.63.155 graylog-master-node
graylog_135 0 p UNASSIGNED
graylog_163 3 p STARTED     2826884   2.8gb 137.187.63.155 graylog-master-node
graylog_163 1 p STARTED     2822466   2.8gb 137.187.63.155 graylog-master-node
graylog_163 2 p STARTED     2828355   2.8gb 137.187.63.155 graylog-master-node
graylog_163 0 p STARTED     2826506   2.8gb 137.187.63.155 graylog-master-node
graylog_152 3 p STARTED     1656832   1.9gb 137.187.63.155 graylog-master-node
graylog_152 1 p UNASSIGNED
graylog_152 2 p STARTED     1653400   1.9gb 137.187.63.155 graylog-master-node
graylog_152 0 p UNASSIGNED
graylog_126 3 p STARTED     1122172   1.1gb 137.187.63.155 graylog-master-node
graylog_126 1 p UNASSIGNED
graylog_126 2 p STARTED     1121562   1.1gb 137.187.63.155 graylog-master-node
graylog_126 0 p UNASSIGNED
graylog_168 3 p STARTED    13028119  11.4gb 137.187.63.155 graylog-master-node
graylog_168 1 p STARTED    13030801  11.4gb 137.187.63.155 graylog-master-node
graylog_168 2 p STARTED    13025547  11.4gb 137.187.63.155 graylog-master-node
graylog_168 0 p STARTED    13024011  11.4gb 137.187.63.155 graylog-master-node
graylog_147 3 p STARTED     5347445   4.7gb 137.187.63.155 graylog-master-node
graylog_147 1 p UNASSIGNED
graylog_147 2 p STARTED     5351073   4.8gb 137.187.63.155 graylog-master-node
graylog_147 0 p UNASSIGNED
graylog_166 3 p STARTED      981683   1.4gb 137.187.63.155 graylog-master-node
graylog_166 1 p STARTED      982596   1.4gb 137.187.63.155 graylog-master-node
graylog_166 2 p STARTED      981535   1.4gb 137.187.63.155 graylog-master-node
graylog_166 0 p STARTED      979641   1.4gb 137.187.63.155 graylog-master-node
graylog_132 3 p STARTED     1177483   1.2gb 137.187.63.155 graylog-master-node
graylog_132 1 p UNASSIGNED
graylog_132 2 p STARTED     1175958   1.1gb 137.187.63.155 graylog-master-node
graylog_132 0 p UNASSIGNED
graylog_150 3 p STARTED     2346955   2.4gb 137.187.63.155 graylog-master-node
graylog_150 1 p UNASSIGNED
graylog_150 2 p STARTED     2345980   2.4gb 137.187.63.155 graylog-master-node
graylog_150 0 p UNASSIGNED
graylog_155 3 p STARTED      558461 572.9mb 137.187.63.155 graylog-master-node
graylog_155 1 p STARTED      557191 570.8mb 137.187.63.155 graylog-master-node
graylog_155 2 p UNASSIGNED
graylog_155 0 p UNASSIGNED
graylog_170 3 p STARTED     3938766   3.6gb 137.187.63.155 graylog-master-node
graylog_170 1 p STARTED     3938402   6.3gb 137.187.63.155 graylog-master-node
graylog_170 2 p STARTED     3940434   3.8gb 137.187.63.155 graylog-master-node
graylog_170 0 p STARTED     3936463   3.8gb 137.187.63.155 graylog-master-node
graylog_149 3 p STARTED     2633192   2.9gb 137.187.63.155 graylog-master-node
graylog_149 1 p UNASSIGNED
graylog_149 2 p STARTED     2636264   2.9gb 137.187.63.155 graylog-master-node
graylog_149 0 p UNASSIGNED
graylog_129 3 p STARTED     1591533   1.5gb 137.187.63.155 graylog-master-node
graylog_129 1 p UNASSIGNED
graylog_129 2 p STARTED     1591007   1.5gb 137.187.63.155 graylog-master-node
graylog_129 0 p UNASSIGNED
graylog_146 3 p STARTED     1830199   2.4gb 137.187.63.155 graylog-master-node
graylog_146 1 p UNASSIGNED
graylog_146 2 p STARTED     1831944   2.4gb 137.187.63.155 graylog-master-node
graylog_146 0 p UNASSIGNED
graylog_159 3 p STARTED     1642996   1.9gb 137.187.63.155 graylog-master-node
graylog_159 1 p STARTED     1644311   1.9gb 137.187.63.155 graylog-master-node
graylog_159 2 p STARTED     1643293   1.9gb 137.187.63.155 graylog-master-node
graylog_159 0 p STARTED     1641786   1.9gb 137.187.63.155 graylog-master-node
graylog_145 3 p STARTED     2790008   3.9gb 137.187.63.155 graylog-master-node
graylog_145 1 p UNASSIGNED
graylog_145 2 p STARTED     2787973   3.9gb 137.187.63.155 graylog-master-node
graylog_145 0 p UNASSIGNED
graylog_127 3 p STARTED     1175855   1.2gb 137.187.63.155 graylog-master-node
graylog_127 1 p UNASSIGNED
graylog_127 2 p STARTED     1174157   1.2gb 137.187.63.155 graylog-master-node
graylog_127 0 p UNASSIGNED
graylog_156 3 p STARTED     4011459   4.2gb 137.187.63.155 graylog-master-node
graylog_156 1 p STARTED     4013292   4.2gb 137.187.63.155 graylog-master-node
graylog_156 2 p STARTED     4011195   4.2gb 137.187.63.155 graylog-master-node
graylog_156 0 p STARTED     4011984   4.2gb 137.187.63.155 graylog-master-node
[vubv@nimhlogs ~]$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 120,
  "active_shards" : 120,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 60,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 66.66666666666666
}
[vubv@nimhlogs ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        56G  6.2G   50G  12% /
devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G  9.0M  7.9G   1% /run
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb1       900G  536G  365G  60% /var
/dev/sda1        20G  274M   20G   2% /home
/dev/sda2       497M  211M  287M  43% /boot
tmpfs           1.6G     0  1.6G   0% /run/user/984
tmpfs           1.6G   12K  1.6G   1% /run/user/42
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Graylog indices and shards are stored in /var. I have another elasticsearch server/node that is tied to this graylog server. Seems like it’s working because the elasticsearch directory is filling up.

How do you assign an unassigned shard? It says I’ve got 60 of them.

Check the logs of your Elasticsearch node and try to find out why the shards weren’t assigned in the first place.
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

If you have ES 5, you can look at the problem with the Explain API:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-explain.html

After knowing the reason, you can hopefully fix it.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.