Graylog running but elasticsearch cluster health is red


(Hong-Duc) #1
[vubv@nimhlogs ~]$ curl -s -XGET http://127.0.0.1:9200/_cat/shards
graylog_139 3 p STARTED    13625117   9.9gb 137.187.63.155 graylog-master-node
graylog_139 1 p UNASSIGNED
graylog_139 2 p STARTED    13630438   9.9gb 137.187.63.155 graylog-master-node
graylog_139 0 p UNASSIGNED
graylog_158 3 p STARTED    10729900   9.2gb 137.187.63.155 graylog-master-node
graylog_158 1 p STARTED    10722570   9.2gb 137.187.63.155 graylog-master-node
graylog_158 2 p STARTED    10724075   9.2gb 137.187.63.155 graylog-master-node
graylog_158 0 p STARTED    10731803   9.2gb 137.187.63.155 graylog-master-node
graylog_130 3 p STARTED     9717149   8.1gb 137.187.63.155 graylog-master-node
graylog_130 1 p UNASSIGNED
graylog_130 2 p STARTED     9724887   8.1gb 137.187.63.155 graylog-master-node
graylog_130 0 p UNASSIGNED
graylog_144 3 p STARTED    12581794  11.7gb 137.187.63.155 graylog-master-node
graylog_144 1 p UNASSIGNED
graylog_144 2 p STARTED    12590860  11.7gb 137.187.63.155 graylog-master-node
graylog_144 0 p UNASSIGNED
graylog_133 3 p STARTED     2217594     2gb 137.187.63.155 graylog-master-node
graylog_133 1 p UNASSIGNED
graylog_133 2 p STARTED     2217861     2gb 137.187.63.155 graylog-master-node
graylog_133 0 p UNASSIGNED
graylog_140 3 p STARTED    12319847   9.2gb 137.187.63.155 graylog-master-node
graylog_140 1 p UNASSIGNED
graylog_140 2 p STARTED    12326557   9.2gb 137.187.63.155 graylog-master-node
graylog_140 0 p UNASSIGNED
graylog_160 3 p STARTED     2324196   2.3gb 137.187.63.155 graylog-master-node
graylog_160 1 p STARTED     2322864   2.3gb 137.187.63.155 graylog-master-node
graylog_160 2 p STARTED     2324044   2.3gb 137.187.63.155 graylog-master-node
graylog_160 0 p STARTED     2322780   2.3gb 137.187.63.155 graylog-master-node
graylog_131 3 p STARTED      925219     1gb 137.187.63.155 graylog-master-node
graylog_131 1 p UNASSIGNED
graylog_131 2 p STARTED      924629     1gb 137.187.63.155 graylog-master-node
graylog_131 0 p UNASSIGNED
graylog_137 3 p STARTED    17004111  13.6gb 137.187.63.155 graylog-master-node
graylog_137 1 p UNASSIGNED
graylog_137 2 p STARTED    16999162  13.6gb 137.187.63.155 graylog-master-node
graylog_137 0 p UNASSIGNED
graylog_165 3 p STARTED     1115730   1.3gb 137.187.63.155 graylog-master-node
graylog_165 1 p STARTED     1118040   1.4gb 137.187.63.155 graylog-master-node
graylog_165 2 p STARTED     1115565   1.4gb 137.187.63.155 graylog-master-node
graylog_165 0 p STARTED     1117694   1.4gb 137.187.63.155 graylog-master-node
graylog_169 3 p STARTED     1101462 963.9mb 137.187.63.155 graylog-master-node
graylog_169 1 p STARTED     1100256 962.8mb 137.187.63.155 graylog-master-node
graylog_169 2 p STARTED     1101492 964.6mb 137.187.63.155 graylog-master-node
graylog_169 0 p STARTED     1103342 965.3mb 137.187.63.155 graylog-master-node
graylog_154 3 p STARTED     2296185   2.3gb 137.187.63.155 graylog-master-node
graylog_154 1 p STARTED     2291285   2.3gb 137.187.63.155 graylog-master-node
graylog_154 2 p UNASSIGNED
graylog_154 0 p UNASSIGNED
graylog_141 3 p STARTED    13444881    10gb 137.187.63.155 graylog-master-node
graylog_141 1 p UNASSIGNED
graylog_141 2 p STARTED    13441770    10gb 137.187.63.155 graylog-master-node
graylog_141 0 p UNASSIGNED
graylog_142 3 p STARTED    16501475  12.2gb 137.187.63.155 graylog-master-node
graylog_142 1 p UNASSIGNED
graylog_142 2 p STARTED    16511232  12.2gb 137.187.63.155 graylog-master-node
graylog_142 0 p UNASSIGNED
graylog_151 3 p STARTED    12073487  10.2gb 137.187.63.155 graylog-master-node
graylog_151 1 p UNASSIGNED
graylog_151 2 p STARTED    12074020  10.2gb 137.187.63.155 graylog-master-node
graylog_151 0 p UNASSIGNED
graylog_164 3 p STARTED     1519501   1.7gb 137.187.63.155 graylog-master-node
graylog_164 1 p STARTED     1517652   1.7gb 137.187.63.155 graylog-master-node
graylog_164 2 p STARTED     1517250   1.7gb 137.187.63.155 graylog-master-node
graylog_164 0 p STARTED     1519092   1.7gb 137.187.63.155 graylog-master-node
graylog_128 3 p STARTED     2087999   1.8gb 137.187.63.155 graylog-master-node
graylog_128 1 p UNASSIGNED
graylog_128 2 p STARTED     2089309   1.8gb 137.187.63.155 graylog-master-node
graylog_128 0 p UNASSIGNED
graylog_138 3 p STARTED    10573340     8gb 137.187.63.155 graylog-master-node
graylog_138 1 p UNASSIGNED
graylog_138 2 p STARTED    10574497     8gb 137.187.63.155 graylog-master-node
graylog_138 0 p UNASSIGNED
graylog_167 3 p STARTED     4264036     4gb 137.187.63.155 graylog-master-node
graylog_167 1 p STARTED     4259222     4gb 137.187.63.155 graylog-master-node
graylog_167 2 p STARTED     4258596     4gb 137.187.63.155 graylog-master-node
graylog_167 0 p STARTED     4258986     4gb 137.187.63.155 graylog-master-node
graylog_148 3 p STARTED     3352209   4.2gb 137.187.63.155 graylog-master-node
graylog_148 1 p UNASSIGNED
graylog_148 2 p STARTED     3355684   4.2gb 137.187.63.155 graylog-master-node
graylog_148 0 p UNASSIGNED
graylog_162 3 p STARTED     3262307   3.1gb 137.187.63.155 graylog-master-node
graylog_162 1 p STARTED     3259799   3.1gb 137.187.63.155 graylog-master-node
graylog_162 2 p STARTED     3263068   3.1gb 137.187.63.155 graylog-master-node
graylog_162 0 p STARTED     3261792   3.1gb 137.187.63.155 graylog-master-node
graylog_157 3 p STARTED     3683725   3.6gb 137.187.63.155 graylog-master-node
graylog_157 1 p STARTED     3686542   3.6gb 137.187.63.155 graylog-master-node
graylog_157 2 p STARTED     3684416   3.6gb 137.187.63.155 graylog-master-node
graylog_157 0 p STARTED     3685160   3.6gb 137.187.63.155 graylog-master-node
graylog_134 3 p STARTED     2703949   2.2gb 137.187.63.155 graylog-master-node
graylog_134 1 p UNASSIGNED
graylog_134 2 p STARTED     2702812   2.2gb 137.187.63.155 graylog-master-node
graylog_134 0 p UNASSIGNED
graylog_143 3 p STARTED    10547255   8.9gb 137.187.63.155 graylog-master-node
graylog_143 1 p UNASSIGNED
graylog_143 2 p STARTED    10547614   8.9gb 137.187.63.155 graylog-master-node
graylog_143 0 p UNASSIGNED
graylog_153 3 p STARTED     2074770   2.1gb 137.187.63.155 graylog-master-node
graylog_153 1 p UNASSIGNED
graylog_153 2 p STARTED     2073780   2.1gb 137.187.63.155 graylog-master-node
graylog_153 0 p UNASSIGNED
graylog_136 3 p STARTED     2388091     2gb 137.187.63.155 graylog-master-node
graylog_136 1 p UNASSIGNED
graylog_136 2 p STARTED     2386369   2.1gb 137.187.63.155 graylog-master-node
graylog_136 0 p UNASSIGNED
graylog_161 3 p STARTED     2822716   2.8gb 137.187.63.155 graylog-master-node
graylog_161 1 p STARTED     2819310   2.8gb 137.187.63.155 graylog-master-node
graylog_161 2 p STARTED     2818809   2.8gb 137.187.63.155 graylog-master-node
graylog_161 0 p STARTED     2819468   2.8gb 137.187.63.155 graylog-master-node
graylog_135 3 p STARTED     3543223   2.9gb 137.187.63.155 graylog-master-node
graylog_135 1 p UNASSIGNED
graylog_135 2 p STARTED     3545060   2.9gb 137.187.63.155 graylog-master-node
graylog_135 0 p UNASSIGNED
graylog_163 3 p STARTED     2826884   2.8gb 137.187.63.155 graylog-master-node
graylog_163 1 p STARTED     2822466   2.8gb 137.187.63.155 graylog-master-node
graylog_163 2 p STARTED     2828355   2.8gb 137.187.63.155 graylog-master-node
graylog_163 0 p STARTED     2826506   2.8gb 137.187.63.155 graylog-master-node
graylog_152 3 p STARTED     1656832   1.9gb 137.187.63.155 graylog-master-node
graylog_152 1 p UNASSIGNED
graylog_152 2 p STARTED     1653400   1.9gb 137.187.63.155 graylog-master-node
graylog_152 0 p UNASSIGNED
graylog_126 3 p STARTED     1122172   1.1gb 137.187.63.155 graylog-master-node
graylog_126 1 p UNASSIGNED
graylog_126 2 p STARTED     1121562   1.1gb 137.187.63.155 graylog-master-node
graylog_126 0 p UNASSIGNED
graylog_168 3 p STARTED    13028119  11.4gb 137.187.63.155 graylog-master-node
graylog_168 1 p STARTED    13030801  11.4gb 137.187.63.155 graylog-master-node
graylog_168 2 p STARTED    13025547  11.4gb 137.187.63.155 graylog-master-node
graylog_168 0 p STARTED    13024011  11.4gb 137.187.63.155 graylog-master-node
graylog_147 3 p STARTED     5347445   4.7gb 137.187.63.155 graylog-master-node
graylog_147 1 p UNASSIGNED
graylog_147 2 p STARTED     5351073   4.8gb 137.187.63.155 graylog-master-node
graylog_147 0 p UNASSIGNED
graylog_166 3 p STARTED      981683   1.4gb 137.187.63.155 graylog-master-node
graylog_166 1 p STARTED      982596   1.4gb 137.187.63.155 graylog-master-node
graylog_166 2 p STARTED      981535   1.4gb 137.187.63.155 graylog-master-node
graylog_166 0 p STARTED      979641   1.4gb 137.187.63.155 graylog-master-node
graylog_132 3 p STARTED     1177483   1.2gb 137.187.63.155 graylog-master-node
graylog_132 1 p UNASSIGNED
graylog_132 2 p STARTED     1175958   1.1gb 137.187.63.155 graylog-master-node
graylog_132 0 p UNASSIGNED
graylog_150 3 p STARTED     2346955   2.4gb 137.187.63.155 graylog-master-node
graylog_150 1 p UNASSIGNED
graylog_150 2 p STARTED     2345980   2.4gb 137.187.63.155 graylog-master-node
graylog_150 0 p UNASSIGNED
graylog_155 3 p STARTED      558461 572.9mb 137.187.63.155 graylog-master-node
graylog_155 1 p STARTED      557191 570.8mb 137.187.63.155 graylog-master-node
graylog_155 2 p UNASSIGNED
graylog_155 0 p UNASSIGNED
graylog_170 3 p STARTED     3938766   3.6gb 137.187.63.155 graylog-master-node
graylog_170 1 p STARTED     3938402   6.3gb 137.187.63.155 graylog-master-node
graylog_170 2 p STARTED     3940434   3.8gb 137.187.63.155 graylog-master-node
graylog_170 0 p STARTED     3936463   3.8gb 137.187.63.155 graylog-master-node
graylog_149 3 p STARTED     2633192   2.9gb 137.187.63.155 graylog-master-node
graylog_149 1 p UNASSIGNED
graylog_149 2 p STARTED     2636264   2.9gb 137.187.63.155 graylog-master-node
graylog_149 0 p UNASSIGNED
graylog_129 3 p STARTED     1591533   1.5gb 137.187.63.155 graylog-master-node
graylog_129 1 p UNASSIGNED
graylog_129 2 p STARTED     1591007   1.5gb 137.187.63.155 graylog-master-node
graylog_129 0 p UNASSIGNED
graylog_146 3 p STARTED     1830199   2.4gb 137.187.63.155 graylog-master-node
graylog_146 1 p UNASSIGNED
graylog_146 2 p STARTED     1831944   2.4gb 137.187.63.155 graylog-master-node
graylog_146 0 p UNASSIGNED
graylog_159 3 p STARTED     1642996   1.9gb 137.187.63.155 graylog-master-node
graylog_159 1 p STARTED     1644311   1.9gb 137.187.63.155 graylog-master-node
graylog_159 2 p STARTED     1643293   1.9gb 137.187.63.155 graylog-master-node
graylog_159 0 p STARTED     1641786   1.9gb 137.187.63.155 graylog-master-node
graylog_145 3 p STARTED     2790008   3.9gb 137.187.63.155 graylog-master-node
graylog_145 1 p UNASSIGNED
graylog_145 2 p STARTED     2787973   3.9gb 137.187.63.155 graylog-master-node
graylog_145 0 p UNASSIGNED
graylog_127 3 p STARTED     1175855   1.2gb 137.187.63.155 graylog-master-node
graylog_127 1 p UNASSIGNED
graylog_127 2 p STARTED     1174157   1.2gb 137.187.63.155 graylog-master-node
graylog_127 0 p UNASSIGNED
graylog_156 3 p STARTED     4011459   4.2gb 137.187.63.155 graylog-master-node
graylog_156 1 p STARTED     4013292   4.2gb 137.187.63.155 graylog-master-node
graylog_156 2 p STARTED     4011195   4.2gb 137.187.63.155 graylog-master-node
graylog_156 0 p STARTED     4011984   4.2gb 137.187.63.155 graylog-master-node
[vubv@nimhlogs ~]$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "graylog",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 120,
  "active_shards" : 120,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 60,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 66.66666666666666
}
[vubv@nimhlogs ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5        56G  6.2G   50G  12% /
devtmpfs        7.9G     0  7.9G   0% /dev
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           7.9G  9.0M  7.9G   1% /run
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb1       900G  536G  365G  60% /var
/dev/sda1        20G  274M   20G   2% /home
/dev/sda2       497M  211M  287M  43% /boot
tmpfs           1.6G     0  1.6G   0% /run/user/984
tmpfs           1.6G   12K  1.6G   1% /run/user/42
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Graylog indices and shards are stored in /var. I have another elasticsearch server/node that is tied to this graylog server. Seems like it’s working because the elasticsearch directory is filling up.


(Hong-Duc) #2

How do you assign an unassigned shard? It says I’ve got 60 of them.


(Jochen) #3

Check the logs of your Elasticsearch node and try to find out why the shards weren’t assigned in the first place.
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


#4

If you have ES 5, you can look at the problem with the Explain API:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-explain.html

After knowing the reason, you can hopefully fix it.


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.