Hosts are not visible on graylog source section


(Tafsir) #1

Hi All,

Missing hosts in “Source Section”. Collector-sidecar is running on all servers but still, servers are not visible on source section. It happened suddenly yesterday. Please help me to solve this. There were more than 30 servers but now I can see only 4 servers on the source section.

Graylog version - 2.4.6
Elasticserach version - 5.6

Elasticsearch Logs

[2018-09-29T06:10:29,986][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.9gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:10:29,986][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 491.7gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.9gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 757.3gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 491.2gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 755.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:11:30,037][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.6gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:11:30,037][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 490.7gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 755.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:12:30,087][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:12:30,087][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 490.5gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 754.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 489.8gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 753.4gb[9%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node

Also data not cpming to any dashboard.

Quick reply will be appreciated. Thanks


(Jan Doberstein) #2

you need to clean up your log data - or add additional disk space to Elasticseach. Just search for high watermark and elasticsearch and you will get some resources that explain the situation.


(Tafsir) #3

Thanks @jan for your help.


(Tafsir) #4

@jan, Also I have one question. After that, I am receiving these in the logs. What does it mean…?

[2018-10-01T18:05:31,345][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:41,074][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:44,626][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:45,025][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:51,339][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:51,397][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:06:05,701][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:06:55,227][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:11,224][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:18,049][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:19,447][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:24,849][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:37,329][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:40,536][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:40,894][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:09:54,616][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:09:56,585][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:12:23,239][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:12:51,631][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:13:19,054][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:13:38,546][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:15:14,394][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:16:31,542][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:20:28,328][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:20:58,047][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:21:49,396][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:29:24,659][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [batchjob_logs_39/V1CUF5hsQFiWYpqmV3HT2Q] update_mapping [message]
[2018-10-01T18:49:18,475][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:02,844][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:03,253][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:07,819][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:57:04,664][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [batchjob_logs_39/V1CUF5hsQFiWYpqmV3HT2Q] update_mapping [message]
[2018-10-01T18:58:12,307][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T19:01:16,792][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]

(Jan Doberstein) #5

This just shows that new fields are added to the index. No worry about!


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.