Hosts are not visible on graylog source section

Graylog Central (peer support)
1

Hi All,

Missing hosts in “Source Section”. Collector-sidecar is running on all servers but still, servers are not visible on source section. It happened suddenly yesterday. Please help me to solve this. There were more than 30 servers but now I can see only 4 servers on the source section.

Graylog version - 2.4.6
Elasticserach version - 5.6

45%20AM
Screen Shot 2018-09-29 at 11.54.45 AM.png3464×1824 380 KB

Elasticsearch Logs

[2018-09-29T06:10:29,986][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.9gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:10:29,986][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 491.7gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.9gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:11:00,010][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 757.3gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 491.2gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:11:30,036][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 755.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:11:30,037][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.6gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:11:30,037][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 490.7gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:12:00,062][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 755.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:12:30,087][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:12:30,087][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 490.5gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 754.8gb[9.1%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node
[2018-09-29T06:12:30,088][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dqDRyP_rR5mY-2vk6SDTwA][es-data04.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 448.6gb[5.4%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [0UGFKvJ1QAmuymVk4aM6tQ][es-data03.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 489.8gb[5.9%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [dkxlWA-BTuui0UvzZ_gf_A][es-data01.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 753.4gb[9%], shards will be relocated away from this node
[2018-09-29T06:13:00,113][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-master01.mykaarma.com] high disk watermark [90%] exceeded on [_VL4LVuSS_yQCTkcbDyjrA][es-data02.mykaarma.com][/vol1/elasticsearch/data/nodes/0] free: 779.7gb[9.4%], shards will be relocated away from this node

Also data not cpming to any dashboard.

32%20AM
Screen Shot 2018-09-29 at 11.59.32 AM.png3322×1916 380 KB

Quick reply will be appreciated. Thanks

2

you need to clean up your log data - or add additional disk space to Elasticseach. Just search for high watermark and elasticsearch and you will get some resources that explain the situation.

3

Thanks @jan for your help.

4

@jan, Also I have one question. After that, I am receiving these in the logs. What does it mean…?

[2018-10-01T18:05:31,345][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:41,074][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:44,626][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:45,025][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:51,339][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:05:51,397][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:06:05,701][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:06:55,227][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:11,224][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:18,049][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:19,447][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:07:24,849][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:37,329][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:40,536][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:08:40,894][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:09:54,616][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:09:56,585][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:12:23,239][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:12:51,631][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:13:19,054][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:13:38,546][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:15:14,394][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:16:31,542][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:20:28,328][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:20:58,047][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:21:49,396][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:29:24,659][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [batchjob_logs_39/V1CUF5hsQFiWYpqmV3HT2Q] update_mapping [message]
[2018-10-01T18:49:18,475][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:02,844][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:03,253][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:54:07,819][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T18:57:04,664][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [batchjob_logs_39/V1CUF5hsQFiWYpqmV3HT2Q] update_mapping [message]
[2018-10-01T18:58:12,307][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
[2018-10-01T19:01:16,792][INFO ][o.e.c.m.MetaDataMappingService] [es-master01.mykaarma.com] [graylog_588/lI2kwe4aRxaa7qfgpYIMng] update_mapping [message]
5

This just shows that new fields are added to the index. No worry about!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.