Hi All,
We are running Graylog 2.3 & Elasticsearch 5.5.2 version in Ubuntu 16.04 machine and I am getting this below notifications in graylog,
Deflector exists as an index and is not an alias. (triggered 44 minutes ago)
The deflector is meant to be an alias but exists as an index. Multiple failures of infrastructure can lead to this. Your messages are still indexed but searches and all maintenance tasks will fail or produce incorrect results. It is strongly recommend that you act as soon as possible.
and also I am getting this error message in graylog server.log file,
2017-11-21T12:37:07.213Z INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2017-11-21T12:37:07.213Z INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2017-11-21T12:37:07.214Z INFO [MongoIndexSet] Cycling from <none> to <accessdata_0>.
2017-11-21T12:37:07.214Z INFO [MongoIndexSet] Creating target index <accessdata_0>.
2017-11-21T12:37:07.231Z INFO [Indices] Successfully created index template accessdata-template
2017-11-21T12:37:07.455Z INFO [MongoIndexSet] Waiting for allocation of index <accessdata_0>.
2017-11-21T12:37:07.457Z INFO [MongoIndexSet] Index <accessdata_0> has been successfully allocated.
2017-11-21T12:37:07.457Z INFO [MongoIndexSet] Pointing index alias <accessdata_deflector> to new index <accessdata_0>.
2017-11-21T12:37:07.477Z INFO [MongoIndexSet] Successfully pointed index alias <accessdata_deflector> to index <accessdata_0>.
2017-11-21T12:39:37.200Z WARN [IndexRotationThread] There is an index called [default_deflector]. Cannot fix this automatically and published a notification.
2017-11-21T12:40:57.216Z INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2017-11-21T12:41:12.224Z INFO [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2017-11-21T12:44:45.093Z INFO [RebuildIndexRangesJob] Recalculating index ranges.
2017-11-21T12:44:45.095Z INFO [RebuildIndexRangesJob] Recalculating index ranges for index set accessdata (accessdata_*): 1 indices affected.
2017-11-21T12:44:45.096Z INFO [RebuildIndexRangesJob] Done calculating index ranges for 1 indices. Took 1ms.
2017-11-21T12:44:45.096Z INFO [SystemJobManager] SystemJob <c0700d50-ceb9-1-000d3aa32669> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 3ms.
2017-11-21T12:44:45.096Z INFO [SystemJobManager] Submitted SystemJob <c0700d50-ceb9-1-000d3aa32669> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
also I could understand in the logs the elasticsearch is down for sometime but not sure on what could be the reason for elasticsearch cluster down…
Below is the error logs of elasticsearch,
[2017-11-21T12:37:04,570][INFO ][o.e.c.m.MetaDataDeleteIndexService] [_637cuJ] [proxyaccessdata_0/GrhQOX8HQv2T-jXaphKF3w] deleting index
[2017-11-21T12:37:07,235][INFO ][o.e.c.m.MetaDataCreateIndexService] [_637cuJ] [accessdata_0] creating index, cause [api], templates [accessdata-template], shards [4]/[0], mappings [message]
[2017-11-21T12:37:07,436][INFO ][o.e.c.r.a.AllocationService] [_637cuJ] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[accessdata_0][3], [accessdata_0][0]] ...]).
[2017-11-21T12:39:25,697][INFO ][o.e.c.m.MetaDataDeleteIndexService] [_637cuJ] [default_1/gcPzSwUGd_d5hlczsGQ] deleting index
[2017-11-21T12:39:25,697][INFO ][o.e.c.m.MetaDataDeleteIndexService] [_637cuJ] [default_0/YZHEKVPhQ2Wht6B1y4OYuA] deleting index
[2017-11-21T12:39:30,099][INFO ][o.e.c.m.MetaDataCreateIndexService] [_637cuJ] [default_deflector] creating index, cause [auto(bulk api)], templates [graylog-internal], shards [5]/[1], mappings [message]
[2017-11-21T12:39:30,478][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:30,504][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:30,571][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:31,098][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:31,127][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:31,199][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:31,223][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] update_mapping [message]
[2017-11-21T12:39:49,782][INFO ][o.e.c.m.MetaDataDeleteIndexService] [_637cuJ] [default_deflector/Fd9jXOb8SVK_yctZ9uveOA] deleting index
[2017-11-21T12:39:50,091][INFO ][o.e.c.m.MetaDataCreateIndexService] [_637cuJ] [default_deflector] creating index, cause [auto(bulk api)], templates [graylog-internal], shards [5]/[1], mappings [message]
[2017-11-21T12:39:50,408][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
[2017-11-21T12:39:50,472][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
[2017-11-21T12:39:50,501][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
[2017-11-21T12:39:59,091][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
[2017-11-21T12:40:01,099][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
[2017-11-21T12:40:01,128][INFO ][o.e.c.m.MetaDataMappingService] [_637cuJ] [default_deflector/jTYd-czuRJGUmCYSbvoVZA] update_mapping [message]
Note:-
I have checked some other threads and I stopped the graylog and delete the default_deflector index from elasticsearch and then warning messages are stopped showing in logs.
We are not sure how the elasticsearch calling this “MetaDataDeleteIndexService” automatically and started deleting the indices.
Is there any inbuilt function in graylog to make this call due to over utilization of memory in elasticsearch or related to some other thing?
Please kindly share any thoughts.
Thanks,
Ganeshbabu R