Hello.
I have a Graylog 3.0.2+1686930, single node with ES 6.8.
Logs are shipped this way: filebeat → kafka → graylog (input → all messages → pipelines → route to 2 streams)
Shipping works well, i see the logs in the kafka topic.
Graylog accept and index the log, i can see the logs in elasticsearch. But since 2019-07-22 19:55:25.844 nothing appears in All Messages stream or in “show received message” in the input.
I have this message log from that time:
2019-07-22T19:55:25.610+02:00 INFO [AbstractRotationStrategy] Deflector index (index set <graylog_4>) should be rotated, Pointing deflector to new index now!
2019-07-22T19:55:25.970+02:00 INFO [MongoIndexSet] Cycling from <graylog_4> to <graylog_5>.
2019-07-22T19:55:25.970+02:00 INFO [MongoIndexSet] Creating target index <graylog_5>.
2019-07-22T19:55:26.222+02:00 INFO [Indices] Successfully created index template graylog-internal
2019-07-22T19:55:26.375+02:00 INFO [MongoIndexSet] Waiting for allocation of index <graylog_5>.
2019-07-22T19:55:26.547+02:00 INFO [MongoIndexSet] Index <graylog_5> has been successfully allocated.
2019-07-22T19:55:26.547+02:00 INFO [MongoIndexSet] Pointing index alias <graylog_deflector> to new index <graylog_5>.
2019-07-22T19:55:26.631+02:00 INFO [SystemJobManager] Submitted SystemJob [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2019-07-22T19:55:26.631+02:00 INFO [MongoIndexSet] Successfully pointed index alias <graylog_deflector> to index <graylog_5>.
2019-07-22T19:55:56.641+02:00 INFO [SetIndexReadOnlyJob] Flushing old index <graylog_4>.
2019-07-22T19:55:57.122+02:00 INFO [SetIndexReadOnlyJob] Setting old index <graylog_4> to read-only.
2019-07-22T19:55:57.142+02:00 INFO [SystemJobManager] Submitted SystemJob [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2019-07-22T19:55:57.156+02:00 INFO [OptimizeIndexJob] Optimizing index <graylog_4>.
2019-07-22T19:55:57.157+02:00 INFO [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_4.
2019-07-22T19:55:58.887+02:00 INFO [MongoIndexRangeService] Calculated range of [graylog_4] in [1729ms].
2019-07-22T19:55:58.890+02:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index graylog_4.
2019-07-22T19:55:58.891+02:00 INFO [SystemJobManager] SystemJob [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 2259ms.
2019-07-22T19:58:25.523+02:00 INFO [SystemJobManager] SystemJob [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 148380ms.
I suppose this behaviour is linked to the index rotation.
I had to recalculate index range in the index set to make it work again.
I have this messages in the log when i clicked:
2019-07-23T10:23:25.324+02:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2019-07-23T10:23:25.324+02:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2019-07-23T10:23:45.383+02:00 INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-07-23T10:23:46.194+02:00 INFO [MongoIndexSet] Pointing to already existing index target <graylog_5>
Is this some part of the configuration i missed? I don’t remember changing something related.
Thanks for any pointer