Remove the enterprise and restart : crash with error on gl_failures index

Graylog Central (peer support)
1

1. Describe your incident:
I try to remove the enterprise packages in Debian for Graylog 4.3. Then after restarting all, there is a log each 10 seconds :

2022-11-13T16:33:43.054+01:00 INFO  [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2022-11-13T16:33:43.055+01:00 INFO  [MongoIndexSet] There is no index target to point to. Creating one now.
2022-11-13T16:33:43.056+01:00 INFO  [MongoIndexSet] Cycling from <none> to <gl-failures_0>.
2022-11-13T16:33:43.056+01:00 INFO  [MongoIndexSet] Creating target index <gl-failures_0>.
2022-11-13T16:33:43.056+01:00 WARN  [Indices] Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
java.lang.IllegalStateException: No index template provider found for type 'failures'
        at org.graylog2.indexer.IndexMappingFactory.resolveIndexMappingTemplateProvider(IndexMappingFactory.java:58) ~[graylog.jar:?]
        at org.graylog2.indexer.IndexMappingFactory.createIndexMapping(IndexMappingFactory.java:50) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.buildTemplate(Indices.java:223) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.ensureIndexTemplate(Indices.java:173) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.create(Indices.java:210) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:291) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:94) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-11-13T16:33:43.057+01:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
        at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:292) ~[graylog.jar:?]
        at org.graylog2.indexer.MongoIndexSet.setUp(MongoIndexSet.java:259) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:138) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
        at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:94) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

2. Describe your environment:

  • OS Information: Debian bullseye, Graylog 4.3.9-1

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
I try to reinstall (with purge) all the elasticsearch stack, and the graylog stack. I kept MongoDB to keep the configurations.
When restarting, I force the indexes to 0 for the normal process (with help of Could not execute search(Index not found for query) - #4 by jan page)

2

Hello

Can I ask what packages you removed? and as for the error/s

WARN  [Indices] Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
ERROR [IndexRotationThread] Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.

Have you tried check the health, indices, etc… of elasticsearch using cURL?

check’s examples

Dangling indices

curl -X GET http://es_node:9200/_dangling?pretty

Check cluster with an explanation

curl -XGET http://es_node:9200/_cluster/allocation/explain?pretty

or

Check health

curl -XGET http://es_node:9200/_cluster/health?pretty=true

or

Elasticsearch’s cat shards API will tell you which shards are unassigned, and why:

curl -XGET http://es_node:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

From what I get in the logs Graylog is looking for the Index prefix: gl-failures

3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.