There were 204,800 failed indexing attempts

Running into what I believe is a very similar problem. I noticed the 200,000+ failed indexing attempts, had the same disk space errors, and can’t get files to process. I am not sure if Olivier was but i am also running off an OVA (upgraded from 3.2->3.3)

Already extended disk

— Volume group —
VG Name graylog-vg
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 79.52 GiB
PE Size 4.00 MiB
Total PE 20358
Alloc PE / Size 20103 / <78.53 GiB
Free PE / Size 255 / 1020.00 MiB
VG UUID gKfX8H-mExe-Q28P-AXMf-bjXe-5wCu-3NpgyN

Not seeing “[WARN ][o.e.c.r.a.DiskThresholdMonitor] [adXw5AG] high disk watermark [90%] exceeded on [adXw5AGtSGCzRcv6DeU5PQ][adXw5AG][/var/lib/elasticsearch/nodes/0]” anymore. Not seeing any more errors in /var/log/elasticsearch/graylog-server.log

Tried stopping GL/ES/MDB deleting journal logs and rebooting
tried running: curl -X PUT “localhost:9200/_all/_settings” -H ‘Content-Type: application/json’ -d’{ “index.blocks.read_only” : false } }’
and
curl -XPUT -H “Content-Type: application/json” http://localhost:9200/_all/_settings -d ‘{“index.blocks.read_only_allow_delete”: null}’
I am however still seeing: Deflector is pointing to [gl-events_1], not the newest one: [gl-events_2]. Re-pointing.

Lots of this repeated in the var/log/graylog-server/server.log

blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at org.graylog2.indexer.cluster.jest.JestUtils.specificException(JestUtils.java:110) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:60) ~[graylog.jar:?]
at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.cycleAlias(Indices.java:655) ~[graylog.jar:?]
at org.graylog2.indexer.MongoIndexSet.pointTo(MongoIndexSet.java:357) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.checkAndRepair(IndexRotationThread.java:166) ~[graylog.jar:?]
at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:76) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) [?:?]
at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
2020-06-13T07:08:12.431Z WARN [IndexRotationThread] Deflector is pointing to [gl-events_1], not the newest one: [gl-events_2]. Re-pointing.
2020-06-13T07:08:12.433Z ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t switch alias gl-events_deflector from index gl-events_1 to index gl-events_2

I have tried manually doing an index rotation but it just creates an empty index and doesn’t even change it to the primary. Not sure where to go next.

Nevermind ran:

curl -XPUT -H “Content-Type: application/json” http://localhost:9200/_all/_settings -d ‘{“index.blocks.read_only_allow_delete”: null}’

again and rebooted and it finally started picking up

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.