Thanks the help.
Ok, I have cloned the system, to do a playground.
Empty graylog,opensearch folders. The original mongo data.
The error is the same.
I recreated again with full empty folders/data. Everything is fine in the system, it shows a green elastic status, and I can recreate active index.
So the problem is somewhere in the mongo’s data.
So the docker logs:
mongo:
No error just some like that
{"t":{"$date":"2023-01-23T17:44:31.092+01:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1674492271:92567][1:0x7fde96783700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 1031585, snapshot max: 1031585 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 8741050"}}
{"t":{"$date":"2023-01-23T17:45:31.156+01:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1674492331:156534][1:0x7fde96783700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 1031751, snapshot max: 1031751 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 8741050"}}
{"t":{"$date":"2023-01-23T17:46:31.212+01:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":"[1674492391:211988][1:0x7fde96783700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 1031923, snapshot max: 1031923 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 8741050"}}
opensearch
Just info, no errors
[2023-01-23T17:39:55,531][INFO ][o.o.j.s.JobSweeper ] [graylog-opensearch] Running full sweep
[2023-01-23T17:44:55,532][INFO ][o.o.j.s.JobSweeper ] [graylog-opensearch] Running full sweep
[2023-01-23T17:46:29,302][INFO ][o.o.c.m.MetadataMappingService] [graylog-opensearch] [graylog_70/l-3UTHBSS8SHOGDMSsCD0A] update_mapping [_doc]
[2023-01-23T17:46:29,353][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [graylog-opensearch] Detected cluster change event for destination migration
Graylog:
Full with errors… 
(I treid to pick one-one errors, not the full log)
2023-01-19 12:02:37,611 WARN : org.graylog2.indexer.indices.Indices - Couldn't create index gl-failures_0. Error: No index template provider found for type 'failures'
java.lang.IllegalStateException: No index template provider found for type 'failures'
2023-01-19 12:02:37,615 ERROR: org.graylog2.periodical.IndexRotationThread - Couldn't point deflector to a new index
java.lang.RuntimeException: Could not create new target index <gl-failures_0>.
2023-01-19 12:02:39,843 ERROR: org.graylog.events.processor.EventProcessorEngine - Caught an unhandled exception while executing event processor <aggregation-v1/Threat IP-vel való kommunikáció/60aa10648ba6ce6b579ef227> - Make sure to modify the event processor to throw only EventProcessorExecutionException so we get more context!
org.graylog2.indexer.IndexNotFoundException: Unable to perform scroll search[graylog_884]
Index not found for query: graylog_884. Try recalculating your index ranges.
2023-01-19 12:02:39,896 WARN : org.graylog.plugins.map.geoip.MaxMindIpResolver - Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2023-01-19 12:02:39,900 WARN : org.graylog.plugins.map.geoip.MaxMindIpResolver - Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2023-01-19 12:02:41,217 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://graylog-opensearch:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #1).
2023-01-19 12:02:41,244 ERROR: org.graylog2.indexer.messages.Messages - Caught exception during bulk indexing: ElasticsearchException{message=ElasticsearchException[An error occurred: ]; nested: IOException[Unable to parse response body for Response{requestLine=POST /_bulk?timeout=1m HTTP/1.1, host=http://graylog-opensearch:9200, response=HTTP/1.1 200 OK}]; nested: NullPointerException;, errorDetails=[]}, retrying (attempt #2).
2023-01-19 12:02:41,557 ERROR: org.graylog.plugins.threatintel.whois.ip.WhoisIpLookup - Could not lookup WHOIS information for [192.168.1.2] at [ARIN].
2023-01-19 12:02:41,561 ERROR: org.graylog.plugins.threatintel.whois.ip.WhoisIpLookup - Could not lookup WHOIS information for [192.168.0.15] at [ARIN].
What I tried:
- delete the deflectors, It created again, and It write data to elasticsearch. I also can “rotate active write index”
- Disable GeoIP resolver, and Threat Int plugins.
- Recalculate index range, becaule it doesn’t find the graylog_884 (I deleted it, so it is normal), but the same error…
- create the 884 index. - It solved the search error, I see the data now under the search.AND it solved the cluster state error also
- create the gl-failures_0
- recalculate all index sets’ range, it was successfull
When I tried to rotate the index set, I get:
2023-01-23 18:26:15,293 INFO : org.graylog2.rest.resources.system.DeflectorResource - Cycling deflector for index set <61fc1f731bf7eb4ce04d9a7d>. Reason: REST request.
2023-01-23 18:26:15,298 INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <gl-failures_0> to <gl-failures_1>.
2023-01-23 18:26:15,299 INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <gl-failures_1>.
2023-01-23 18:26:15,306 WARN : org.graylog2.indexer.indices.Indices - Couldn't create index gl-failures_1. Error: No index template provider found for type 'failures'
java.lang.IllegalStateException: No index template provider found for type 'failures'
This is the only error left.
I did a new research, and I found this
https://community.graylog.org/t/migration-from-elasticsearch-to-opensearch-gone-wrong/27237
@gsmith mentioned a security plugin. I haven’t installed it. BUT the old graylog was an enterprise one (demo, we don’t need the feature). It could be the problem. I can’t delete the Index Set, because an stream connectred to it.
Can I delete the stream and the Index set? How?
(I tried to check how can I remove the enterprise feature, but I find only “remove the plugin and restat”)