Collected documents(logs) are getting deleted after sometime(45 mins approx.) in index and range is recalculating every 45 mins and getting started collecting logs again and the process repeats. Due to this i m losing previous(deleted) documents permenantly. The range i set for 7 days rotation. I am using 2-node setup each node contains graylog, elasticsearch, mongodb. I am guessing the issue is with elasticsearch configuration and i m unable to find the solution for it. I am attaching the relevant screenshots of conf files. Feel free to ask me for any further details if needed.
Your system is in an inconsistent state. If you search on the error message in this forum you will find several threads that offer trouble-shooting steps.
Typically, Elastic/Opensearch and the Graylog MongoDB have gotten out of sync, e.g. due to manually deleting data in ES/OS. If that is an option for you, you could drop the entire Graylog database in Mongo and restart.
Strange. Need more information to figure this out. Right now it’s just guesswork.
You pasted settings for the global index set defaults rotation_strategy and elasticsearch_max_time_per_index. How are the actual indices configured?
Are there other errors in the log?
Check your ES cluster health as described in related thread:
Actual Indices configurations:
Index shards: 2
Index replicas: 0
Field type refresh interval:5
rotation strategy: Index Time
Rotation period: P1W
retention strategy: Delete Index
Max number of indices: 20
2.There are other errors in logs
3.After check ES cluster health using command
curl -XGET http://es_node:9200/_cluster/allocation/explain?pretty
Getting this Message
“error” : {
“root_cause” : [
{
“type” : “illegal_argument_exception”,
“reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]”
}
],
“type” : “illegal_argument_exception”,
“reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]”
},
“status” : 400
I think the main problem with elastisearch as it is deletings indices
Not sure what’s going on - sorry. What exactly do you mean when you say logs are being deleted? The index is being deleted in ES? Is there anything in GL or ES logs that might indicate what is initiating the deletion?
The documents(logs from Syslogs) collected in an index are unexpectedly deleted, and the index does not rotate to a new set.For example, if an index initially collects 98 documents, after some time, it shows 0 documents collected, and the system starts collecting documents (logs) again from the beginning.
And there is no error log regarding this issuse in GL or ES logs
My Elasticsearch (ES) is not displaying any logs, including general Elasticsearch log
Graylog will never delete messages from an index. Entire indices get rotated and may ultimately be deleted, based on the rotation and retention settings for the index set.
Something else is going on here. I agree that it sounds like something is happening on the ES side; and the GL errors are just a consequence of that. You mention this happening at 45 minute intervals. What could be running on that schedule?
I foolishly didn’t read your initial post thoroughly. Currently you are running an unsupported version of Elastic, the last supported version was 7.10.2.
At this point my recommendation would be to move to Opensearch 2.9 and see if the issue persists, if the system is so compromised that it currently can’t store data then use it as an opportunity to start fresh with a new install.
