# Collected documents(logs) are getting deleted in index **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [November 15, 2023, 2:59pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685 "2023-11-15T14:59:38Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 15, 2023, 2:59pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/1 "2023-11-15T14:59:39Z")
1. Collected documents(logs) are getting deleted after sometime(45 mins approx.) in index and range is recalculating every 45 mins and getting started collecting logs again and the process repeats. Due to this i m losing previous(deleted) documents permenantly. The range i set for 7 days rotation. I am using 2-node setup each node contains graylog, elasticsearch, mongodb. I am guessing the issue is with elasticsearch configuration and i m unable to find the solution for it. I am attaching the relevant screenshots of conf files. Feel free to ask me for any further details if needed. - Package Version: graylog - 5.0.3 elasticsearch - 8.11 mongodb - 6.0 ![Screenshot 2023-11-15 at 8.19.09 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0a7da8daef808c68aa4e17edba54d1f55c331752.png) graylog node -1 configuration for node i just kept no commented is\_master = true ![Screenshot 2023-11-15 at 8.24.47 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/95da3555f11c5c057804e1a79002327cb681ff58.png) elasticsearch node-1 configuration ![Screenshot 2023-11-15 at 8.26.57 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0548bc944fb44644f0eb98489451a1a87bd077c8.png) elasticsearch node-2 configuration **Helpful Posting Tips:** [Tips for Posting Questions that Get Answers](https://community.graylog.org/t/tips-for-posting-questions-that-get-answers/21828) [Hold down CTRL and link on link to open tips documents in a separate tab] ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [November 16, 2023, 9:59am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/2 "2023-11-16T09:59:39Z")
What does the log file show at the time of rotation/deletion? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 16, 2023, 10:23am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/3 "2023-11-16T10:23:47Z")
The Graylog log file continuously show this logs ![Screenshot 2023-11-16 at 3.52.53 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c2f4180830bd71d42be13ebe2249388765f35cd9.jpeg) ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [November 16, 2023, 10:45am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/4 "2023-11-16T10:45:42Z")
Your system is in an inconsistent state. If you search on the error message in this forum you will find several threads that offer trouble-shooting steps. Typically, Elastic/Opensearch and the Graylog MongoDB have gotten out of sync, e.g. due to manually deleting data in ES/OS. If that is an option for you, you could drop the entire Graylog database in Mongo and restart. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 16, 2023, 11:46am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/5 "2023-11-16T11:46:35Z")
I completed the entire process, dropped the Graylog database, and restarted both MongoDB and Graylog, but the problem is repeating. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 16, 2023, 11:49am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/6 "2023-11-16T11:49:58Z")
My index range is recalculating every time and data is getting lost ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [November 16, 2023, 1:40pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/7 "2023-11-16T13:40:06Z")
Strange. Need more information to figure this out. Right now it’s just guesswork. You pasted settings for the global index set defaults `rotation_strategy` and `elasticsearch_max_time_per_index`. How are the actual indices configured? Are there other errors in the log? Check your ES cluster health as described in related thread: > [@Remove the enterprise and restart : crash with error on gl\_failures index](https://community.graylog.org/t/remove-the-enterprise-and-restart-crash-with-error-on-gl-failures-index/26524/2): > > Hello Can I ask what packages you removed? and as for the error/s WARN [Indices] Couldn't create index gl-failures\_0. Error: No index template provider found for type 'failures' ERROR [IndexRotationThread] Couldn't point deflector to a new index java.lang.RuntimeException: Could not create new target index \. Have you tried check the health, indices, etc… of elasticsearch using cURL? check’s examples Dangling indices curl -X GET http://es\_node:9200/\_dangling?pretty Check c… ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 16, 2023, 2:01pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/8 "2023-11-16T14:01:36Z")
Actual Indices configurations: Index shards: 2 Index replicas: 0 Field type refresh interval:5 rotation strategy: Index Time Rotation period: P1W retention strategy: Delete Index Max number of indices: 20 2.There are other errors in logs 3.After check ES cluster health using command curl -XGET http://es\_node:9200/\_cluster/allocation/explain?pretty Getting this Message “error” : { “root\_cause” : [ { “type” : “illegal\_argument\_exception”, “reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]” } ], “type” : “illegal\_argument\_exception”, “reason” : “unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]” }, “status” : 400 I think the main problem with elastisearch as it is deletings indices ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [November 17, 2023, 9:41am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/9 "2023-11-17T09:41:30Z")
Not sure what’s going on - sorry. What exactly do you mean when you say logs are being deleted? The index is being deleted in ES? Is there anything in GL or ES logs that might indicate what is initiating the deletion? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 17, 2023, 11:03am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/10 "2023-11-17T11:03:27Z")
- The documents(logs from Syslogs) collected in an index are unexpectedly deleted, and the index does not rotate to a new set.For example, if an index initially collects 98 documents, after some time, it shows 0 documents collected, and the system starts collecting documents (logs) again from the beginning. - And there is no error log regarding this issuse in GL or ES logs - My Elasticsearch (ES) is not displaying any logs, including general Elasticsearch log ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [November 17, 2023, 11:23am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/11 "2023-11-17T11:23:14Z")
Graylog will never delete messages from an index. Entire indices get rotated and may ultimately be deleted, based on the rotation and retention settings for the index set. Something else is going on here. I agree that it sounds like something is happening on the ES side; and the GL errors are just a consequence of that. You mention this happening at 45 minute intervals. What could be running on that schedule? ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 17, 2023, 3:00pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/12 "2023-11-17T15:00:22Z")
- The timing is not fixed; sometimes it occurs every 45 minutes, sometimes every hour, and occasionally even after 24 hours. - There is nothing scheduled at that time. ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [November 20, 2023, 1:54pm UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/13 "2023-11-20T13:54:46Z")
Hello @adarsh, How is the range being recalculated every 45 mins? What is the output of these individual api calls? `curl -XGET "http://node.name:9200/_cat/shards?=pretty"` and `curl -XGET "http://node.name:9200/_cat/indices?=pretty"` and `curl -XGET "http://node.name:9200/_cluster/health?=pretty"` ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 21, 2023, 9:12am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/14 "2023-11-21T09:12:24Z")
- The timing is not fixed; sometimes it occurs every 45 minutes, sometimes every hour, and occasionally even after 24 hours. **curl -XGET “[http://node.name:9200/\_cat/shards?=pretty](http://node.name:9200/_cat/shards?=pretty)”`** ![Screenshot 2023-11-21 at 2.32.11 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/57fb9bcb81e8757b751cbb8c83fe2d859c2809f5.png) **curl -XGET “[http://node.name:9200/\_cat/indices?=pretty](http://node.name:9200/_cat/indices?=pretty)”** ![Screenshot 2023-11-21 at 2.39.06 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/f/ffca93bf6187d6e6a5f24a7cfa052d273581c4c6.png) **curl -XGET “[http://node.name:9200/\_cluster/health?=pretty](http://node.name:9200/_cluster/health?=pretty)”** ![Screenshot 2023-11-21 at 2.40.39 PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/f/fcaed927da68a94936b20e394ceeb5d05b4e22f9.png) ---
**Author:** ![Wine\_Merchant](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/wine_merchant/32/14596_2.png) [@Wine\_Merchant](https://community.graylog.org/u/Wine_Merchant)\ **Post date:** [November 21, 2023, 10:13am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/15 "2023-11-21T10:13:00Z")
I foolishly didn’t read your initial post thoroughly. Currently you are running an unsupported version of Elastic, the last supported version was 7.10.2. At this point my recommendation would be to move to Opensearch 2.9 and see if the issue persists, if the system is so compromised that it currently can’t store data then use it as an opportunity to start fresh with a new install. ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [November 29, 2023, 5:14am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/16 "2023-11-29T05:14:07Z")
My Graylog logs are filled with this error. Is there any connection with my problem? ![Screenshot 2023-11-29 at 10.41.27 AM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/b94b6be96cf5f2b89557a28fd173b2018c3bbf5c.jpeg) ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [November 29, 2023, 5:57am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/17 "2023-11-29T05:57:29Z")
Hey @adarsh Just chimming in, I looked over you configuration for ES/OS, Since this looks like a cluster of 2 nodes I dont see the following configured. ```auto node.roles: [cluster_manager] ``` > **[Creating a cluster](https://opensearch.org/docs/latest/tuning-your-cluster/index/)** > > Creating a cluster ---
**Author:** ![adarsh](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@adarsh](https://community.graylog.org/u/adarsh)\ **Post date:** [December 9, 2023, 3:52am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/18 "2023-12-09T03:52:32Z")
I entirely changed my configuration from Elasticsearch to OpenSearch with these settings, but I am still getting the same error in the Graylog logs My node-1 configuration (cluster manager) ![Screenshot 2023-12-09 at 9.16.00 AM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/8/8b13d1c345e96ddbe212ef36cafb04391617acca.png) Node-2 configuration (data node) ![Screenshot 2023-12-09 at 9.17.15 AM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/3dfc917a11207449e2dd64d69942a6abbf67fbea.png) ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [December 11, 2023, 7:28am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/19 "2023-12-11T07:28:55Z")
Did you also rebuild your mongoDB when you switched to opensearch? You need to make sure Mongo doesn’t still have data from the old indices. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [December 25, 2023, 7:29am UTC](https://community.graylog.org/t/collected-documents-logs-are-getting-deleted-in-index/30685/20 "2023-12-25T07:29:18Z")
Closed on December 25, 2023, 7:29am UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.