I have a rather large deployment that seems to have difficulty identifying the indices required to open in order to perform a search. When I run a search on the last 5 mins I expect the current indice to be the only indice opened in order to fulfill the query (if the indice holds more than 5mins which it does), is that not a correct assumption?
I have rerun the maintenance job a few times to recalculate the ranges but it doesn’t seem to have helped. I can’t see any errors in the logs, is there anything I should be investigating further?
I have 700+ indices, replicated 3 times at 50GB each across 25+ ES nodes storing about ~40TB of data for reference. Ingestion rate is about 7500 msgs/sec. 3 Graylog nodes, about to be expanded to 4 possibly 5. ES 2.3.4 Centos 7.2 Graylog 2.2.0-11 java 1.8.0.65-2
Someone else feel free to correct me if I’m wrong, but I’m pretty sure the search will return all messages that match the query (like you said, 5 minutes) from all nodes/indices within the cluster. Unless you specify a certain node, it will return all messages that match that query within every node and every active index.
If you’re wanting the logs from just one node, you can specify by node in the query statement.
The index ranges simply store about the oldest message and the newest message in an index, in other words what timespan is covered by the index.
If you’re recording messages with invalid time stamps, it’s possible that queries will have to include all indices because they might contain the wanted messages according to the calculated index ranges.
The first step for you would be to check the calculated index ranges for anomalies, like always the same end date.
okay yeah fairly certain this is the cause. Any idea how to limit a search to a certain set of indices to try and determine the source?
ie if I have 70 odd indices being opened, if I limit the search to only the oldest index returning a result then I should see the source and be able to remove it
