Removing unwanted logs from an index

[m_mlk]

1. Describe your incident:
Some sensible information between 2 dates has been logged in GL by mistake and now I need to find and remove it from the corresponding index.

2. Describe your environment:

• OS Information: Ubuntu 20.04 LTS

• Package Version: GL 4.3.9 + OpenSearch 1.3.3

• Service logs, configurations, and environment variables:
  

  

3. What steps have you already taken to try and solve the problem?

I found this post Deleting logs from graylog/elasticsearch (a howto) but I don’t feel quite confident about its approach.

4. How can the community help?

Is it OK if I change the rotation method to:

…and apply the changes? would that auto-remove everything older than, say, 1 day?

If not, what would be the recommended method to purge those unwanted logs from the index?

TIA!

[tmacgbay]

I had something similar happen that made my data dirty - I wrote up a short post on it here that might help get you started down the road. The short is that I created a curl command that would find messages with a particular field:data combination and I deleted it. Shouldn’t be that hard to modify it for a date range.

Assuming that this is enough to magically make it work for you, make sure you post back what your solution was for future searchers!!!

EDIT: The link to the script you posted has at it’s core something similar to my curl command - results may vary per elastic version you are on.

If you don’t mind loosing all the historical data, the cleanest way to do it is as you suggest, shorten the retention to a day and rotate your index manually. That way you can avoid scripts and curl etc…

[gsmith]

@m_mlk

and apply the changes? would that auto-remove everything older than, say, 1 day?

Short answer is yes.
After that you can manually rotate the index set.
I would suggest using @tmacgbay suggestion if you can.