Delete/Reload Data

My testing is now well underway, my biggest issue is how to retest. I have everything running, filebeat on my laptop is sending logs I copy to a specified folder, extractors etc parse the data.

As I work on extractors I’m looking for a way to repeatedly submit the same data and verify results, to do so I believe that I need two things:

  1. A way to delete messages from graylog/elasticsearch
  2. A way to reset filebeat so that I can repeatedly resubmit the same file and have it process

Is there a recommended way to do these things? I have tried the “cleanse” with graylog-ctl but I think it wipres my whole configuration, I’ve also tried deleting an index but that seems to give me some unexpected results trying to get things going again. Can I generate a query in graylog then use that somehow with CURL to delete those messages?

Similarly with filebeat - does it remember the files I have submitted already? If I start fresh and copy a file to the folder it is consumed, how do I get it to consume it again? Delete/re-copy or do I need to reset the filebeat registry?

Things are going very well and are exciting, my data is just a little messy as I evolve my extraction :slight_smile:

Chris.

You can simply delete the old indices in Graylog, see System / Indices.

Please take note that you cannot delete the currently active index, so you’ll have to manually rotate indices (Maintenance / Rotate active write index).

You can remove the Filebeat registry file: Updating the Registry File | Filebeat Reference [5.5] | Elastic

Thanks Jochen, the Index piece worked beautifully, thank you.

Regarding filebeat under C:\Program Files\Graylog\collector-sidecar\cache\filebeat\data I see two files, registry and registry.old - should I stop the Sidecar service, delete these files and restart?

I read a little about doing this via filebeat but that seems to require modifying filebeat.yml which is regenerated and seems to overwrite any changes I try to make.

Yes, exactly.

You shouldn’t need to modify the Filebeat configuration for that.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.