My testing is now well underway, my biggest issue is how to retest. I have everything running, filebeat on my laptop is sending logs I copy to a specified folder, extractors etc parse the data.
As I work on extractors I’m looking for a way to repeatedly submit the same data and verify results, to do so I believe that I need two things:
- A way to delete messages from graylog/elasticsearch
- A way to reset filebeat so that I can repeatedly resubmit the same file and have it process
Is there a recommended way to do these things? I have tried the “cleanse” with graylog-ctl but I think it wipres my whole configuration, I’ve also tried deleting an index but that seems to give me some unexpected results trying to get things going again. Can I generate a query in graylog then use that somehow with CURL to delete those messages?
Similarly with filebeat - does it remember the files I have submitted already? If I start fresh and copy a file to the folder it is consumed, how do I get it to consume it again? Delete/re-copy or do I need to reset the filebeat registry?
Things are going very well and are exciting, my data is just a little messy as I evolve my extraction