Delete/Reload Data

Graylog Central (peer support)
1

My testing is now well underway, my biggest issue is how to retest. I have everything running, filebeat on my laptop is sending logs I copy to a specified folder, extractors etc parse the data.

As I work on extractors I’m looking for a way to repeatedly submit the same data and verify results, to do so I believe that I need two things:

  1. A way to delete messages from graylog/elasticsearch
  2. A way to reset filebeat so that I can repeatedly resubmit the same file and have it process

Is there a recommended way to do these things? I have tried the “cleanse” with graylog-ctl but I think it wipres my whole configuration, I’ve also tried deleting an index but that seems to give me some unexpected results trying to get things going again. Can I generate a query in graylog then use that somehow with CURL to delete those messages?

Similarly with filebeat - does it remember the files I have submitted already? If I start fresh and copy a file to the folder it is consumed, how do I get it to consume it again? Delete/re-copy or do I need to reset the filebeat registry?

Things are going very well and are exciting, my data is just a little messy as I evolve my extraction :slight_smile:

Chris.

2

You can simply delete the old indices in Graylog, see System / Indices.

Please take note that you cannot delete the currently active index, so you’ll have to manually rotate indices (Maintenance / Rotate active write index).

You can remove the Filebeat registry file: Updating the Registry File | Filebeat Reference [5.5] | Elastic

3

Thanks Jochen, the Index piece worked beautifully, thank you.

Regarding filebeat under C:\Program Files\Graylog\collector-sidecar\cache\filebeat\data I see two files, registry and registry.old - should I stop the Sidecar service, delete these files and restart?

I read a little about doing this via filebeat but that seems to require modifying filebeat.yml which is regenerated and seems to overwrite any changes I try to make.

4

Yes, exactly.

You shouldn’t need to modify the Filebeat configuration for that.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.