Hi,
I have created a couple of indices what I have set to act as retention policies.
Now that i have created the index, where do I set the option to say to which message should it pass a certain rule to go to that index rather than default. index?
Hey @mr_m_cox,
this is done via streams. When you create or edit a stream, you can choose which index it will write to and if the messages should be removed from the default index.
Greetings - Phil
Great, thanks for the pointer.
As I understand it then I can set an index to only have say 1000 per index and only 2 indexes means that I will never have more than 2000 of those messages saved at anyone time?
If you set the retention policy to delete, yes, that is correct.
I guess this will be very inefficient. You got more than 200 logs because the index rotation is not checked that often I guess.
Take a look at the drop_message() function of Graylog Pipelines. This will be way more efficient, reliable and correct (since it’s the way the Graylog Devs intended for abundant message deletion).
I did set the limits especially low to allow for quick and clear testing.
If I set the limit to 1000 over 2 indices would that likely be better?
Well, try it. I don’t know how well the index retention algorithm handles short periods.
I got a testing index that holds 100000 messages in 2 indices and a second one that holds 3 days worth of messages. They both never exceed 50MB. Why do you need such a short period?
Greetings - Phil
I don’t really need it that short, just wanted to be able to see it happen ‘live’ for testing.
I can see the index rotation can be timed in days so if i set that to 1 day and then allow 2 indices with a delete setting that should give me 2 days worth at any one time?
Something like this…
@mr_m_cox you might want to reread this part of the documentation
1 Like
Correct, that will keep the last two days (keep in mind, that the current day is also included). So these settings would for example hold the data of monday (yesterday) and today.
1 Like