Separate storage time

Development
,
1

Hello. In our company, we use different inputs for different services. Could you add a setting for each input, so that we can set how old are we you need to store the logs for this input? Because I need to store some logs for a maximum of a week, and the other 30 days, and it is very inconvenient when all the data from different inputs is stored at the same time. For example: I need to store the input “foo” logs for 7 days, and the input data “bar” for 30 days. I think this feature will be useful to many. Thanks.

2

Data storage time/count is based on which stream index you place it in. Build a different index for each retention style you have and use pipelines to route the data to the stream associated to the index retention you want.

3

can you show me some examples please?

4

Building Indexes:
https://docs.graylog.org/en/latest/pages/configuration/index_model.html

Building streams that flow to an index - you can have multiple streams go to a single index so think of it as tagging. You can either have stream rules that decides what messages are put into that particular stream or you can use pipelines to shift a message to a stream based on what it finds inthe message
https://docs.graylog.org/en/latest/pages/streams.html

Building pipelines that wrap around a stream which allows you to apply changes to the messages in the stream - (You can also apply changes with extractors associated with inputs but you didn’t ask about that.)
https://docs.graylog.org/en/latest/pages/pipelines/pipelines.html

Example pipeline rule that shifts a message from a firewall stream with a short term index to a Remote Access stream that has an index that retains longer with the route_to_stream() function:

rule "PA-Firewall - ex1 - GLOBALPROTECT fields"
when
    has_field("log_type")           &&
    to_string($message.log_type) == "GLOBALPROTECT"
then
    set_field(field: "ra_tag", value: "globalprotect");

    //This is no longer a firewall event - its a remote access event.
    route_to_stream(name: "Remote Access Global");

end
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.