Currently all of my logs are stored in the default index.
But now I need to split them into multiples indexes.
As far as I know the only way to route logs into indexes is to use Streams. Am I correct ?
So I need to create Stream A to route logs into Index A and Stream B for Index B (and removed them from All messages to not duplicate logs).
The problem is Streams can also be used as an input for Alerts.
For example I’ve a Stream for failed authentications, another for blocked connections…
So it means if I have failed authentications in logs related to Index A and in those related to Index B, I must create 2 Streams : failed authentication A (assigned to Index A) and failed authentication B (assigned to Index B).
And unfortunately it’s mandatory to assign an index to a Stream.
To my mind it’s really not handy to use Streams for 2 very different use cases which are routing logs into indexes and filtering logs for Alerts.
Do you agree ?
Or am I doing wrong ? Maybe the recommended way to filter logs for Alerts is to use a query and not a Stream ? (I’m doint this because in old releases we couldn’t use a query, Streams were the only way to filter logs for Alerts)
Yes, streams are the method for routing data from an input to an index. You can have multiple streams going to an index. Alerts and pipelines take actions based on message data (Fields, aggregations…) that is passing through the streams. You can point multiple alerts and/or pipelines to a stream. Pipelines can shift messages to another stream (and resulting index) or drop messages so that they don’t land in an index. From the way you have described it, it appears that you can adjust your alert to monitor both new streams… from there I am not sure what the issue is?
Thank you for trying to help me.
Yes I know all of this.
Pipelines can shift messages to another stream (and resulting index)
Yes but if the source Stream is used by an Alert, logs wouldn’t be in this Stream anymore when the Alert would check conditions (every X minutes), because logs would have already been moved to the destination Stream.
From the way you have described it, it appears that you can adjust your alert to monitor both new streams
I agree with you but it’s a pain to have to create 2 different Streams just because of the indexing issue.
It’s tipically these kinds of issue I want to show you.
Sometimes the “routing to indexes” use case is not compatible with the “alert” use case. Or it’s compatible but it makes things more complex.
If we had a dedicated way to route log into indexes it would be better and easier.
Maybe just allowing to create Streams without pointing to any index could do the job.
I guess I am still not seeing it. Couldn’t you just have the alert watch multiple streams for the specific message you want to alert on?
In my environment I have set up specific streams that receive messages that should be alerts, I have pipeline rules on the main streams that find what I want and shunt a copy to the appropriate alert stream/index. This way I have an retention specific index of only alerts and they simply alert on anything that comes in to their particular alert level. (L1, L2, L3 R3…)
