1.Describe your incident:
I need to enable the Data Lake and Illuminate features at the same time. I’m unsure whether there are orders of operations needed to enable both features together. I’d like to know the recommended steps and any known constraints or conflicts when enabling them concurrently.
2.Describe your environment:
OS Information:Windows 11
Graylogversion:7.0.3 (security license)
3.What steps have you already taken to try and solve the problem?
Checked documentation for enabling Data Lake and Illuminate separately.
Verified my account has admin-level permissions for feature management.
Questions:
1)Taking Windows logs as an example, I want to route a subset of the logs to the Data Lake so they do not consume license usage, while keeping the rest stored in the default index used by the Illuminate features.
2)I attempted to use a pipeline rule to route a subset of logs from the Default Stream into a different sub-stream. However, the pipeline rule does not appear to work as expected. I suspect the issue is related to using the EventID field as the condition in the pipeline rule.
You could put them into different streams, but you actually don’t have to. If you go into the data routing section of the stream settings, you can just set rules VERY similar to a pipeline rule, that will exclude certain messages from one side or the other, so you can exclude certain things from hot and others from the datalake if you wanted.
You could take this a step further and have custom pipeline rules that set a field of what should happen at the routing, and then use that field for the exculsion rules.
The benefit of keeping them in the same stream is that when you restore them they are then restored into the original stream rather than into another stream.
I tried this approach as well, but I wasn’t able to route the logs exclusively to the Data Lake without them being stored in the index. In other words, I’m not sure how to correctly specify the field names in the filter rules, as I don’t know what the field is called before the logs are processed by Illuminate. Could you provide a detailed example of how to configure this correctly?
Routing and custom pipeline rules happen after illuminate (unless you have changed the default order) messages being sent to the datalake are fully processed before being sent there.
