Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. Don’t forget to select tags to help index your topic!
1. Describe your incident:
I have a UDP input setup that is pulling in syslog from a FW and using that input as well as a parsed field I have this syslog information going into a specific stream. (remove from all messages selected)
This stream above goes into it’s own index.
Now I am trying to create a new stream that is pulling information from messages that are being routed into the above stream but I can’t get them to show up when in the search page and selecting the new stream.
Sending this to a different index than above.
Do the streams only pull from the all messages queue or can pull messages from one stream into another?
2. Describe your environment:
OS Information:
Debian
Package Version:
4.2.9-1
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
I have been trying to figure out why it won’t pull these messages into my stream, when go into the original stream and then filter on the criteria in the new stream the messages are filtered. When I take those messages and match them against the new stream I get all green but for some reason when I select just that stream on the search page I don’t get any messages.
I just tried using the filter from stream #2 above and inversing on stream #1 so that it wouldn’t get pulled into the first stream and it still doesn’t show up.
4. How can the community help?
I have tried to access the documentation but can’t seem to find the answer as to whether the streams can only pull from all messages and not from another stream which removed the messages from “all messages”
Good to know this should be working, I ended up just saving them as searches and looking through the data that way but it would be nice to use streams so I could tie them to notifications if needed.
Here is my config as best I know how to post it
1 Message Filter Chain active
2 Pipeline Processor active
3 AWS Instance Name Lookup disabled
4 GeoIP Resolver active
Stream-A
application_name must match exactly filterlog
gl_source_input must match input PFsense UDP (Syslog UDP: 60abedc170314b73e1afb538)
Stream-B
tracker must match exactly 1650119255
Now here is the thing, any messages that match that tracker number will also match Stream-A criteria and I have gone back to storing both of these streams in the same index after learning more about indexes I really don’t want two copies of this message.
But for some reason whenever I try and load just the Stream-B on the search page I don’t get any messages even through when I do a manual query with the same tracker number I get messages with or without using Stream-A as a stream filter.
I included message processing at the top since it seems this could come into play as well but since I have Message Filter Chain first I am thinking it shouldn’t be messing this up, please correct me if I am wrong.
As for dup’s. noticed it says during the configurations it reply’s to All Message stream. Remove matches from ‘All messages’ stream, It does not reply to Stream-A.
I believe a pipeline would be you best option in sending the correct messages to the correct stream.
Using All Messages stream keep the others streams/indices that were created. Now just use a pipeline.
Example: You may have to fine tune this.
Rule "application Route to stream"
when
contains(to_string($message.application_name), "filterlog") && has_field("gl_source_input ")from_input([3A62730f30eeb96e7f96518123])
then
route_to_stream(id:"5d8acba383d72e04cba8888");
remove_from_stream(All Messages Stream _id)
end
Last,
Rule "tracker Route to stream"
when
has_field("tracker ") AND contains(to_string($message.tracker), "1650119255",true)
then
route_to_stream(id:"5d8acba383d72e04cba9999");
remove_from_stream(All Messages Stream _id)
end
I was able to get it working using the examples you provided and while it is more troublesome than just using the GUI to build stream I guess it will have to do.
I still was unable to get the streams working through basic stream rules whether using the same index or different ones.
Maybe it has to do with the pipelines I am using to parse certain information out of the original message? (even though I had message filter chain above pipeline processor)
Anyways I took the current 2 stage pipeline and added a third stage and placed your template in there with all of the tracker id’s defined routing them to the proper stream id and everything is working now.
