And the screenshot of me testing the stream rules. It clearly says that " This message would be routed to this stream." Yet, the message is ending up in default stream.
We do use pipeline rules, but none of them apply to these messages. Also, pipeline are processed after streams the way we have it setup:
|1|GeoIP Resolver
|2|Message Filter Chain
|3|Pipeline Processor
|4|AWS Instance Name Lookup
But, in the meantime I’ve found the root of the problem. Not sure if this is intended behavior or possibly a bug.
It seems if the filed doesn’t exists in the massage inverted match will not work.
In our case, this was the problem:
“Field program must not contain httpd”
Any messages that had field “program” were sorted to a proper stream (no matter the value). But any massage that did not even have the field “program” went to “All Messages” stream.
The workaround that seems to work, weirdly enough is using regex.
“Field program must not match regular expression .httpd.”
This behavior seems counter intuitive to me. If the field doesn’t exist, inverted match should still function as expected. Especially because testing the stream shows the rules match.