Is it possible to create hidden fields in a message using a Pipeline rule? I have a series of rules in which I’d like some rules to pass a value to rules that follow it in later stages, but there’s no sense in the value being visible (and thus cluttering the view). I know there are some hidden fields already, like gl2_remote_ip, so I’m hoping Pipelines can do the same thing…
those hidden fields are hardcoded and you can’t add dynamic new to this array.
But why not delete all fields that you do not want to keep in a last stage of your processing pipeline?
I’m trying to parse firewall logs from a pfSense firewall, and they use a CSV format with a dynamic set of fields depending on the type of packet. So the messages won’t all follow the same path through the pipeline rules (some may not even make it through all of the stages), which makes it difficult to know when to remove the field.
See https://www.netgate.com/docs/pfsense/monitoring/filter-log-format-for-pfsense-2-2.html.
I’m hoping to include a field that indicates the last field position that was extracted so the subsequent rule knows where to pick up the extraction process. For example, to create the pipeline rule that extracts the TCP fields, I would either need to create a rule for IPv4 TCP and a rule for IPv6 TCP, or if I just had the preceding IPv4 and IPv6 rules set a field indicating the field number last used, I could just create one IPv4/6 TCP rule that increments the step for each field.
If anyone has any better ideas, that would be appreciated! When I implemented this idea of passing on the step in the extra field, I found that the Pipeline rule editor forced me to cast the variable to a long each time I needed to reference it (even though the variable holds a long-time value). Considering the number of time the variable is referenced, I think it was too much for our server to cast it each time, because the web interface takes literally hours to load, and the CPU has been maxed out since yesterday afternoon. So now I’m attempting to remove the pipeline from the respective stream… 
I’m just dealing with the extra field. In order to keep it at the bottom of the view (out of the way of the rest of the fields), I just named it “z_”.
For the record, casting the variable was not the issue that caused issues with the web interface. Poorly-written RegEx’s ended up being the culprit. Casting in a pipeline rule doesn’t seem to have much affect on the CPU in general.
Not sure why I didn’t think of this earlier… Since all the stages are a “match at least one rule,” I created an always-true rule, included it in each stage that may not match all messages, and then created a final stage that has a rule to remove the extra field, similar to @jan’s idea.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.