Pipeline on new stream not working


I’ve been at this a while now and I can’t get it to work.

I am forwarding pfsense filter logs to Graylog into a udp syslog input.
I created a stream with a rule that must match: ‘^filterlog[.]:\s+(.)$’. This same regex is used in many pfsense extractors and work fine. I can see only those matches are sent into this new stream.

I’ve setup extractors with a lookup that translate port numbers to service names; these work fine and the respective fields appear in the stream.

Next I followed the guide from your blog to setup geoip database and lookup with cache and adapter. I can test it with an IP address and it works.

I then create a new pipeline, I connect it to the pfsense_filter_logs stream.

At this point I would expect to see the throughput, but its always 0.

I create a new rule with the following code:

rule "GeoIP lookup field src_ip"

when has_field("src_ip")
  let geo = lookup("geoip", to_string($message.src_ip));
  set_field("src_ip_geo_location", geo["coordinates"]);
  set_field("src_ip_geo_country", geo["country"].iso_code);
  set_field("src_ip_geo_city", geo["city"].names.en);

Note: my extractors do create the field “src_ip”, I can see them when I click one of the filterlog lines and it opens up with all the fields.

With that rule in place, I edit Stage 0 on the pipeline I created earlier, select ‘all rules must match’ and I select the rule I just created.

Errors and total remain at 0 and I have no idea where I can see why its not working. I checked the messages in the stream and the new fields aren’t created.

If anyone has an idea or two, please let me know :).

Thanks in advance.

1 Like

Check, if you have right Message Processors Configuration order in your graylog. Your Pipeline processor should be after Message Filter Chain in System/Configurations, because you try to use extracted fields in your pipeline rule.



Thank you! That was the issue, pipelines are now processing properly :+1:t3:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.