Pipeline not processing correct rules

1. Describe your incident:
Hi Graylog community.
I have created some pipeline rules to process the GeoIP of the source country into its own field. For example, if the source IP address comes from the US, I want the rule to set a field named “US”. This way I can have some more granularity in terms of dashboard aggregation. However, several of the country pipeline rules I created are not processing, while a few of them are. They are all created with the same syntax minus the naming convention I’m using for the country

Here is the syntax of the rule I have for the United States:

rule "SrcCountryUnitedStates"
when
  (has_field("Src_ip_geo_country") &&
   to_string($message."Src_ip_geo_country") == "US"
   )
then
  set_field(
    field : "UniSt",
    value : "UniSt",
    clean_field : false
  );
end

Every country rule I have created makes the field and value the same name.

2. Describe your environment:
Running a single-node deployment of Graylog 5.2 on Ubuntu 22.04 LTS.

3. What steps have you already taken to try and solve the problem?

  • Tried recreating the rule and updating it on the pipeline

  • Changed the naming convention for the field the rule would set (i.e. instead of creating a field named US, create one named UniSt)

  • Rearranged the message processors Configuration

  • Rewrote the rule in the source code editor instead of the rule building interface that v5.2 has.

  • Using the Simulator to check for syntax errors and to see if my rule works: It does.

  • Checked for index errors in Overview: None (all of these rules were made in the last 24 hours.)

no-index-errors

If I look at the particular stream I am wanting to run these rules against, I can see that the GeoIP processor is working and creates a field called “Src_ip_geo_country”
srcipgeocountry

The pipeline does show that the stream messages are going through the rules, but don’t process for certain countries. Here is a screenshot of the rule processing for Germany, to show that some rules are working.

However, after having this pipline run overnight, it still hasn’t created a field for US or the Netherlands, and those are two of the highest hits I am getting in the stream.

4. How can the community help?

What other troubleshooting steps should I look at to see why the pipeline rules are not creating fields? Could this be an issue of reaching the maximum number of fields Graylog can store (if that is a thing)? I’m at a loss for ideas on why some of the rules are processing, but other’s are not. Down below is the order that I have my message processors, just so I’m inclunding that information as well.

MessageProcessorOrder

I’m fairly certain I found a fix for this issue. I had two pipelines for the same input, one of them being the country sorting pipeline, and the other being another sorting pipeline. I think the two were causing a conflict in some sort of way, because when I combined all of the rules into one pipeline, everything started working just fine. I now have fields for the countries that were not working prior to my original post.

1 Like

Glad you already resolved your issue.

For the future, this is a good discussion of multiple pipelines / stages / rules:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.