1. Describe your incident:
Hi Graylog community.
I have created some pipeline rules to process the GeoIP of the source country into its own field. For example, if the source IP address comes from the US, I want the rule to set a field named “US”. This way I can have some more granularity in terms of dashboard aggregation. However, several of the country pipeline rules I created are not processing, while a few of them are. They are all created with the same syntax minus the naming convention I’m using for the country
Here is the syntax of the rule I have for the United States:
rule "SrcCountryUnitedStates" when (has_field("Src_ip_geo_country") && to_string($message."Src_ip_geo_country") == "US" ) then set_field( field : "UniSt", value : "UniSt", clean_field : false ); end
Every country rule I have created makes the field and value the same name.
2. Describe your environment:
Running a single-node deployment of Graylog 5.2 on Ubuntu 22.04 LTS.
3. What steps have you already taken to try and solve the problem?
Tried recreating the rule and updating it on the pipeline
Changed the naming convention for the field the rule would set (i.e. instead of creating a field named US, create one named UniSt)
Rearranged the message processors Configuration
Rewrote the rule in the source code editor instead of the rule building interface that v5.2 has.
Using the Simulator to check for syntax errors and to see if my rule works: It does.
Checked for index errors in Overview: None (all of these rules were made in the last 24 hours.)
If I look at the particular stream I am wanting to run these rules against, I can see that the GeoIP processor is working and creates a field called “Src_ip_geo_country”
The pipeline does show that the stream messages are going through the rules, but don’t process for certain countries. Here is a screenshot of the rule processing for Germany, to show that some rules are working.
However, after having this pipline run overnight, it still hasn’t created a field for US or the Netherlands, and those are two of the highest hits I am getting in the stream.
4. How can the community help?
What other troubleshooting steps should I look at to see why the pipeline rules are not creating fields? Could this be an issue of reaching the maximum number of fields Graylog can store (if that is a thing)? I’m at a loss for ideas on why some of the rules are processing, but other’s are not. Down below is the order that I have my message processors, just so I’m inclunding that information as well.