1. Describe your incident:
Hi Graylog community.
I have created some pipeline rules to process the GeoIP of the source country into its own field. For example, if the source IP address comes from the US, I want the rule to set a field named “US”. This way I can have some more granularity in terms of dashboard aggregation. However, several of the country pipeline rules I created are not processing, while a few of them are. They are all created with the same syntax minus the naming convention I’m using for the country
Here is the syntax of the rule I have for the United States:
rule "SrcCountryUnitedStates"
when
(has_field("Src_ip_geo_country") &&
to_string($message."Src_ip_geo_country") == "US"
)
then
set_field(
field : "UniSt",
value : "UniSt",
clean_field : false
);
end
Every country rule I have created makes the field and value the same name.
2. Describe your environment:
Running a single-node deployment of Graylog 5.2 on Ubuntu 22.04 LTS.
3. What steps have you already taken to try and solve the problem?
-
Tried recreating the rule and updating it on the pipeline
-
Changed the naming convention for the field the rule would set (i.e. instead of creating a field named US, create one named UniSt)
-
Rearranged the message processors Configuration
-
Rewrote the rule in the source code editor instead of the rule building interface that v5.2 has.
-
Using the Simulator to check for syntax errors and to see if my rule works: It does.
-
Checked for index errors in Overview: None (all of these rules were made in the last 24 hours.)
If I look at the particular stream I am wanting to run these rules against, I can see that the GeoIP processor is working and creates a field called “Src_ip_geo_country”
The pipeline does show that the stream messages are going through the rules, but don’t process for certain countries. Here is a screenshot of the rule processing for Germany, to show that some rules are working.
However, after having this pipline run overnight, it still hasn’t created a field for US or the Netherlands, and those are two of the highest hits I am getting in the stream.
4. How can the community help?
What other troubleshooting steps should I look at to see why the pipeline rules are not creating fields? Could this be an issue of reaching the maximum number of fields Graylog can store (if that is a thing)? I’m at a loss for ideas on why some of the rules are processing, but other’s are not. Down below is the order that I have my message processors, just so I’m inclunding that information as well.
