Any examples of set_field usage with 6 parameters?

[S11 (Steve)]

I’m new to Graylog and trying to setup a pipeline rule.

The Pipeline “Rules quick reference” shows that set_field can accept 6 parameters:

field
value
prefix
suffix
message
default

I would like to use the “default” parameter to handle cases where a geoip field is missing/null. I am assuming I need to supply all 6 set_field parameters in order to supply a default value of empty string.

For example, I would like to have a default value in case the geoip subdivisions array is empty:

 set_field("dst_ip_geo_state", geo["subdivisions"].[0].iso_code, "", "", $message, "");

I’ve tried several permutations of set_field with 6 parameters, but I can’t figure out the syntax. I’m getting validation errors like “Expected type Message for argument message”.

I can’t find any examples of using set_field() with all 6 parameters. Can anyone provide an actual example of set_field showing how to supply the Message object as the 5th parameter?

Environment:

Graylog v4.3.5+32fa802

[gsmith]

Hello,
By chance have you searched the forum for geoip? There are a lot of examples. I would show you some examples but I’m working on my car rn:

[S11 (Steve)]

My question is not about geoip. My question is about using set_field() with 6 parameters.

And yes, I’ve searched the forums and documentation, and haven’t found a single actual example showing set_field with 6 parameters. Every example I’ve found only uses 2 parameters, and the documentation doesn’t have an example of 6 parameter usage.

[tmacgbay]

Without seeing any of the surrounding process and code, you could do something like below… This uses the “default” parameter of to_string()

set_field("dst_ip_geo_state", to_string(geo["country"].iso_code,"no_iso_code"));

NOTE: Removed and updated stuff in that line as well…

[S11 (Steve)]

Thank you! That is a clever workaround that looks simpler and cleaner! I’ll give that a try.

[S11 (Steve)]

Unfortunately, to_string() with a default didn’t work. It throws this error when the geoip object is not available:

Error evaluating action for rule <GeoIP lookup> - In call to function 'to_string' at 13:34 an exception was thrown: Index 0 out of bounds for length 0

I tried concat(geo[“subdivisions”].[0].iso_code, “”) to see if that would convert the geoip value to a string, but the parser says that the iso_code is an object, and concat only accepts string.

Based on the github issues I’ve read about dealing with the geoip “subdivisions” value (US state), my understanding is that set_field() with the default parameter was created specifically for this issue , making it possible to deal with missing geoip array values.

Unfortunately, I can’t find a single example of how to call set_field with 6 params, or any examples of the recommended syntax for including the message object as a parameter.

[tmacgbay]

Are you using the free version of Maxmind? That wouldn’t have subdivisions in it. Also, take the indexing out like I did in my example.

If you do have subdivisions (makes me think of RUSH) then can you post and example form the database so we can help reference it?

[S11 (Steve)]

I am using the free version of Maxmind, but it does appear to include subdivisions–US states.

This is the Lookup Table result for 216.239.38.120 (google.com):

"subdivisions": [
      {
        "confidence": null,
        "geoname_id": 5332921,
        "iso_code": "CA",
        "names": {
          "de": "Kalifornien",
          "ru": "Калифорния",
          "pt-BR": "Califórnia",
          "ja": "カリフォルニア州",
          "en": "California",
          "fr": "Californie",
          "zh-CN": "加州",
          "es": "California"
        }
      }
    ]

Because subdivisions always returns an array, the index [0] is required to read a value.

If I do not supply the [0] index, like the example below, the field is not created.

set_field("dst_ip_geo_state", geo["subdivisions"].iso_code);

Hence my issue–when no subdivision is returned from Maxmind, it’s an empty array, so if I reference an array element, it will throw an exception.

Error evaluating action for rule <GeoIP lookup: dstIP> - In call to function 'set_field' at 13:4 an exception was thrown: Index 0 out of bounds for length 0

Here is the GitHub issue from 2019 discussing this:

Pipeline rule with geo lookup and possible empty array

That issue references this pull request from 21 days ago where they apparently added the additional parameter for a default value to handle this scenario:

Add default param to set_field function

[tmacgbay]

I stand corrected on nearly all counts - I was looking at Maxmind Country rather than maxmind city that I already have set up from when I was experimenting… and you are correct about the indexing… I did not see that. I am still correct about Rush… AND that my to_string idea works… In my lab:

rule "RUSH-TEST"
when
  true 
then

 let geo = lookup("Geolisting-City", $message.IP);
// debug(concat("IP-stuff: ", to_string(geo))); 
 
 set_field("dst_ip_geo_state", to_string(geo["subdivisions"].[0].name.fr,"no_iso_code"));
 
end

for IP 216.239.38.120 this resulted in

for IP 216.249.38.120 this resulted in (No French name)

[S11 (Steve)]

Thank you for the update and the help.

I think I might have discovered a strange issue. I did confirm that that your technique of to_string() does work when the subdivisions [0] value is not present–but only sometimes! Which is weird.

In my initial test of to_string(), I saw the errors and assumed that to_string() on subdivisions was failing–but it looks like city and country were also failing due to missing values. So I added to_string around city and country, but I was still seeing errors.

This internal IP address has no geoip values, and the to_string() handles that well:

But I am still seeing errors on other records.

After some digging, I found this example. It is a record for 192.175.48.6 (blackhole-1.iana.org):

In this case, to_string() works on the missing city value, but the script then seems to throw an error on subdivision. The error references line 9, position 34 of my script, which is the subdivision line. I don’t understand why to_string() works on some records when city exist, but subdivision [0] doesn’t exist, then fails on records where city does not exist.

I found that IP 142.250.68.42 triggers the same error. And that IP also has no city value in the geoIP data.

I’m stumped at the moment–it doesn’t make any sense. I’ve stared at this too long, so I’m going to step away from it for a while.

rule "GeoIP lookup: dstIP"

when
  has_field("dstIP")
then
    let geo = lookup("geoip", to_string($message.dstIP));
    
    set_field("dst_ip_geo_city", to_string(geo["city"].names.en, "no_value"));
    set_field("dst_ip_geo_state", to_string(geo["subdivisions"].[0].iso_code, "no_value"));
    set_field("dst_ip_geo_country", to_string(geo["country"].iso_code, "no_value"));
end

[tmacgbay]

but… you are so close!!

In your example of looking up 192.168.25.7 - that is a non-routable/private address and would not be in the database so no_value… not surprised at all. You could filter for that in your rule:

rule "GeoIP lookup: dstIP"

when
  has_field("dstIP")  &&
  NOT in_private_net(to_string($message.dstIP))
then
...

For IP 142.250.68.42 subdivisions is completely empty so you would have to error check for empty/nonexistant subdivisions to avoid getting the error. Finding that goes beyond the scope of the question… which we have already gone beyond…