Need help with GeoIP setup please

[Filisimus]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I would like to implement https://www.graylog.org/post/how-to-set-up-graylog-geoip-configuration/ but I dont have dest_ip or src_ip available - how can I extract them from the message, please?

2. Describe your environment:

• OS Information:
  Ubuntu 20.04
  Graylog 4.3.11
  MongoDB 4.0.28

3. What steps have you already taken to try and solve the problem?

So the setup is that we have Suricata listening to the network traffic and if anything comes up as susipicous it sends a message to the Graylog server as Beats input. There we have an extractor which splits the message into key / value pairs and it’s being processed later on.

Now I tried to implement https://www.graylog.org/post/how-to-set-up-graylog-geoip-configuration/ and it didnt work. Further investigations (playing with the rules) I figured out that I dont have dest_ip and src_ip available when the rule kicks in even though both are available in the message… so I guess the rules are processed before the extractor.

Here you can see a message and that the key / value pairs of the message are being added to the message by the extractor:

But if I use a simple rule like this the rule isnt triggered:

rule "Test"
when
has_field("src_ip")
then
set_field("Test","Test123");
end

4. How can the community help?

So I thought that it should be possible to extract the key_value pairs from the message and add them as additional fields (would also save the extractor). I used the key_value function together with set_fields but that doesnt seem to have any impact:

rule "Test"
when
has_field("message")
then
set_fields(key_value(to_string($message.message)),",",":");
end

Or maybe my thought process was wrong because I thought I would be able to access the fields in the next stage of my pipeline?

rule "Test"
when
has_field("message")
then
set_fields(key_value(to_string($message.message)),",",":");
end

when
has_field("src_ip")
then
set_field("Test","Test123");
end

What am I doing wrong here, please?

Cheers,
Fili

[tmacgbay]

Check out your System/Configurations/Message Processors Configuration, you want the have the Message Filter Chain before the Pipeline Processor - that’s usually why rules aren’t working. I would also consider cleaning up the timestamp issue that is showing up as the gl2-processing-error because that will keep things from being processed properly as well. I would guess that your key/value is picking up timestamp in the message and trying to overwrite the Graylog message timestamp (its unclear but that’s what it looks like)

[gsmith]

hey @Filisimus

Adding on to @tmacgbay suggestion.

• Lookup Table Returning Null Value

[Filisimus]

Hey @gsmith and @tmacgbay

thanks for your replies.

I thought about it and from my understanding the “src_ip” field got to be already there without splitting the key_value pairs because in the pipeline I can choose the streams to use and when configuring the rules for the streams I have the field “src_ip” available.

I changed the order for the message processors as suggested but unfortunately that didnt help. The gl2 processing error already occurs in the original message after extraction before the key/value function kicks in and I dont see any possiblility to transform the original timestamp in the extractor. Changing the timestamp before the message gets submitted to Graylog is not possible.

For the moment I think we should focus on the question why the src_ip field is not available in the pipeline rules?

Thanks,
Fili

[Filisimus]

Hi guys,

turns out I’m stupid and had the wrong Stream selected… I’m very sorry for this.

So this works… kinda now:

rule "GeoIP lookup"
when
  true
then
set_field("test_geo_src_ip",$message.src_ip);
set_field("test_geo_dest_ip",$message.dest_ip);

let geo = lookup("geoip", to_string($message.src_ip));
set_field("src_ip_geo_location", geo["coordinates"]);
set_field("src_ip_geo_country", geo["country"].iso_code);
set_field("src_ip_geo_city", geo["city"].names.en);

let geo2 = lookup("geoip", to_string($message.dest_ip));
set_field("dest_ip_geo_location", geo2["coordinates"]);
set_field("dest_ip_geo_country", geo2["country"].iso_code);
set_field("dest_ip_geo_city", geo2["city"].names.en);
end

The fields “test_geo_dest_ip” and “test_geo_src_ip” are being set and populated with the correct IPs. But the fields “src_ip_geo_" and "dest_ip_geo_” dont appear. There are no errors in the server.log file and the lookup cache for geoip has 170 entries and 99.74% hit rate, so I have no idea what could be the problem here. I also went to “Systems” → “Configurations” and put the GeoIP Resolver at top, middle and bottom of the list with no effect.

Anyone got any ideas, please?

Cheers,
Fili

[Filisimus]

Nevermind, it works now. Probably I was too impatient when I set the “Geo IP Resolver” Processor at the bottom before and didnt get immediately any results.

Thanks all!

Fili

[tmacgbay]

Thanks for posting the code for future searchers! Glad it worked out!!