Pipeline for creating a GeoIP dashboard

Hello to all,

I would like to create a worldmap dashboard to visualize where the user connections come from on our stormshield firewall

I followed this procedure : How to Set Up Graylog GeoIP Configuration | Graylog
But I’m not really comfortable with pipelines despite the documentation I have trouble to understand and make it work

Here is the technical problem:
image

I would like to extract the ip address from the src=x.x.x.x message field to put it in a new field to create my dashboard on this field

already is it a pipeline rule, an extrator that I need?
can someone explain the profound difference? i don’t understand

This is my rules pipeline for now… i found it on this forum but i don’t know how use it…

rule “function ExtractIPv4Pairs”
when
regex(“.^0-9a-zA-Z[^0-9a-zA-Z].^0-9a-zA-Z[^0-9a-zA-Z].“,to_string($message.message)).matches
then
let pair = regex(”.
^0-9a-zA-Z[^0-9a-zA-Z].^0-9a-zA-Z[^0-9a-zA-Z].”,to_string($message.message),[“src”,“dst”]);
set_field(“IP Source”,to_ip(pair.src));
end

Thx all …

Hi @tco,

From what I see, you have at least two IPs in your messages.

You are using regex, but you can also use GROK patterns, there are preconfigured GROK for IPv4 & IPv6.

I’ve done a similar thing that you’re trying to achieve, and here is the element I can provide you, hoping that it will help.

In my case, I used the %{IP} GROK extractor on the input of the message to create two seperate fields :

(something here) %{IP:Source_IP} (something here) %{IP:Destination_IP} (something here)

Then, I use the pipeline provided in the Graylog documentation on the “Source_IP” field in order to get the map working.

Maybe if you post the entire log (with the addresses anonymized) in an editable format (like using the block code here) I might be able to help you more ! :slight_smile:

thank you for your quick answer

I have to create an extractor using the grok pattern ip?
how can i create my extractor, on the input that receives the logs from my firewall?

do I have to import an extractor?

sorry for these questions, i have never created one and i am not very comfortable with the subject

You’re welcome @tco.

Well, that’s quite easy to create the extractor. You go on the message, and on the field you wanna extract you click on the little arrow pointing down, create extractor. Then, select “Grok pattern” for the type and submit.

On the page that opens, you can write your extractor and try it against the message you’ve selected to see if it works or not ! :slight_smile:

Again, feel free to ask if you have difficulties, but again, it’lle be hard to help you without the entire message that you’re working on ^^

Hello @tco

if you conformable with making a regex extractor, I have done this all ready with this…

srcip=+?((?:\d+\.){3}\d+).+

That matches my GeoIP Pipeline

rule "GeoIP lookup: srcip"

when

  has_field("srcip")

then

let geo = lookup("geoip", to_string($message.srcip));

set_field("src_ip_geo_location", geo["coordinates"]);

set_field("src_ip_geo_country", geo["country"].iso_code);

set_field("src_ip_geo_city", geo["city"].names.en);

end

HowTo here

Thanks to both of you for your answer,

I didn’t manage to use the grok pattern… but the regular expression worked. I have a new IPSource field that appeared, thanks

I created the pipeline rule by adapting it:

rule “GeoIP lookup: IPSource”

when

has_field(“IPSource”)

then

let geo = lookup(“geoip”, to_string($message.IPSource));

set_field(“src_ip_geo_location”, geo[“coordinates”]);

set_field(“src_ip_geo_country”, geo[“country”].iso_code);

set_field(“src_ip_geo_city”, geo[“city”].names.en);

end

Is there any way to verify that this works?
Also, I’m French, does changing the “names.en” to “names.fr” in my rule work?

I also have my lookup_tables with my GeoLite2-City.mmdb database to set the location of source ip addresses

1 Like

Hi @tco,

Great if it works ! :stuck_out_tongue:

I think that there might be a cleaner solution than just test it against a real stream, but that’s how I personnaly proceed when I write a new pipeline.

French too, the pipeline I use also have the “names.en”… I didn’t try changing it by “names.fr” because it works flawlessly for me.

Sorry I can’t get the geoip feature to work
To summarize: I created my lookup table and it works when I do the geolocation tests:

I created my pipeline rule : (the name of my lookup_table is “geoip”)

//
rule "geoip"

when

  has_field("IPSource")

then

let geo = lookup("geoip", to_string($message.IPSource));
set_field("ipsrc_location", geo["coordinates"]);
set_field("ipsrc_country_name", geo["country"].names.en);

end
//

I created my pipeline then

The problem :
When I generate a log on my firewall, I don’t have the “ipsrc_location” and “ipsrc_country_name” fields that my pipeline should create, I have the IPSource field that appears

Log :

Thank’s all :slight_smile:

You can use the debug() function in your pipeline rule to see what is going on (see below for example) Also, check to see if your Message Filter Chain is set BEFORE your pipeline processor (See further below) in system/configurations…

rule "geoip"

when

  has_field("IPSource")

then
    // tail -f /var/log/graylog-server/server.log to see results of debug message
    debug(concat("=========== IPSource is: ", to_string($message.IPSource)));
    
    let geo = lookup("geoip", to_string($message.IPSource));
    set_field("ipsrc_location", geo["coordinates"]);
    set_field("ipsrc_country_name", geo["country"].names.en);

end

image

1 Like

THANK YOU IT WORKS!
I understand the pipeline system better now… thanks a lot

Last thing, I would like to make a data table with the identification information
I would have to create a “user” extraction and put it in a new field
I understand the principle of extractor now but I don’t understand JSON languages, regex, etc…
Is there an easier way to extract the user field into a new field?

Hello,

REGEX extractor should work.

user="(.*?)\"

thank you very much!!!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.