Pipelines Configuration

[geox]

Hello,

I’ve followed step-by-step that article:

How to Set Up Graylog GeoIP Configuration | Graylog

There’re two things that confuses me:

1 . I created a pipeline, I connected Stage0 with a Stage Rule sucessfully.
However this message appears :

This pipeline is currently not connected to any streams. You have to connect a pipeline to at least one stream to make it process incoming messages. Note that this is not required if you intend to use this pipeline only for search result transformation using decorators.

Is it mandatory to create a stream ? The article not mentions something like that.
Check please the screenshot:

2. After following all these steps according to the article , how Im going to view the field: src_ip_geo_location ?
   Please note , that all the log information is reaching my Graylog setup through Sidecar/Beats from the server nodes.

[shoothub]

Check docs, to better understanding:
https://docs.graylog.org/en/4.0/pages/pipelines/stream_connections.html

[geox]

Thanks , but this article (Stream connections — Graylog 4.0.0 documentation) doesn’t really mention anything valuable, just some vague points that pipelines do not process any messages.

Any other ideas ?
Its rather unclear how the geolocation configuration is connected with pipelines and streams.

[shoothub]

For a pipeline to actually do any work it must first be connected to one or more streams.

So it’s mandatory to connect pipeline to stream. If you want to process all messages connect to default stream All messages.

https://docs.graylog.org/en/4.0/pages/pipelines/usage.html#connect-pipelines-to-streams

Geolocation uses lookup table which convert IP to geoip parameters like state, city and so on. This lookup table is used in pipeline rule, which uses function lookup() to get this paramaters and store them as custom fields. Pipeline rule runs in specific stage of pipeline. And pipeline is connected to stream to run.

Please read at least these sections to better understanding how it works:
https://docs.graylog.org/en/4.0/pages/pipelines.html
https://docs.graylog.org/en/4.0/pages/lookuptables.html
https://docs.graylog.org/en/4.0/pages/geolocation.html

[geox]

Thanks for the reply.
I’ve already done the following steps

1.Download the Geolocation Database
2.Configure Data Adapters
3.Configure Caches
4.Configure Lookup Tables
5.Configure Pipeline Rules

I have created a new Stream , named “Geolocation” with the following rule:

Then , I connected this Stream with the Pipeline:

How Im gonna view now the Geolocation IPs now ?
In the Fields, the following are visible:

thanks a lot!

[gsmith]

@geox
Hello,

Try using the search. Should look something like this below. If not you may have something missed configured.

[shoothub]

Hi @geox , for which field do you try to use geo ip location. In howto you have posted they used field src_ip with IP address that was used for geo location. I don’t see this field in your fields list. Please post you pipeline rule you used, and field which contains ip address.

[geox]

thanks for the reply!

The pipeline rule is:

rule “GeoIP lookup: src_ip”
when
has_field(“src_ip”)
then
let geo = lookup(“geoip”, to_string($message.src_ip));

set_field(“src_ip_geo_location”, geo[“coordinates”]);
set_field(“src_ip_geo_country”, geo[“country”].iso_code);
set_field(“src_ip_geo_city”, geo[“city”].names.en);
end

[shoothub]

You’ve used pipeline rule which uses field src_ip with ip address which convert to geoip fields. But I don’t see this field src_ip in your field list.

Which field in your message contains IP address you want to use as input for GeoIP? If field has different name as src_ip change pipeline rule and replace src_ip with your real field name which contains internet IP address.

[geox]

You were right , I made the following adjustments:

rule “GeoIP lookup: IP”
when
has_field(“IP”)
then
let geo = lookup(“geoip”, to_string($message.IP));

set_field(“IP_geo_location”, geo[“coordinates”]);
set_field(“IP_geo_country”, geo[“country”].iso_code);
set_field(“IP_geo_city”, geo[“city”].names.en);
end

Im not sure if the Stream that I have connected with the Pipeline is correct:

and the Stream Rule:

thanks a lot for your support!

[shoothub]

It seam ok if stream contains all messages for which you want to use geoip.

[geox]

I should have seen fields like:

IP_geo_location
IP_geo_country
IP_geo_city

but unfortunately nothing yet:

Is there something else that I should check?

[shoothub]

Check:

1. If your logs are correctly forwarded to stream
2. Check your processing order. Please move your Message Filter chain before Pipeline Processor in System Configuration - Message Processors Configuration
3. Try to debug pipeline rule using debug() function like this:
   debug(concat("IP: ", to_string($message.IP)));
   And then check logs:
   sudo tail -f /var/log/graylog-server/server.log

[geox]

thanks for the support so far!