Split & index pipeline rules

[Bilal-SG]

Hi all!

I want to split a key, value pair which contains geolocation data as in “data_eventdata_DestinationIp_geolocation”:“Y,X” into the following “lat”, “lon” fields respectively as in:
“lat”:“Y”
“lon”:“X”
As I am using the Graylog Geolocation process to add geolocation information to received messages it turned out to me I couldn’t use extractors to perform this function because the Geolocation processor comes in a subsequent order with respect to the stream rule processor as you can see below

So it looks like I have to create a pipeline rule for this… Any idea about this please?

Note:
I am using graylog 5.0.8 on Ubuntu 22.04, and it stores data in OpenSearch 2.6

pipeline-rules

[ihe]

1. create a Dataadapter for your Maxmind-Database
2. create a cache for your geoip-lookups
3. create a lookuptable with the Dataadapter and the cache
4. create a rule like this:

rule "geolocation coords lookup"
when
  has_field("client_address") //the field you want to do the lookup for
then
  let result = lookup_value("geolite-country-lookup", $message.client_address);
  set_field(field:"client_geo_location", value:result);
end

where geolite-country-lookup is the name of the lookup table and client_geo_location the field with two values. Lat and Lon will be on one field, which you can use for geo-mapping.

[Bilal-SG]

Hello ihe,

Thanks for your reply. Actually, I have to make Lat and Lon on two separate fields for my use case to work out… so my question was how to create a pipeline rule to do this. As I understood from your suggestion it only made Lat and Lon to be stored in the ‘client_geo_location’ field but I wanted to split Lat and Lon as well.

Additionally, here is the output I could obtain without having to do the steps 1 → 3.

I got this by enabling Maxmind Geo IP only

And it already contained Lat and Lon on “data_eventdata_DestinationIp_geolocation” as you can see. so why would I need to do the steps 1 → 3 ?

And to be clear, I need to split Lat and Lon into two separate fields. I think the most proper way to do this is by using pipeline rules (correct me if I am wrong) because it didn’t work with me with extractors because of the message processing order, I guess.

Thank you!!

[ihe]

Hi @Bilal-SG ,
I quess you enabled the Geo-Location Processor, right? Then it will do it out of the box.
WIth the pipelines you will be able to fully control the enrichment and not leave it to the magic of the Geo-Location Processor.

Depending on the order of your processing you will be able to split this fields in two fields with pipelines. Is that field a string? you might have a look at the split-function and use the “,” as a delimiter.