Create GeoIP geohash_grid field with Graylog Pipeline

Graylog Central (peer support)
1

I’m using a GELF input to get Apache2 logs into Graylog (after a complete reset). I’m stuck on how to create a geolocation field properly. I believe the correct way is to do this via a pipeline, but I’m pretty sure that I’m doing it incorrectly and I don’t know why.

I’ve created the proper lookup tables, which seems pretty straight forward. First the Data Adapter, then the Cache, then the actual table. Everything went fine, without issues, and here are my tables:

image
image1461×254 23.6 KB

Next, I created a stream to put the Apache2 messages in their own index (and remove them from the main index) based on a specific tag that I add to the message. This works as expected.

Next I went to Pipelines… I created a pipeline, and it appears to be receiving the messages:

image
image724×137 9.51 KB

I created the rule, and that also appears to be getting messages (also good, I believe LOL):

image
image1366×176 16.6 KB

Now… Here is the rule source:

image
image700×393 17.3 KB

Also of note is the order of my Message Processors… I’m not sure if this is correct, but I’ve seen guides recommend this order.

image
image709×225 14.8 KB

When I look at the messages, they don’t have any of the dst_ip_geo* fields. Any idea what I’m doing wrong?

2

I am an idiot. There is a simple, but key piece missing from the rule I adapted and that is within the to_string function call. I now have this:

image
image713×361 15.6 KB

The fields are created, AWESOME. But, the actual geolocation field is created as a “string” type and in order for me to use it in Grafana, I need to to be a “geohash” type… Is there a different value I should be pulling from the GeoIP Database?

3

Ok. So I have solved the issue, but I needed to manually update the base template for these logs. This was tricky LOL.

Is there any way to edit the default index template in Graylog directly before the index is created? I used Cerebro because I couldn’t figure out how to do it in Graylog.

Holy evolving topic batman…

4

You can create custom graylog-elastic mappings with rudimentary instructions here:

5

Excellent! I will give this a shot with my next set of indexes!

6

I while back I wrote up something about creating custom indexes and fixing historical types. It might help a bit since the Graylog docs are spartan:

1 Like
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.