Pipeline does not work (caching or something?)

Graylog Central (peer support)
1

rule “GeoIP remote_addr”
when
has_field(“remote_addr”)
then
let geo = lookup(“geoip”, to_string($message.remote_addr));

set_field(“SourceIP_geolocation”, geo[“coordinates”]);
set_field(“SourceIP_geo_country”, geo[“country”].iso_code);
set_field(“SourceIP_geo_city”, geo[“city”].names.en);

end

(Field name is copied, as some typo there would be my first guess.)

With google I ran in to this:

Which seems to be the precise problem I have. Except that it works somewhere else and the order is already correct. Also it feels always arbitrary. Is there some caching involved here? Is it in some sense not nearly as live as it seems?

I am fairly certain that i had this exact situation before and it just worked a few hours later.
As in: I gave up for the day. And the next day it just worked. Like did not touch any config, it just started adding the geolocations at some point.

Are there any ways to test if a pipeline/rule works? The test rule feature seems to be either very limited or I do not understand it at all. For example to my understanding it can’t handle any custom fields so the "has_field(“remote_addr”) " can never be checked.

2

Looking though the entries where geolocation works I get the suspicion that there is some other way to add this data in Graylog and that I have used that there, because there is no pipeline for those. Not just that, no rules either.

So my guess was that some update broke it. But I am currently looking at an example that is less then 10 minutes old.

I got the pipeline to add other fields but the geolocation still does not work in this example. It seem to happily work on other messages via some older setup I can’t find.

3

What version of Graylog are you using?

Is your question/issue that the pipeline rule isn’t working or doing anything?

Graylog does have a built in geo-location plugin as well (i think added in 4.3), you can see its configuration via System / Configuration. When enabled, it will automatically add geo location fields for any field that contains an IP Address.

Regarding testing pipeline rules, I’m attaching a document i drafted that goes through how to use the pipeline simulator:
SE-Testing and Debugging Processing Pipelines-070723-163205.pdf (428.3 KB)

Hope that helps.

1 Like
4

That seems to be the case. In that case I think currently no (geolocation related) pipeline works. But since the Plugin does and Iknow that know that is not a problem. Thanks.

I tested pipeline functioning in general by adding “true” as condition and adding a datetime field. General Processing works.
So if I understand this correctky the Plugin does Geolocation data for a certain list of field names, correct?

Documentation the Plugin points me to:
https://go2docs.graylog.org/5-0/making_sense_of_your_log_data/geolocation.html

That would make sense. That means I got lucky with the naming in the past and did not know about the plugin.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.