Whois plugin looking up RFC1918 addresses


(Leigh) #1

I have the Whois plugin enabled using the standard config methods as per the (github/blog) documentation. In the server logs i continually see lookups for RFC1918 addresses. I thought this wasn’t supposed to happen? The queries seem to be being sent to ARIN, and obviously adding to my throttle limit, which is why i’d like to stop this happening.

ERROR [WhoisIpLookup] Could not lookup WHOIS information for [10.70.111.37] at [ARIN].
ERROR [WhoisIpLookup] Could not lookup WHOIS information for [10.5.1.90] at [ARIN].
ERROR [WhoisIpLookup] Could not lookup WHOIS information for [10.5.3.169] at [ARIN].
ERROR [WhoisIpLookup] Could not lookup WHOIS information for [10.70.109.119] at [ARIN].

I’m using the 2.5.0+ed06ce7, codename Trippy Trampoline version of Graylog.

Any ideas where I should be looking?

Many thanks


#2

Please share information to help.
How do you use the whois plugin?


(Leigh) #3

That’s kind of where I’m a bit confused and probably why I’m experiencing this. Once the plugin moved to being a default part of Graylog the documentation on Github/Graylog Blog appears to be confusing.

At the moment (I’m not sure if its the correct way) I’m using whois lookups on the Threat_Intel pipeline rules. Using the IP field doesn’t seem to capture all of the IP’s from my firewall logs, so I’ve setup additional pipeline rules for dst_addr and src_addr fields. This is my pipeline rule with another setup for src_addr:

rule “dst_addr IP Address WHOIS Lookup”
when
has_field(“dst_addr”)
then
let dst_addr_whois = whois_lookup_ip(to_string($message.dst_addr), “dst_addr”);
set_field(“dst_addr_whois”);
end

I also have extractors setup, because initially this is where I thought I should be doing the lookups, but currently they don’t seem to get any hits. I’ve left them configured as it doesnt seem to be causing any performance hits. Those extractors simply do a lookup on the Whois table for IP/src_addr/dst_addr fields.

I’m not sure if this is needed but the current order of my message processors is:
Message Filter Chain
Pipeline Processor
GeoIP Resolver

Do you need any more information?

Thanks in advance.


#4

I have never used the GL’s whois module, but I saw your post doesn’t contain so much information.
I can suggest, to find where do you call the whois with private ip. (simple disable the pontint what you mentioned one-by-one).
At pipelines, you can add more conditions. And you can use pipelines insted of extractors.

The message processor order is good.

I hope someone else can help about informations.


(Leigh) #5

Thanks macko003. I cant actually disable the pipline rules, if I do then I lose all whois lookups. According to the plugin documentation RFC1918 addresses are excluded from being looked up, but from the logs i’m seeing they actually are.

If its not a config error then is it maybe a bug?

Oh and if it is a condition I need to add why isn’t this mentioned in the setup documentation?..Let me guess, it is in the Enterprise version?


(Jan Doberstein) #6

If its not a config error then is it maybe a bug?

If you believe that this is a bug, please open a bug issue.

…Let me guess, it is in the Enterprise version?

No - Graylog will not have half of a feature in the OSS Version while you need to have Enterprise to use this complete. In addition, the Enterprise documentation is included in the OSS Documentation - just to have it written once.

What @macko003 pointed out that you could use in_private_net function inside of a processing rule to verify that you do not run the lookup on a private IP. You could also exclude well-known IPs with the cidr_match function or with a lookup table.


(Leigh) #7

Thanks Jan, I don’t believe it is a bug (yet) its just clear documentation on implementing it isn’t easily found. As you can see above I already have pipeline rules and extractors setup. But which one is best? (I know the answer to that is probably dependant on what I’m logging) But still, I can’t find anything to point me in the right direction. Especially when the documentation states RFC1918 addresses are excluded https://github.com/Graylog2/graylog-plugin-threatintel (at the bottom of the page).

So is the manual lookup (that the URL above mentions) what I have configured above?


(Jan Doberstein) #8

I don’t believe it is a bug (yet) its just clear documentation on implementing it isn’t easily found.

I doubt that - if is written down but not working is either wrong documented or not working as it should.

I would always go with the processing pipelines and give you this slidedeck ( https://github.com/jalogisch/OpenSourceDay2018/ ) to learn more on them.


(Leigh) #9

Thanks Jan, could you point me in the direction of any documentation on using the in_private_net and cidr_match functions in the whois plugin? I can search myself but I just want the defacto suggestions from yourself.


(Jan Doberstein) #10

you have the online help in the editor and the official docs: http://docs.graylog.org/en/2.5/pages/pipelines/functions.html#cidr-match


(Leigh) #11

Thanks again Jan. I do find the online help in the editor not very straight forward, but that’s probably because I don’t fully understand pipeline processing yet. I promise to go through the slidedeck in detail!


(Leigh) #12

Ok, so here’s what I’ve done. But for the sake of this thread let me just explain a few things.

I have several different fields containing an IP address that I need the Threat Intel plugin to check, excluding internal RFC1918 addresses. Those fields come from a range of different model Juniper firewalls, their log formats are quite different and I have a number of Extractors setup to use src_addr & dst_addr fields, the other fields below are automatically extracted by Graylog (which is why I’ve ended up with so many) but I don’t want to stop Graylog doing anything its supposed to (will save that for later). The different fields are:

source-address
destination-address
src_addr
dst_addr
IP
nat-source-address
nat-destination-address

For reference, my message processors are in the following order:
|#|Processor|Status|
|1|Message Filter Chain|active|
|2|Pipeline Processor|active|
|3|GeoIP Resolver|active|

My Extractors run within the Message Filter processor, and my Pipeline or course runs in the Pipeline processor.

I have setup a new Threat_Intel pipeline with the following rule, for each of the fields above, at each stage.

Stage 0 (gather the intel and set threat fields true or false)

rule "Threat Intelligence Lookups: dst_addr"
when
has_field(“dst_addr”) && ! in_private_net(to_string($message.“dst_addr”))
then
set_fields(threat_intel_lookup_ip(to_string($message.dst_addr), “dst_addr”));
end

and one specifically for OTX lookups

rule "Threat Intelligence Lookups (OTX): dst_addr"
when
has_field(“dst_addr”) && ! in_private_net(to_string($message.dst_addr))
then
let intel = otx_lookup_ip(to_string($message.dst_addr));
set_field(“threat_indicated_OTX_dst_addr”, intel.otx_threat_indicated);
set_field(“threat_ids”, intel.otx_threat_ids);
set_field(“threat_names”, intel.otx_threat_names);
end

I would just like to point out that this part “&& ! in_private_net(to_string($message.“dst_addr”))” in each of the rules is what prevents the lookup of internal IP ranges (RFC1918 addresses).

Stage 1 (because of the different fields added by the Threat Intel and OTX sources we need to unify them into common fields for us to search on)

rule "inflate threat intel results fields dst_addr_threat_indicated"
when
to_bool($message.dst_addr_threat_indicated, true) && ! in_private_net(to_string($message.“dst_addr”))
then
set_field(“threat_indicated”, true);
set_field(“threat_ip”,($message.dst_addr));
end

and don’f forget the rules for OTX lookup fields

rule "inflate threat intel results fields threat_indicated_OTX_dst_addr"
when
to_bool($message.“threat_indicated_OTX_dst_addr”, true) && ! in_private_net(to_string($message.“dst_addr”))
then
set_field(“threat_indicated”, true);
set_field(“threat_ip”,($message.“dst_addr”));
end

Stage 2 (now that we have common fields to search on i.e. threat_indicated and threat_ip we can use the Whois table to lookup who they are)

rule "WHOIS_for_Threats"
when
contains(to_string($message.threat_indicated), “true”, true) && ! in_private_net(to_string($message.threat_ip))
then
set_fields(whois_lookup_ip(to_string($message.threat_ip), “threat_ip”));
end

So now I’m able to do whois lookups for identified threats only, saving me some cpu cycles and reducing the hit rate for each of the lookup sources, hopefully preventing me from hitting any rate limit they impose. And I can simply search for any indicated threats by searching for threat_indicated:true

A couple of things to point out, if you haven’t picked up on them already.
1 - You needs to create a rule for each of the fields at Stage 0 and 1, I have used dst_addr in my example above.
2 - Each threat source (Threat_intel and OTX) at Stage 0 adds their own fields to each log. Threat intel adds ‘fieldname_threat_indicated’ and OTX adds ‘threat_indicated_OTX_fieldname’.

Hopefully this helps someone else out, I always hate finding a thread on a forum and it hasn’t been updated with an answer! There is no doubt more efficient ways of doing this, so if anyone has the time to read all this and point out where it could be improved that would be great.


(system) #13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.