Ok, so here’s what I’ve done. But for the sake of this thread let me just explain a few things.
I have several different fields containing an IP address that I need the Threat Intel plugin to check, excluding internal RFC1918 addresses. Those fields come from a range of different model Juniper firewalls, their log formats are quite different and I have a number of Extractors setup to use src_addr & dst_addr fields, the other fields below are automatically extracted by Graylog (which is why I’ve ended up with so many) but I don’t want to stop Graylog doing anything its supposed to (will save that for later). The different fields are:
source-address
destination-address
src_addr
dst_addr
IP
nat-source-address
nat-destination-address
For reference, my message processors are in the following order:
|#|Processor|Status|
|1|Message Filter Chain|active|
|2|Pipeline Processor|active|
|3|GeoIP Resolver|active|
My Extractors run within the Message Filter processor, and my Pipeline or course runs in the Pipeline processor.
I have setup a new Threat_Intel pipeline with the following rule, for each of the fields above, at each stage.
Stage 0 (gather the intel and set threat fields true or false)
rule "Threat Intelligence Lookups: dst_addr"
when
has_field(“dst_addr”) && ! in_private_net(to_string($message.“dst_addr”))
then
set_fields(threat_intel_lookup_ip(to_string($message.dst_addr), “dst_addr”));
end
and one specifically for OTX lookups
rule "Threat Intelligence Lookups (OTX): dst_addr"
when
has_field(“dst_addr”) && ! in_private_net(to_string($message.dst_addr))
then
let intel = otx_lookup_ip(to_string($message.dst_addr));
set_field(“threat_indicated_OTX_dst_addr”, intel.otx_threat_indicated);
set_field(“threat_ids”, intel.otx_threat_ids);
set_field(“threat_names”, intel.otx_threat_names);
end
I would just like to point out that this part “&& ! in_private_net(to_string($message.“dst_addr”))” in each of the rules is what prevents the lookup of internal IP ranges (RFC1918 addresses).
Stage 1 (because of the different fields added by the Threat Intel and OTX sources we need to unify them into common fields for us to search on)
rule "inflate threat intel results fields dst_addr_threat_indicated"
when
to_bool($message.dst_addr_threat_indicated, true) && ! in_private_net(to_string($message.“dst_addr”))
then
set_field(“threat_indicated”, true);
set_field(“threat_ip”,($message.dst_addr));
end
and don’f forget the rules for OTX lookup fields
rule "inflate threat intel results fields threat_indicated_OTX_dst_addr"
when
to_bool($message.“threat_indicated_OTX_dst_addr”, true) && ! in_private_net(to_string($message.“dst_addr”))
then
set_field(“threat_indicated”, true);
set_field(“threat_ip”,($message.“dst_addr”));
end
Stage 2 (now that we have common fields to search on i.e. threat_indicated and threat_ip we can use the Whois table to lookup who they are)
rule "WHOIS_for_Threats"
when
contains(to_string($message.threat_indicated), “true”, true) && ! in_private_net(to_string($message.threat_ip))
then
set_fields(whois_lookup_ip(to_string($message.threat_ip), “threat_ip”));
end
So now I’m able to do whois lookups for identified threats only, saving me some cpu cycles and reducing the hit rate for each of the lookup sources, hopefully preventing me from hitting any rate limit they impose. And I can simply search for any indicated threats by searching for threat_indicated:true
A couple of things to point out, if you haven’t picked up on them already.
1 - You needs to create a rule for each of the fields at Stage 0 and 1, I have used dst_addr in my example above.
2 - Each threat source (Threat_intel and OTX) at Stage 0 adds their own fields to each log. Threat intel adds ‘fieldname_threat_indicated’ and OTX adds ‘threat_indicated_OTX_fieldname’.
Hopefully this helps someone else out, I always hate finding a thread on a forum and it hasn’t been updated with an answer! There is no doubt more efficient ways of doing this, so if anyone has the time to read all this and point out where it could be improved that would be great.
