Lookup Tables + Threat Feeds + Alerts

Development
1

Available on my GITHUB (https://github.com/breakandinspect/graylog) you will find two .sh (bash) files which download (threat_feed.sh) and convert (convert_feeds.sh) some OpenSource threat intelligence feeds for use with GL-2.3 Lookup Tables. I followed the directions on this page: http://docs.graylog.org/en/2.3/pages/lookuptables.html to setup the data as well as cache/adapters/tables. Everything is loaded, but what I was disappointed to discover is that there is no way to crossmatch/search against the Lookup table and generate an Alert
 for instance, if you’re monitoring IDS/IPS information, or DNS information, etc the IPs/URLs could be bounced off the Lookup Tables to see if any known-bad, hot, threats etc have been white/black/grey listed for the IPs/URLs that appear in the logs. (this is low level SIEM functionality)

is there a chance that this sort of functionality will be available in the near future? this sort of ability could take graylog to the next level.

thanks - Break&Inspect.

1 Like
2

I was also playing with the idea of pipelines/pipeline rules to parse the log against the lookup tables then populate an independent field on the parsed log called “threat_detected” then an alert could be triggered on that field.

something along the lines of 


rule "threat_domain-feed-lookup"
when true
then
let threat_domain = lookup_value(“URL”, $message.xxxx);
set_field(“threat_detected”, “true”);
end

rule "threat_IP-feed-lookup"
when has_field(“sip”) || has_field(“dip”)
then when true
let threat_IP = lookup_value(“IPADDR”,$message.xxxx);
set _field(“threat_detected”,“true”);
end

followed by a pipeline
pipeline "threat detection"
stage 1 match either
rule threat_domain-feed-lookup;
rule threat_IP-feed-lookup;
end

a problem became evident early on as the documentation doesn’t specify how to setup (or doesn’t imply) that a pipeline rule with parse all the imported threat feeds (loaded into separate lookup tables; per the bash scripts i provided there are about 10 that get loaded, with 10’s of 1000’s of records in them).

any thoughts or helpful hints/direction would be appreciated. I am still in the design phase of this. getting the threat data was the easy part.

thanks, Break&Inspect.

3

hi,

thanks for your threat feed input. Just some suggestions for the script, in case you are interested:

adding something like this to the beginning of the script:

DIR="/graylog/threatintel"
FEEDDIR=$DIR/feeds
export https_proxy=proxy.company.com:443
export http_proxy=proxy.company.com:80


function convert {
    i=0
    cat $1 | while read line
    do
        let i++
        echo "$i$3$line" >> $2
    done
}

cd $FEEDDIR

allows for several things:

  1. no need to have a separate script to run the conversion. Just use function convert with the same parameters
  2. allows using a proxy, in case direct communication is blocked
  3. the script can be run from anywhere (the cd command switches to the rigth place anywhere.
4


 and adding |grep -v ^# between the cat and while gets rid of the comment lines.

I noticed that lookup tables do not seem to behave nicely, if the key contains the equal sign ‘=’. The lookup adapter seems to think that the two keys A=B and A=C are the same. I wonder if some quoting would help here


5

merged some of your changes and uploaded a new version of the threat feed script. now on my GITHUB as gl_threatfeeds.sh (single file)

leaving the comments from the top of some of the feeds shouldn’t really change the search but agree they could be dropped; i have to double check the OpenSource agreements of those threat feeds i believe those comment lines must remain there. either way, the “search” should bypass those lines when you’re just searching for IPs or URLs (domains).

equal sign “=” definitely comes into play with URLs, could wrap those lines in quotes but would be better if the search could just ignore them. should be doing string comparisons. does Record1=Query1? if not does Record2=Query1 if yes
 then let’s get an ALERT generated.

6

I think this would be a bug in Graylog in itself (or lack of documentation, if quoting works). The keys with ‘=’ do need to be handled properly, when thinking of threatintel feeds.

Some examples that don’t work now:

java.lang.IllegalArgumentException: Multiple entries with same key: =2 and =1
java.lang.IllegalArgumentException: Multiple entries with same key: http://111.9.32.249:8066/transmanagerment/attachment/comm/all=1077 and http://111.9.32.249:8066/transmanagerment/attachment/comm/all=1076
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.