Threat Intel not working 2.4.6

[CypherBit]

I need help with the Treat Intel plugin, it just doesn’t work for me, I created a similar thread before: Testing Theat Intel plugin not working, but was extra busy in that time period and wanted to test on the newest version (if that helped).

I’m now running the latest version of Graylog as well as the plugin (confirmed by browsing to the /plugins folder).

I’m testing the Ransomware domain blocklist: https://ransomwaretracker.abuse.ch/downloads/TC_C2_DOMBL.txt in particular with chaliawala.com (IP 45.35.227.45).

I visit the site and the the DestinationIp_threat_indicated is false:

The plugin is enabled:

Lookup Tables seem to fine:

As well as Data Adapters:

This is what I see in the current log (nothing related to Threat Intel if I’m looking correctly):
2018-08-02_08:24:56.81994 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:135) ~[graylog.jar:?]
2018-08-02_08:24:56.81994 at org.graylog2.alerts.AlertConditionFactory.createAlertCondition(AlertConditionFactory.java:52) ~[graylog.jar:?]
2018-08-02_08:24:56.81994 at org.graylog2.alerts.AlertServiceImpl.fromPersisted(AlertServiceImpl.java:170) ~[graylog.jar:?]
2018-08-02_08:24:56.81995 at org.graylog2.streams.StreamServiceImpl.getAlertConditions(StreamServiceImpl.java:294) [graylog.jar:?]
2018-08-02_08:24:56.81995 at org.graylog2.periodical.AlertScannerThread.doRun(AlertScannerThread.java:55) [graylog.jar:?]
2018-08-02_08:24:56.81995 at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
2018-08-02_08:24:56.81995 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
2018-08-02_08:24:56.81996 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
2018-08-02_08:24:56.81996 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
2018-08-02_08:24:56.81996 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
2018-08-02_08:24:56.81997 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
2018-08-02_08:24:56.81997 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
2018-08-02_08:24:56.81997 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2018-08-02_08:24:56.82020 ERROR [StreamServiceImpl] Skipping alert condition.
2018-08-02_08:24:56.82021 java.lang.IllegalArgumentException: Unknown alert condition type: Aggregates Alert
2018-08-02_08:24:56.82021 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:135) ~[graylog.jar:?]
2018-08-02_08:24:56.82021 at org.graylog2.alerts.AlertConditionFactory.createAlertCondition(AlertConditionFactory.java:52) ~[graylog.jar:?]
2018-08-02_08:24:56.82021 at org.graylog2.alerts.AlertServiceImpl.fromPersisted(AlertServiceImpl.java:170) ~[graylog.jar:?]
2018-08-02_08:24:56.82022 at org.graylog2.streams.StreamServiceImpl.getAlertConditions(StreamServiceImpl.java:294) [graylog.jar:?]
2018-08-02_08:24:56.82022 at org.graylog2.periodical.AlertScannerThread.doRun(AlertScannerThread.java:63) [graylog.jar:?]
2018-08-02_08:24:56.82022 at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
2018-08-02_08:24:56.82022 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
2018-08-02_08:24:56.82022 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_172]
2018-08-02_08:24:56.82023 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
2018-08-02_08:24:56.82023 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_172]
2018-08-02_08:24:56.82023 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
2018-08-02_08:24:56.82025 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
2018-08-02_08:24:56.82025 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]

The Graylog server has unrestricted access to the internet.

Please assist.

[jan (Jan Doberstein)]

how did you run those thread intel lookups? Are you using a processing pipeline?

What is the content of the rules? Did you checked if the rules run on the messages?

[CypherBit]

I tested by visiting the site in the browser, didn’t know to do much more.

The rules are as follows:

rule “Threat Intelligence Lookups: domain”
when
has_field(“DestinationHostname”)
then
set_fields(threat_intel_lookup_domain(to_string($message.DestinationHostname), “DestinationHostname”));
end

rule “Threat Intelligence Lookups: dst_addr”
when
has_field(“DestinationIp”)
then
set_fields(threat_intel_lookup_ip(to_string($message.DestinationIp), “DestinationIp”));
end

rule “Threat Intelligence Lookups: src_addr”
when
has_field(“SourceIp”)
then
set_fields(threat_intel_lookup_ip(to_string($message.SourceIp), “SourceIp”));
end

The rules have a lot of throughput every time I check.

I’m pretty sure the rules run on these messages, since this is the only stream that has the threat_indicated entry and it is visible on the messages:

How else can I check?

[jan (Jan Doberstein)]

Did you checked your Graylog server.log ?

Some of the external services block or throttle your request when you are above certain limits.

[CypherBit]

I posted the output of the current log above. I might add that not a single threat has been detected so far, which seems a bit odd.

[jan (Jan Doberstein)]

you posted a few lines not a complete log - with the given information it is not possible to find a reason.

[CypherBit]

My appologies, Here’s the complete log: https://1drv.ms/u/s!Amnw_mqG-fo2s4IWWdDPuhQ7OxxiFA.

[CypherBit]

Have you perhaps found anything, I really need to get this to work not sure what’s the problem.

Is there perhaps a list of URLs, IPs that the plugin connects to, so we’d at least know if it’s making the connection at all?