Graylog 4 Integration with Threat intel plugin

wajahatraza · October 31, 2021, 8:56pm

Description

Hi all,

I am trying to configure threat intel plugin in Graylog 4.1.2-1. But I am getting following errors in log file. I have posted my configuration in detail below. Can anyone tell me if I am missing anything?

2021-11-01T00:50:07.021+05:00 WARN  [LookupTableService] Lookup table  does not exist
2021-11-01T00:50:07.036+05:00 WARN  [LookupTableService] Lookup table  does not exist
2021-11-01T00:50:07.038+05:00 WARN  [LookupTableService] Lookup table  does not exist
2021-11-01T00:50:07.039+05:00 WARN  [LookupTableService] Lookup table  does not exist
2021-11-01T00:50:07.040+05:00 WARN  [LookupTableService] Lookup table  does not exist
2021-11-01T00:50:07.048+05:00 WARN  [LookupTableService] Lookup table  does not exist

Description of steps I’ve taken to attempt to solve the issue

I have done following configuration,

I have enabled the Plugin from configuration:

Plugin is already installed in graylog server, see jar file in plugin directory

[root@greylogs plugin]# ls -lrht /usr/share/graylog-server/plugin/ | grep threat
-rw-r–r-- 1 root root 8.2M Jul 28 22:43 graylog-plugin-threatintel-4.1.2.jar

I have also configured the pipeline to search for specific field:

rule "Threat intel "

when

  has_field("src_ip")

then

let intel = spamhaus_lookup_ip(to_string($message.src_ip));
set_field("threat_indicated", intel.threat_indicated);

Apparently my configuration is correct as Graylog is able to tag fields. and a fields is created “src_ip_threat_indicated:false”.
However my logs are flooded with “WARN [LookupTableService] Lookup table does not exist” logs.

Environmental information

Operating system information

CentOS

Package versions

Graylog graylog-server-4.1.2-1.noarch
MongoDB mongodb-org-server-4.2.15-1.el7.x86_64
Elasticsearch elasticsearch-oss-7.10.2-1.x86_64

gsmith · November 1, 2021, 10:13pm

Hello && Welcome

I might be able to help. How did you install and configure Graylog? Or was this an upgrade?
Do you have other plugins enabled? if so are they also having problems?
By chance have you tried to restart Graylog-server service?

wajahatraza · November 2, 2021, 9:59am

Hi gsmith,

Thank you for the response. I installed Graylog using official documentation (see URL below) and installed using yum command after adding repo. It was not an upgrade as it was fresh installation. Yes, I have Geo-IP processor plugin enabled and it is working correctly (for this I followed the documenation to download .mmdb file and created data adapters and lookup table. And Yes, I have also tried restarting Graylog-server service but status is same.

CentOS installation - Installing Graylog

Regards
Wajahat

gsmith · November 2, 2021, 9:59pm

Hello,

Curious, what order does you Message Processors Configuration look like. This is located System/Configurations.

Example:

Also. Since this is a fresh installation have you tried to remove/disable you GeoIP. For obvious reasons Graylog cant find your lookup table. Either your installation maybe incorrect or a configuration made could be the cause of the issue. For troubleshooting purposes try roll back you installation to the basic’s and see if you still have issues. A greater detail about you environment would be helpful.

EDIT: By chance do you have Selinux enabled? can you confirm that all your Data Adapters are running. This can be seen when you restart Graylog service while TAIL’ing the server.log file.
If you could show your full Graylog file while starting up that would be great.

Something like this after restarting Graylog service.

2021-11-02T20:35:19.687-05:00 INFO  [LookupTableService] Data Adapter greynoise-test/613bca94817e6d2f90d55526 [@7af6f0aa] STARTING
2021-11-02T20:35:19.690-05:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a501ec3ffe8b12df3bcc46f [@38af613e] STARTING
2021-11-02T20:35:19.692-05:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2021-11-02T20:35:19.698-05:00 INFO  [LookupTableService] Data Adapter greynoise-test/613bca94817e6d2f90d55526 [@7af6f0aa] RUNNING
2021-11-02T20:35:19.699-05:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a501ec3ffe8b12df3bcc46a [@1bc39f69] STARTING
2021-11-02T20:35:19.699-05:00 INFO  [LookupTableService] Data Adapter geo-locator/5deb235a83d72ece8f020fa7 [@34967eb6] STARTING
2021-11-02T20:35:19.699-05:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a501ec3ffe8b12df3bcc46a [@1bc39f69] RUNNING
2021-11-02T20:35:19.700-05:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2021-11-02T20:35:19.699-05:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a501ec3ffe8b12df3bcc46c [@1e670b5] STARTING
2021-11-02T20:35:19.701-05:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/615e1b1650787f565f29c017 [@791c4705] STARTING
2021-11-02T20:35:19.701-05:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/615e1b1650787f565f29c017 [@791c4705] RUNNING
2021-11-02T20:35:19.707-05:00 INFO  [LookupTableService] Data Adapter whois/5a501ec3ffe8b12df3bcc46d [@3978594d] STARTING
2021-11-02T20:35:19.708-05:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a501ec3ffe8b12df3bcc470 [@5e74ce3f] STARTING
2021-11-02T20:35:19.710-05:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a501ec3ffe8b12df3bcc470 [@5e74ce3f] RUNNING
2021-11-02T20:35:19.709-05:00 INFO  [LookupTableService] Data Adapter whois/5a501ec3ffe8b12df3bcc46d [@3978594d] RUNNING
2021-11-02T20:35:19.722-05:00 INFO  [LookupTableService] Data Adapter geo-locator/5deb235a83d72ece8f020fa7 [@34967eb6] RUNNING
2021-11-02T20:35:19.722-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geo-locator/5deb235a83d72ece8f020fa7/@34967eb6> [interval=60000ms]
2021-11-02T20:35:19.797-05:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a501ec3ffe8b12df3bcc462 [@400c9c69] STARTING
2021-11-02T20:35:19.797-05:00 INFO  [LookupTableService] Cache watchlist-cache/615e1b1650787f565f29c015 [@38951750] STARTING
2021-11-02T20:35:19.797-05:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a501ec3ffe8b12df3bcc463 [@19affb41] STARTING
2021-11-02T20:35:19.798-05:00 INFO  [LookupTableService] Cache grey-cache/613bcb34817e6d2f90d55771 [@3a7b211c] STARTING
2021-11-02T20:35:19.801-05:00 INFO  [LookupTableService] Cache geoip/5f28d50c654a4710e38da6eb [@5db1f9c0] STARTING
2021-11-02T20:35:19.803-05:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a501ec3ffe8b12df3bcc462 [@400c9c69] RUNNING
2021-11-02T20:35:19.803-05:00 INFO  [LookupTableService] Cache watchlist-cache/615e1b1650787f565f29c015 [@38951750] RUNNING
2021-11-02T20:35:19.803-05:00 INFO  [LookupTableService] Cache whois-cache/5a501ec3ffe8b12df3bcc465 [@ddd4e04] STARTING
2021-11-02T20:35:19.803-05:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a501ec3ffe8b12df3bcc463 [@19affb41] RUNNING
2021-11-02T20:35:19.803-05:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a501ec3ffe8b12df3bcc466 [@40ccbc3b] STARTING
2021-11-02T20:35:19.804-05:00 INFO  [LookupTableService] Cache geoip/5f28d50c654a4710e38da6eb [@5db1f9c0] RUNNING
2021-11-02T20:35:19.804-05:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a501ec3ffe8b12df3bcc464 [@6b4ec5df] STARTING
2021-11-02T20:35:19.806-05:00 INFO  [LookupTableService] Cache whois-cache/5a501ec3ffe8b12df3bcc465 [@ddd4e04] RUNNING
2021-11-02T20:35:19.806-05:00 INFO  [LookupTableService] Cache grey-cache/613bcb34817e6d2f90d55771 [@3a7b211c] RUNNING
2021-11-02T20:35:19.806-05:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a501ec3ffe8b12df3bcc464 [@6b4ec5df] RUNNING
2021-11-02T20:35:19.808-05:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a501ec3ffe8b12df3bcc466 [@40ccbc3b] RUNNING
2021-11-02T20:35:20.453-05:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a501ec3ffe8b12df3bcc46f [@38af613e] RUNNING
2021-11-02T20:35:20.454-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <spamhaus-drop/5a501ec3ffe8b12df3bcc46f/@38af613e> [interval=43200000ms]
2021-11-02T20:35:20.560-05:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2021-11-02T20:35:20.913-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <tor-exit-node/5a501ec3ffe8b12df3bcc46c/@1e670b5> [interval=3600000ms]
2021-11-02T20:35:20.913-05:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a501ec3ffe8b12df3bcc46c [@1e670b5] RUNNING
2021-11-02T20:35:20.927-05:00 INFO  [LookupTableService] Starting lookup table otx-api-ip/5a501ec3ffe8b12df3bcc472 [@228144f5] using cache otx-api-ip-cache/5a501ec3ffe8b12df3bcc463 [@19affb41], data adapter otx-api-ip/5a501ec3ffe8b12df3bcc46a [@1bc39f69]
2021-11-02T20:35:20.928-05:00 INFO  [LookupTableService] Starting lookup table whois/5a501ec3ffe8b12df3bcc473 [@1c292765] using cache whois-cache/5a501ec3ffe8b12df3bcc465 [@ddd4e04], data adapter whois/5a501ec3ffe8b12df3bcc46d [@3978594d]
2021-11-02T20:35:20.928-05:00 INFO  [LookupTableService] Starting lookup table spamhaus-drop/5a501ec3ffe8b12df3bcc476 [@221444ca] using cache spamhaus-e-drop-cache/5a501ec3ffe8b12df3bcc466 [@40ccbc3b], data adapter spamhaus-drop/5a501ec3ffe8b12df3bcc46f [@38af613e]
2021-11-02T20:35:20.928-05:00 INFO  [LookupTableService] Starting lookup table otx-api-domain/5a501ec3ffe8b12df3bcc477 [@eef6be8] using cache otx-api-domain-cache/5a501ec3ffe8b12df3bcc464 [@6b4ec5df], data adapter otx-api-domain/5a501ec3ffe8b12df3bcc470 [@5e74ce3f]
2021-11-02T20:35:20.929-05:00 INFO  [LookupTableService] Starting lookup table tor-exit-node-list/5a501ec3ffe8b12df3bcc478 [@2661302c] using cache threat-intel-uncached-adapters/5a501ec3ffe8b12df3bcc462 [@400c9c69], data adapter tor-exit-node/5a501ec3ffe8b12df3bcc46c [@1e670b5]
2021-11-02T20:35:20.929-05:00 INFO  [LookupTableService] Starting lookup table geoip/5f28d53e654a4710e38da79c [@7326398a] using cache geoip/5f28d50c654a4710e38da6eb [@5db1f9c0], data adapter geo-locator/5deb235a83d72ece8f020fa7 [@34967eb6]
2021-11-02T20:35:20.929-05:00 INFO  [LookupTableService] Starting lookup table GreyNoise-Lookup/613bcb53817e6d2f90d557d3 [@73940130] using cache geoip/5f28d50c654a4710e38da6eb [@5db1f9c0], data adapter greynoise-test/613bca94817e6d2f90d55526 [@7af6f0aa]
2021-11-02T20:35:20.929-05:00 INFO  [LookupTableService] Starting lookup table watchlist/615e1b1650787f565f29c019 [@6d19481] using cache watchlist-cache/615e1b1650787f565f29c015 [@38951750], data adapter watchlist-mongo/615e1b1650787f565f29c017 [@791c4705]

wajahatraza · November 3, 2021, 11:31am

Hi,

Please see below screenshot of the configuration:

Selinux is disabled in my environment. Can you please share if I have to do any additional steps after enabling plugins from configuration. Do we have to create lookup tables manually for Tor-exit-node and Spamhaus?

gsmith · November 3, 2021, 9:42pm

Can you show your full Graylog log file after restarting Graylog service? I showed you this in my post above.

What have you tried to resolve this issue?

I never had to, the install created it.

Without more information its difficult to understand what is happing in your environment.

wajahatraza · November 9, 2021, 4:05pm

Hi Gsmith,

Thanks a lot for your time and support. I was able to configure it successfully. I initially enabled the Plugins from configuration and defined it in Pipeline, the fields were populated but was giving error that lookup table was not found. The step I missed here was to install Threat intel plugins from “system/contentpacks” page which actually creates Lookup tables (or create manually). However I created the lookup tables manually and the lookup started working as expected (because I totally missed the content packs page).

Thanks and regards

gsmith · November 9, 2021, 10:10pm

Nice, glad your issue is resolved

system · November 23, 2021, 10:10pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Reinstall Threat Intel Plugin Graylog Add-ons	3	888	May 13, 2019
2021-12-09 09:05:40,063 WARN : org.graylog2.lookup.LookupTableService - Lookup table <abuse-ch-ransomware-ip> does not exist Graylog Central (peer support) basic-configuration	6	1415	December 25, 2021
Validating Threat Intelligence in v3.2.4 Graylog Central (peer support)	3	824	April 16, 2020
Threat Intel Setup Graylog Central (peer support)	6	851	June 6, 2022
Trying to get the Open Threat Exchange - Threat intel plugin working on a Graylog Graylog Central (peer support) pipeline-rules	10	1314	June 25, 2021