SLOOKUP vs Lookup Tables


(Naren) #1

Hello Community/ Developers,

We are currently feeding IoCs from threat intel feeds to dedicated graylog stream on a daily basis (approx 4 MB of data) and using slookup function in pipeline rules to find a match in real time (P.S: Relative data timeframes: 3days). Right now, running smooth when tested on 4000 message/second throughput.

In future, we are planning to increase more sources to graylog. Is slookup way efficient? or do we need to use Lookup Tables that we are not confident with? Your valuable feedback will help us.

Sample raw message (Threat Intel IoC) : 046217f5bae309bf79fff719e18892570aa092febb0096b9169760ae2bab24c2;Intel::FILE_HASH;(100|43|Gen:Variant.Symmi) https://www.hybrid-analysis.com/feed?raw&hts

Details:
Graylog v2.4.4
RAM: 64 GB
Nodes: 2
Last 30 days Throughput: 1037 messages/sec
Peak Hour Throughput: ~7000 messages/sec
Future Throughput (expecting): ~15000 messages/sec


(Philipp Ruland) #2

Hey @naren,

I would recommend you have a look at lookup tables. From a performance standpoint, it is substantially faster than the SLOOKUP plugin. The plugin is awesome and has a lot of usecases, I’ll give it that, but it’ll run a query against Elasticsearch every time it is invoked. This is very resource intensive in comparison to the lookup tables. If you would put your data in a CSV file that is loaded by Graylog, the loading time is lower (since it will only need some I/O from your Graylog node to read that CSV instead of having to run an entire search against Elasticsearch, and you can define an appropriate cache, speeding things up even more.

Don’t worry, lookup tables are straight forward. Have a look here:
http://docs.graylog.org/en/2.4/pages/lookuptables.html

Step 1: Create Data Adapter (make sure that Graylog has read permissions on the file)
Step 2: Create Cache (Select a size and for your data relevant eviction policy)
Step 3: Create Lookup Table (Combine the Adapter and the Cache to a Lookup Table)
Step 4: Profit. (Use it in Extractors, Converters, Decorators or Pipeline Rules)

Greetings,
Philipp


(Naren) #3

@derPhlipsi Thank you very much for your quick revert on this query. I will immediately start exploring Lookup Tables and will update the status.


(Naren) #4

@derPhlipsi Is there any option to do multi-value lookup using CSV as a Data Adapter Type?

ex:
key: Indicator
value: Feed_Name, Feed_Name

CSV Header:“Indicator”,“Indicator_Type”,“Feed_URL”,“Feed_Name”,“Feed_Name”


(Philipp Ruland) #5

Heyo @naren,

AFAIK there is no multi-value lookup for the CSV adapter.
You could give this ticket a bump to let the devs know that this is a wanted feature :slight_smile:

A workaround at the moment would be to set up multiple lookup tables, one for each field you want to add :slight_smile:

Greetings,
Philipp


(Naren) #6

@derPhlipsi
Thank you very much, Philipp :slight_smile: You are always there whenever we need help.


(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.