Hello Community/ Developers,
We are currently feeding IoCs from threat intel feeds to dedicated graylog stream on a daily basis (approx 4 MB of data) and using slookup function in pipeline rules to find a match in real time (P.S: Relative data timeframes: 3days). Right now, running smooth when tested on 4000 message/second throughput.
In future, we are planning to increase more sources to graylog. Is slookup way efficient? or do we need to use Lookup Tables that we are not confident with? Your valuable feedback will help us.
Sample raw message (Threat Intel IoC) : 046217f5bae309bf79fff719e18892570aa092febb0096b9169760ae2bab24c2;Intel::FILE_HASH;(100|43|Gen:Variant.Symmi) https://www.hybrid-analysis.com/feed?raw&hts
RAM: 64 GB
Last 30 days Throughput: 1037 messages/sec
Peak Hour Throughput: ~7000 messages/sec
Future Throughput (expecting): ~15000 messages/sec