GrayLog Stream Lookup (SLookup) Pipeline Processor function

the NEW Marketplace Plug-Ins
1

GrayLog Stream Lookup (SLookup) Pipeline Processor function

@billmurrin

Download from Github
View on Github
Issues
Stargazers

Plugin SLookup 2.0.0 - Multiple Return Fields

Stream Lookup function for GrayLog2 Pipeline Processor

SLookup facilitates the lookup of a local stream’s field value on a remote stream field, and if it matches, returns the requested fields for enrichment in the source stream.

For example, say there are two streams, one contains some http logs with source IPs (E.g. src_ip ) from internal hosts and the other stream contains information about the systems on the network such as IP address (E.g. ip_address ), computer name (E.g. computer_name ), MAC address (E.g. mac_address ), OU, make/model, etc.

In the example above, you might want to return the computer_name and mac_address fields where the value of src_ip matches ip_address .

The thought behind this function is to implement a similar functionality to the VLOOKUP function in Excel.

With features like index sets being introduced in Graylog 2.x, it is possible to use data in one stream to enrich data in another with Pipeline Processor rules.

Version 2.0.0 tested to work with Graylog 2.3.2 and 2.4.0

2

Github user jimzz2live created a fork of this project that adds support for Graylog 4.1. See his releases here: Releases · jimzz2live/graylog-plugin-slookup-function · GitHub

1 Like
3

Thanks for heads up, @inventor96!

In the New Marketplace community, I’d like to welcome Github user jimzz2live to contact me at david.sciuto@graylog.com to help highlight the added support feature.

4

Anyone tested this on Graylog 5?

5

Yes, works nice on graylog 5.0.6 with jimzz2live fork mentioned by inventor96 :slight_smile:

6

Trouble Using slookup – Error: Cannot invoke “Object.toString()” because getField(String) is null

Hi Graylog Community,

I’m currently using Graylog 5.2.12 and working on a pipeline rule that uses the slookup function to correlate Event ID 4662 (directory access) with 4624 (logon events) from a different stream.

The goal is to enrich 4662 logs with logon context (IpAddress, WorkstationName, timestamp) by looking them up from a custom stream (UserLogon). But, I occasionally get the following error:

Error: In call to function ‘slookup’ at 10:15 an exception was thrown: Cannot invoke “Object.toString()” because the return value of “org.graylog2.plugin.Message.getField(String)” is null

It seems the error happens when one or more of the fields to return from slookup (e.g. IpAddress, WorkstationName) are missing in the matched message. In some logon events (especially with certain protocols), these fields are simply not populated.
What I’ve tried:

Confirmed that SubjectUserName is always present in the source event.

Checked the destination stream (UserLogon) and noticed some messages are present sometimes with missing values such as IpAddress or WorkstationName.

Tried to_string() with checks, but it seems the plugin fails before the rule executes further when the result is null.

My Question:

Is there a way to safely use slookup even if some of the returned fields might be null?

Is there a recommended way to protect against nulls in the fields returned by slookup?

Could this be handled gracefully in a newer version of the plugin or Graylog?

Should I pre-clean the data in the UserLogon stream to drop events with missing fields?

Rule:

rule “Correlate 4662 with 4624 logon”
when
has_field(“EventID”) &&
to_string($message.EventID) == “4662” &&
has_field(“SubjectUserName”) &&
to_string($message.SubjectUserName) != “”
then
let user = to_string($message.SubjectUserName);

let result = slookup(
“6849522f78f9a85e9f8cab00”,“TargetUserName”, user, [“IpAddress”, “WorkstationName”, “timestamp”],“7200”,“desc” );

//set_field(“correlated_with_logon”, true);

set_field(“logon_ip”, result[0]);
set_field(“logon_workstation”, result[1]);
set_field(“logon_time”, result[2]);
end

Any advice or ideas would be greatly appreciated. Thanks in advance!