Converting the OID'd for streams Into Human Readable

Templates and Rules Exchange Pipeline Rules
1

In my Rest Access file there are stream OID’s, to display them on a Widget I needed to created a lookup table and Attached it to a pipeline.

Example:

Acquire the OID’s from MongoDb:

> use graylog
switched to db graylog
> db.streams.find({},{"title":1,"ObjectId":1});
{ "_id" : ObjectId("000000000000000000000001"), "title" : "All messages" }
{ "_id" : ObjectId("5a502736ffe8b10359166cf1"), "title" : "Firewall: Configuration Changed" }
{ "_id" : ObjectId("5a5027b7ffe8b10359166d83"), "title" : "Linux: Failed Authentication  " }
{ "_id" : ObjectId("5a502865ffe8b10359166e4d"), "title" : "Switches: Failed Logon Attempts " }
{ "_id" : ObjectId("5a5028a4ffe8b10359166e9a"), "title" : "Windows: User Account Was Lockout" }
{ "_id" : ObjectId("5a5028f6ffe8b10359166efa"), "title" : "Windows: Audit Logs Are Cleared" }
{ "_id" : ObjectId("5a502a36ffe8b1035916706d"), "title" : "Windows: Service Failed Authentication Attempt  " }
{ "_id" : ObjectId("5a502a80ffe8b103591670c3"), "title" : "Windows: User Credentials Failed logon Attempt" }
{ "_id" : ObjectId("5a502aebffe8b10359167141"), "title" : "Windows: User was Granted Administrator " }
{ "_id" : ObjectId("5a502b15ffe8b10359167174"), "title" : "Windows: User Account was Created " }
{ "_id" : ObjectId("5a502b71ffe8b103591671e0"), "title" : "Windows: Virtual Machine Faulty Disk " }
{ "_id" : ObjectId("5a6a7b9183d72e84ac7dd681"), "title" : "Windows: Server Name cannot be Resolved" }
{ "_id" : ObjectId("5ab06e1183d72e1a4a7ae1e0"), "title" : "Windows: Warning StorVSC " }
{ "_id" : ObjectId("5ade5c9c83d72e8a1240789f"), "title" : "Microsoft Antimalware " }
{ "_id" : ObjectId("5b9201d583d72e03a33047d6"), "title" : "Windows: User Successful Logon Local" }
{ "_id" : ObjectId("5b99ecd383d72e03a48fbfd1"), "title" : "Windows: Domain Controller  DNS Error" }
{ "_id" : ObjectId("5b9b013d83d72e03a490eb82"), "title" : "Windows: DNS Server Errors" }
{ "_id" : ObjectId("5b9b016183d72e03a490ebaa"), "title" : "Windows: Domain Control DFS Replication" }
{ "_id" : ObjectId("5b9b017c83d72e03a490ebcb"), "title" : "Windows: Domain Controller Directory Service" }
{ "_id" : ObjectId("5b9c490d83d72e03a4924ee9"), "title" : "Windows: Domain Controller Time Sync Warning" }
Type "it" for more

Copy & Paste into lookup_table.csv.

[root@graylog graylog]# cat lookup_streams.cvs
"objectid","title"
"000000000000000000000001","All messages"
"5a502736ffe8b10359166cf1","Configuration Changed"
"5a5027b7ffe8b10359166d83","Failed Authentication"
"5a502865ffe8b10359166e4d","Switch Failed Logon Attempts"
"5a5028a4ffe8b10359166e9a","Windows User Account Was Lockout"
"5a5028f6ffe8b10359166efa","Windows Audit Logs Are Cleared"
"5a502a36ffe8b1035916706d","Windows Service Failed Authentication Attempt"
"5a502a80ffe8b103591670c3","Windows User Credentials Failed logon Attempt"
"5a502aebffe8b10359167141","Windows User was Granted Administrator"
"5a502b15ffe8b10359167174","Windows User Account was Created"
"5a502b71ffe8b103591671e0","Windows Virtual Machine Faulty Disk"
"5a6a7b9183d72e84ac7dd681","Windows Server Name cannot be Resolved"
"5ab06e1183d72e1a4a7ae1e0","Windows Warning StorVSC"
"5ade5c9c83d72e8a1240789f","Microsoft Antimalware"
"5b9201d583d72e03a33047d6","Windows User Successful Logon Local"
"5b99ecd383d72e03a48fbfd1","Windows Domain Controller DNS Error"
"5b9b013d83d72e03a490eb82","Windows DNS Server Errors"

I created a regex Extractor On Graylog Server INPUT to grab the OID and place it into a field called “stream_id”.

Using the Lookup_table in a pipeline.

rule"stream-lookup"
when
 has_field("stream_id")
then
let streamoid = lookup_value("streamoid",$message.stream_id);
set_field("stream_name",streamoid);
end

Attach it to a stream that Graylog Server is using.

2

rather than $message.message, don’t you want something like $message.objectid?

1 Like
3

What, if a message is “member” of multiple streams? I have the case with a firewall: low-level firewalling is part of one stream, high-level Antivirus is part of another streams. Both messages are part of the complete FW-Stream. Each field “streams” contains all the IDs for the streams it is member of - it’s one of the magic fields from Graylog.

To save diskspace on elastic these fields could be added via a decorators and only be visible live on the fly.

1 Like
4

Good catch @tmacgbay Yeah I should have slowed down.

5

If the firewall logs come in one input/port then the execrator can create that field.
I believe then the rest should function by adding stream/s to the pipeline connection:

image
image1266×376 38.1 KB

To be honest I’m not sure, I have only tested this on “graylog .rest access log”, and “MongoDb Log” files.
Since MongoDb holds all the metadata( OID’s, UUID’s, GUID, etc…) from streams/INPUTS or any other user configurations I can grab those and send them to Graylog -Server.
This is how I’m looking at it.

MongoDb → Log_file → Graylog → Input → extractor → Stream → Pipeline

I probably should have stated that in the extractor I added conditions, depending on the environment these settings can be modified .

image
image1027×448 31.6 KB

Adding on , I used conditions to get only what I needed. It took a while in getting these oid’s and from where.

EDIT: @ihe for a better Idea, I did this grabbing the User UUID and making it readable.

1 Like