Pipeline - Create field with lookup tables

[s0p4L1N]

Context

I am using nxlog and sidecar to ingest Windows DHCP Server Logs into Graylogs.

The logs are automatically parsed thanks to GELF format but there is no EventCategory sent by Windows.

I’ve followed this tutorial: How to use graylog lookup table

I have created the lookup table to match EventID with EventCategory.

• event.csv

"EventID";"EventCategory"
"70";"DHCPv4.ScopeConfigured"
"71";"DHCPv4.ScopeModified"
"72";"DHCPv4.ScopeDeleted"
"73";"DHCPv4.ScopeActivated"

And it is working:
EventID 107 lookup with EventCategory DHCPv4.ReservationDeleted

Now I’m looking to create a pipeline to add a new field “EventCategory” where the value will be written according to EventID value.

I tried this Pipeline rule but the rule simulation gives no result:

rule "Windows Server DHCP: EventID lookup with EventCategory"

when
  has_field("EventID")
then
let update_source = lookup_value("eventID_eventCategory",$message.EventID);
set_field("EventCategory", update_source);

end

Am I missing something ?
Thank you !

OS & Package Information:

• Rocky Linux 9
• docker compose 2.18.1
• Graylog 5.1.2
• traefik 2.10.1
• mongodb 6.0.6
• elasticsearch 7.10.2
• Windows Server DHCP
• nxlog latest
• sidecar latest

[gsmith]

Hey @s0p4L1N

The piepline look good. How does you Data Adaptor look? By default "," the delimiter to use for separating entries.Looks like your using ";" instead.

[s0p4L1N]

Here’s my data adapter config:

• I use ; as a separator in my case.

• When I try the pipeline simulator in Pipelines > Simulator

• I select the DHCP Stream

• RAW Message:

{
  "Task": 0,
  "Keywords": -9223372036854776000,
  "EventType": "INFO",
  "collector_node_id": "srvad1",
  "gl2_remote_ip": "192.168.1.11",
  "gl2_remote_port": 50586,
  "Opcode": "Informations",
  "IP_Name": "[[192.168.1.0]company.lan]",
  "source": "SRVAD1.COMPANY.lan",
  "gl2_source_input": "64a3c7a431459c203793e12c",
  "ClientName": "COMPANY\\username",
  "SeverityValue": 2,
  "Version": 0,
  "UserID": "S-1-5-21-3256819886-XXXXXXXXX-XXXXXXXXXX-XXX",
  "gl2_source_node": "3b147713-efd6-45b0-83e4-f3b8aeea69ef",
  "ProcessID": 2404,
  "timestamp": "2023-07-04T15:35:32.000Z",
  "gl2_accounted_message_size": 977,
  "OpcodeValue": 0,
  "gl2_source_collector": "78fd7c59-b789-4596-be22-1754f8a179cf",
  "SourceModuleType": "im_msvistalog",
  "level": 6,
  "Channel": "Microsoft-Windows-Dhcp-Server/Operational",
  "streams": [
    "64a3f93a565c4c3e82b3ef81"
  ],
  "gl2_message_id": "01H4GRPTD0000016SAPK1ZCAVB",
  "SourceName": "Microsoft-Windows-DHCP-Server",
  "Severity": "INFO",
  "message": "La réservation [[192.168.1.170]] pour IPv4 est supprimée sous l’étendue [[192.168.1.0]company.lan] par COMPANY\\username.",
  "AccountType": "User",
  "EventReceivedTime": "2023-07-04 17:35:33",
  "SourceModuleName": "dhcp_server_eventlog",
  "ProviderGuid": "{6D64F02C-A125-4DAC-9A01-F0555B41CA84}",
  "full_message": "La réservation [[192.168.1.170]] pour IPv4 est supprimée sous l’étendue [[192.168.1.0]company.lan] par COMPANY\\username.",
  "ThreadID": 11076,
  "EventID": 107,
  "ReservationName": "[[192.168.1.170]]",
  "_id": "80143c92-1a80-11ee-858f-0242ac150005",
  "Domain": "COMPANY",
  "RecordNumber": 134,
  "AccountName": "username"
}

• Message input:

gelfTCP / GELF TCP

• Message codec:

GELF

But now that I understand by writing my message, I just forgot to add the pipeline and the stage in order to the simulator to work !

I though the simulator was here to test the rule before applying it to any stream ?
But instead you need to create the pipeline and add the rule to then test it with the simulator.

Anyway, it works

[gsmith]

Hey @s0p4L1N

Awesome Some time it helps just talking about it.