1. Describe your incident:
I’m working with pipelines to parse security events from VSphere 8.X, it works out well but i’m encountering an issue and i’m stuck.
Some messages are split in two because the size of the message is more than 1024 characters
As you can see below, my pipeline rule can not parse the two splitted messages:
It look like this:
- Message 1/2
Event [2528773] [1-2] [2023-10-31T14:39:02.14294Z] [vim.event.VmReconfiguredEvent] [info] [LAB\adm.user] [Datacenter] [2528772] [Reconfigured o8-test1 on 192.168.1.201 in Datacenter.
- Message 2/2
Event [2528773] [2-2] , level = "normal"), limit = -1), externalId = <unset>, uptCompatibilityEnabled = true, uptv2Enabled = false);
The default message format when size is inferior to 1024 character look like this:
Event [2528772] [1-1] [2023-10-31T14:38:59.283901Z] [vim.event.TaskEvent] [info] [LAB\adm.user] [Datacenter] [2528772] [Task: Reconfigure virtual machine]
My pipeline rule is this one:
rule "Syslog - VMWARE VCSA"
when
has_field("message") AND regex("Event \\[", to_string($message.message)).matches == true
then
let msg = to_string($message.message);
let vcsa1 = grok(pattern: "Event \\[%{BASE10NUM:eventId}\\] \\[%{DATA:partInfo}\\] \\[%{TIMESTAMP_ISO8601:createdTime}\\] \\[vim.event.%{DATA:eventType}\\] \\[%{DATA:severity}\\] \\[%{DATA:user}\\] \\[%{DATA:target}\\] \\[%{DATA:chainId}\\] \\[%{DATA:desc}\\]", value: to_string(msg), only_named_captures: true);
set_fields(vcsa1);
end
This rule is not working with the two splitted message because the last field does not have the bracket closing as it is is splitted in another message.
2. Describe your environment:
Graylog 5.1.7
opensearch 2.9.0
3. What steps have you already taken to try and solve the problem?
Searching for pipeline function without success.
4. How can the community help?
Could you help me on this issue, meaning adapt my existing pipeline rule to parse the two or more splitted message, and create the corresponding fields like if it was 1 message.
As graylog does not have the vmware limitation for character size, could I concatenate the two message, remove the two splitted, and parse the new message which is the concatenated message ?
