Pipeline rules for messages splitted in 2 or more parts

1. Describe your incident:

I’m working with pipelines to parse security events from VSphere 8.X, it works out well but i’m encountering an issue and i’m stuck.

Some messages are split in two because the size of the message is more than 1024 characters

As you can see below, my pipeline rule can not parse the two splitted messages:

It look like this:

  • Message 1/2
Event [2528773] [1-2] [2023-10-31T14:39:02.14294Z] [vim.event.VmReconfiguredEvent] [info] [LAB\adm.user] [Datacenter] [2528772] [Reconfigured o8-test1 on in Datacenter.
  • Message 2/2
Event [2528773] [2-2] , level = "normal"), limit = -1), externalId = <unset>, uptCompatibilityEnabled = true, uptv2Enabled = false);

The default message format when size is inferior to 1024 character look like this:

Event [2528772] [1-1] [2023-10-31T14:38:59.283901Z] [vim.event.TaskEvent] [info] [LAB\adm.user] [Datacenter] [2528772] [Task: Reconfigure virtual machine]

My pipeline rule is this one:

rule "Syslog - VMWARE VCSA"
  has_field("message") AND regex("Event \\[", to_string($message.message)).matches == true
  let msg = to_string($message.message);
  let vcsa1 = grok(pattern: "Event \\[%{BASE10NUM:eventId}\\] \\[%{DATA:partInfo}\\] \\[%{TIMESTAMP_ISO8601:createdTime}\\] \\[vim.event.%{DATA:eventType}\\] \\[%{DATA:severity}\\] \\[%{DATA:user}\\] \\[%{DATA:target}\\] \\[%{DATA:chainId}\\] \\[%{DATA:desc}\\]", value: to_string(msg), only_named_captures: true);


This rule is not working with the two splitted message because the last field does not have the bracket closing as it is is splitted in another message.

2. Describe your environment:

Graylog 5.1.7
opensearch 2.9.0

3. What steps have you already taken to try and solve the problem?

Searching for pipeline function without success.

4. How can the community help?

Could you help me on this issue, meaning adapt my existing pipeline rule to parse the two or more splitted message, and create the corresponding fields like if it was 1 message.

As graylog does not have the vmware limitation for character size, could I concatenate the two message, remove the two splitted, and parse the new message which is the concatenated message ?

Hey @s0p4L1N

What type of input are you using?
Normally a log shipper can handle multi-line logs like filebeat, etc…

Then send it to Graylog.

I send the log directly without an agent, on Port 1515/TCP on VCenter directly cause VCenter OS is VMWare Photon OS, it is not installed on Windows Server nor Linux.

And Syslog input TCP:

allow_override_date: true
charset_name: UTF-8
expand_structured_data: false
force_rdns: false
max_message_size: 2097152
number_worker_threads: 4
override_source: <empty>
port: 1515
recv_buffer_size: 1048576
store_full_message: false
tcp_keepalive: false
timezone: NotSet
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
use_null_delimiter: false
Throughput / Metrics
1 minute average rate: 10 msg/s
Network IO: 3.0KiB 0B (total: 1.8GiB 0B )
Active connections: 1 (143 total)
Empty messages discarded: 14

Found this discussion:

2 years later and still multiline not supported natively ? Or maybe there is a setting that enable multiline reading ?

Looks like @d_a_parker is having a very similar issue…

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.