Shipping multiline logs to Graylog

[H2Cyber]

I have an application that produces logs in the following format :

[#|2022-08-31T13:23:51.641+0100|INFO|glassfish3.1.2|redacted|_ThreadID=44;_ThreadName=Thread-2;|DEBUG SomeRandomText
SomeOthorRandomTextinAnotherLine
|#]

[#|2022-08-31T13:23:51.650+0100|INFO|glassfish3.1.2|redacted|_ThreadID=33;_ThreadName=Thread-2;|INFO  RandomText
|#]


[#|2022-08-31T13:23:51.654+0100|INFO|glassfish3.1.2|redacted|_ThreadID=50;_ThreadName=Thread-2;|INFO  AnotherRandomText
AnotherRandomTextinASeperateLine
AnotherRandomTextinAnotherSeperateLine
AnotherRandomTextinAThirdSeperateLine
|#]

As you can see in the three messages above, a single log message starts with [#| and ends with |#]. The single log can spawn multiple lines, and the number of lines varies from one log to another.

Did anyone experience with shipping similar multi-line logs to Graylog ? What method would you recommend in this case to send the logs to Graylog ?

[tmacgbay]

How is the message being shipped? If it’s filebeat, you can likely use the multi-line message feature to define beginning and end of each message - I am sure NXlog has something similar but I don’t use that. If you don’t have control on shipping, what input are you using? You could likely break that up with a pipeline rule.

[H2Cyber]

The messages are not being shipped just yet, as I am still figuring that part out.

Given the constraints that I have (which I cannot disclose) Syslog is preferable, but if it has to be Filebeat then I’ll use that. I am open to suggestions or recommendations should you have other ideas

[gsmith]

Hello,
You can use Nxlog, most preferred FileBeat. Personally I use both.

Parsing multi-line logs :: NXLog Documentation