Issue with Grok patterns failing on multi-line logs from automated execution scripts?

[enola90 (Erwin Kaufmann)]

Hi everyone,
I’m currently trying to set up a pipeline to monitor some security events coming from a local testing environment, but I’m running into a consistent parsing error. I’ve been using Graylog to ingest logs from various automation tools to see if I can flag unauthorized API calls or suspicious execution behavior.

Specifically, I’m seeing an issue where my Grok patterns fail whenever the log contains nested JSON or multi-line strings. I’ve been stress-testing my setup using several blox fruit scripts to see how Graylog handles the rapid-fire logs generated by game executors, but the messages keep getting truncated or shoved into the unmapped_fields bucket. It seems like the default extractors are having a hard time with the way these scripts output their bypass logic.

Does anyone have a recommended pipeline rule or a specific regex for handling high-velocity, non-standard logs? I’m worried that if I can’t get the parsing right for these gaming scripts, I’m going to have even bigger issues when I try to scale this up for my actual production environment. Should I be looking into using a RAW/Plaintext input with a custom extractor instead of GELF, or is there a way to optimize the buffer so I don’t lose data during these high-activity bursts? I’d really appreciate it if anyone has a “best practice” guide for ingesting script-heavy logs without crashing the indices!

[Wine_Merchant]

Hey @enola90,

I suggest you carry out parsing within pipelines/rules as extractors are no longer supported and offer less functionality.

Without an example it’s hard to say how to approach parsing these logs. If they are valid JSON then I would suggest flattening them in a pipeline then mapping the json to fields. The issue is that this would be blunt and can crate a very large amount of fields. A more measured approached of multiple streams/pipelines/rules to better organise and parse/normalise in a way that will be useful to you is the best route.