Pipeline rule key/value with specific delimiter not working

[s0p4L1N]

1. Describe your incident:

I use winlogbeat to ingest Windows Defender Security Event to Graylog. The message field is not parsed correctly as it does not get all the data.

I would like to parse it with the key/value pipeline but it give me some weird result:

• The message looks like this:

{
  "tags": [
    "windowsdefender"
  ],
  "message": "Antivirus Microsoft Defender a détecté un logiciel malveillant ou potentiellement indésirable.\n Pour plus d’informations, reportez-vous aux éléments suivants :\nhttps://go.microsoft.com/fwlink/?linkid=37020&name=Virus:DOS/EICAR_Test_File&threatid=2147519003&enterprise=0\n \tNom : Virus:DOS/EICAR_Test_File\n \tID : 2147519003\n \tGravité : Grave\n \tCatégorie : Virus\n \tChemin : containerfile:_C:\\Users\\adm.user\\Downloads\\eicar_com.zip; file:_C:\\Users\\adm.user\\Downloads\\eicar_com.zip->eicar.com; webfile:_C:\\Users\\adm.user\\Downloads\\eicar_com.zip|https://secure.eicar.org/eicar_com.zip|pid:1352,ProcessStart:133385545470169081\n \tOrigine de la détection : Internet\n \tType de détection : Concret\n \tSource de détection : Téléchargements et pièces jointes\n \tUtilisateur : LAB\\adm.user\n \tNom du processus : Unknown\n \tVersion de la veille de sécurité : AV: 1.397.528.0, AS: 1.397.528.0, NIS: 1.397.528.0\n \tVersion du moteur : AM: 1.1.23080.2005, NIS: 1.1.23080.2005"
}

And i want to extract all the field starting from \n \tNom :

• Where \n \t is the delimiter,

• : is the separator

• as there is space before and after the separator, I tried many things but nothing work

• The pipeline rule:

when
 contains(to_string($message.tags),"windowsdefender")

then
set_fields(
		fields:
				key_value(
					value: to_string($message.message),
					trim_value_chars: " ",
					trim_key_chars:" ",
					delimiters:"\\n \\t",
					kv_delimiters:":"
					)
		);
end

Here’s the result…

(EventDescription comes from another pipeline)

The trace logs of the simulations shows no errors.

2. Describe your environment:

• OS Information: docker

• Package Version: Graylog 5.1.4 / Opensearch 2.9.0

3. What steps have you already taken to try and solve the problem?

I already try to set the space in the trim or in separator but does not change the problem.

4. How can the community help?

Is it possible to set return carriage and tabulation as separator for this case ?

[s0p4L1N]

I found a workaround with another pipelines function:

I used the replace function, by replacing:

• \n \t with || and : with =
• replacing with _ key name containing space
• replacing é by e

rule "Winlogbeat - W-Defender - Message Replace"

when
  contains(to_string($message.tags),"windowsdefender")
then


   let test6 = replace(to_string($message.message), "; ", ";");
   set_field("message", test6);

   let test0 = replace(to_string($message.message), "Antivi", "Information = Antivi");
   set_field("message", test0);

   let test4 = replace(to_string($message.message), ".\n", " || Detail_Info = ");
   set_field("message", test4);
   
   let test3 = replace(to_string($message.message), ":\nhtt", " || URL = htt");
   set_field("message", test3);
   
   
   let test = replace(to_string($message.message), "\n \t", " || ");
   set_field("message", test);
   
   let test2 = replace(to_string($message.message), " : ", "=");
   set_field("message", test2);
 
   
   let test9 = replace(to_string($message.message), "Origine de la détection", "Origine_de_la_detection");
   set_field("message", test9);
   
   let test10 = replace(to_string($message.message), "Type de détection", "Type_de_detection");
   set_field("message", test10);
   
   let test11 = replace(to_string($message.message), "Source de détection", "Source_de_detection");
   set_field("message", test11);
   
   let test12 = replace(to_string($message.message), "Nom du processus", "Nom_du_processus");
   set_field("message", test12);
   
   let test13 = replace(to_string($message.message), "Version de la veille de sécurité", "Version_de_la_veille_de_securite");
   set_field("message", test13);
   
   let test13 = replace(to_string($message.message), "Version du moteur", "Version_du_moteur");
   set_field("message", test13);
   
   
end

The message look like this after the first pipeline rule:

"message": "Information = Antivirus Microsoft Defender a detecte un logiciel malveillant ou potentiellement indesirable || Detail_Info = Pour plus d’informations, reportez-vous aux elements suivants || URL = https://go.microsoft.com/fwlink/?linkid=37020&name=Virus:DOS/EICAR_Test_File&threatid=2147519003&enterprise=0 || Nom = Virus:DOS/EICAR_Test_File || ID = 2147519003 || Gravite = Grave || Categorie = Virus || Chemin = containerfile:_C:\Users\adm.user\Downloads\eicar_com.zip;file:_C:\Users\adm.user\Downloads\eicar_com.zip->eicar.com;webfile:_C:\Users\adm.user\Downloads\eicar_com.zip|https://secure.eicar.org/eicar_com.zip|pid:6316,ProcessStart:133385640422091285 || Origine_de_la_detection = Internet || Type_de_detection = Concret || Source de detection = Telechargements et pièces jointes || Utilisateur = ISS\adm.user|| Nom_du_processus = Unknown || Version_de_la_veille_de_securite = AV: 1.397.528.0, AS: 1.397.528.0, NIS: 1.397.528.0 || Version_du_moteur = AM: 1.1.23080.2005, NIS: 1.1.23080.2005",

And with a second stage pipeline with key value function pipeline:

rule "W-DEFENDER Message Parser"

when
has_field("message")

then
set_fields(
		fields:
				key_value(
					value: to_string($message.message),
					trim_value_chars: "",
					trim_key_chars:"",
					delimiters:"||",
					kv_delimiters:"="
					)
		);
end

The results are here :

I wonder why Winlogbeat does not parses automatically the XML Event ?? Because it already contains all the key/value pairs !

`

[Joel_Duffield]

Your workaround is the correct approach, the key value function accepts multiple characters as delimiters and seperators, so it doesnt see /n it sees / or n. I had the same problem as you once and pulled my hair out for way to long until I found that out.