Set field my pipeline don't work

[peaile (CéPL)]

1. Describe your incident:
I receive logs from my firewall which follow this pattern :

{
  "gl2_accounted_message_size": 740,
  "gl2_receive_timestamp": "2025-02-21 15:40:01.265",
  "level": 6,
  "gl2_remote_ip": "168.63.121.79",
  "gl2_remote_port": 10089,
  "streams": [
    "67b7510e5c826156f87a81a3"
  ],
  "gl2_message_id": "01JMMJMT400001WPJBKCEZGQQX",
  "source": "AZIS-FW01",
  "message": "id=firewall time=\"2025-02-21 16:40:00\" fw=\"AZIS-FW01\" tz=+0100 startime=\"2025-02-21 16:37:30\" pri=5 confid=01 slotlevel=2 ruleid=105 rulename=\"16c0465cf5b_1f\" srcif=\"Ethernet1\" srcifname=\"in_vm\" ipproto=udp dstif=\"Ethernet0\" dstifname=\"out\" proto=dns_udp src=10.0.195.8 srcport=58244 srcportname=ephemeral_fw_udp srcname=ISDC08 srcmac=c0:d6:82:e9:9a:dd dst=8.8.8.8 dstport=53 dstportname=dns_udp dstname=dns.google.com dstcontinent=\"na\" dstcountry=\"us\" modsrc=10.0.194.254 modsrcport=54729 origdst=8.8.8.8 origdstport=53 ipv=4 sent=50 rcvd=82 duration=0.01 action=pass logtype=\"connection\"",
  "gl2_source_input": "67b442697ac2da6b406c1c8e",
  "gl2_processing_timestamp": "2025-02-21 15:40:01.265",
  "application_name": "asqd",
  "facility_num": 1,
  "gl2_source_node": "50b2695e-e2c4-438e-8dfb-efcee33a727b",
  "_id": "1cd44e14-f06a-11ef-9bf3-6045bd6bce7e",
  "facility": "user-level",
  "gl2_processing_duration_ms": 0,
  "timestamp": "2025-02-21T15:40:00.000Z"
}

As you can see, most of the interesting information are in the message field.
My goal is to extract some informations like the IP addresss or the port
However, while my rule seems to work (u will have a screenshot in the 3rd section) when I open a message in my stream, the new field doesn’t appearc

2. Describe your environment:

• OS Information: Debian 12

• Package Version:
  Graylog 6.1
  OpenSearch 2.19

• Service logs, configurations, and environment variables:
  Here’s my stream
  
  Here’s my pipeline
  

  
  
  This is the only rule that I have in my pipeline :

rule "Extract Destination Country"
when
    has_field("message") AND contains(to_string($message.message), "dstcountry=")
then
    let lePays = grok(pattern: "dstcountry=\"%{WORD:dstcountry}\"", value: to_string($message.message));
    
    set_fields(lePays);
end

As you can see, it seems to work when I test it with a message, as it create the field dstcountry

Also, the rule is correctly binded with the pipeline

3. What steps have you already taken to try and solve the problem?
I’ve checked the connexion between streams, pipeline and the rules
I’ve also restarted the server but nothing change

[Joel_Duffield]

I wonder if something is not happy in your when statement of your rule. Try and set it to

when
true

and it will always run, and see if then it processes any messages.

You can also try the pipeline simulator, it will tell you if the rule is matching or not.

[peaile (CéPL)]

Hey Joel
Sadly no it changed nothing

However we see that the pipeline simulator work as it add the field.

I think I will try to restart from zero and recreate the stream, pipeline and rule

[Joel_Duffield]

If you click “more results” in the top right of your screen shot and cycle through the options there may be some clues in there when it shows you all the steps it took.

[Joel_Duffield]

You may also want to use the “debug” function in your pipeline rule. The problem with all the simulators is that they go off the message that was already stored and sometimes the data gets slightly changed when its stored. For example fields with dots in the name are automatically converted to underscores, and some other things. So sometimes the way that the pipeline rule actually “sees” the data when its running for real is different.

You could also try and use the key value function rather than grok to get your value.

[peaile (CéPL)]

I’ve used debug function and as we can see in the picture, it send a log to my server. However, this is send only when I run a rule simulation

Also, I tried to delete my stream where the logs were redirected. So now it’s received by the default stream and it work

As I do not want to send all the logs to the same stream, I will search in this area.

[peaile (CéPL)]

I noticed that I didn’t include the message processors configuration
I think it is correct but maybe i forgor an element

[peaile (CéPL)]

I restarted from zero and recreated all the components and now it work so I dont’t really know what was the issue