Boolean conditions on pipeline rules conditions - Need help, not working

[aguida79 (Alejandro Guida)]

Hi everyone,

I need some help with this, because I think it is something really simple, but is not working,

1. Describe your incident:
I’m working with CEF format events received from and ArcSight Logger, coming in CEF Syslog inside a JSON string. The “path” of the events is ArcSight Logger Forwarder CEF Syslog → Fluentbit Syslog Input → Fluentbit Kafka Output → Graylog Kafka Input.
After I do the CEF parsing with a pipeline rule, I’m trying to do a second pipeline rule to replace the message field every time that message field (msg in CEF) is not present in the original event, because in that cases, the message field in Graylog is the complete JSON/CEF string obtained from Kafka, and I want to replace it with the parsed name field (when msg is not present in the original event, I want msg and name to have the same content).

So, I can have to scenarios:

• One, where both msg and name are present in the original event, and, after parsed with pipeline rules ends like this, for example, for a Fortigate event:
  message:
  progress IPsec phase 1
  name:
  event:vpn negotiate failure
  In this case I don’t want to do anything, is ok

• Two, where msg is not present in the original event, and after parsed with pipeline rules ends like this, for example, for a Fortigate event:
  message:{"@timestamp":1647168508.203043,“log”:“CEF:0|Fortinet|Fortigate|v6.4.7|forward|traffic:forward accept|Low| eventId=291293317 externalId=2234144878 start=1647168505000 end=1647168505000 app=DNS proto=UDP etc etc etc”}
  name:
  traffic:forward accept
  In this case, I want to put the content of name field on message field

2. Describe your environment:

• OS Information:
  Ubuntu 20.04
• Package Version:
  Kubernetes, Kongz Helm Chart, using 4.2.7 version of Graylog

3. What steps have you already taken to try and solve the problem?
I tried to accomplish that with a simple pipeline rule:

rule "Regla Message"
when
  is_json($message.message)
then
  set_field("message", (to_string($message.name)));
end

I also tried as a condition on “when”

• is_json($message.message) == true
• starts_with(to_string($message.message), “{”)
• contains(to_string($message.message), “CEF”)

No option is working, I never get replaced the message field with the name field when the condition is true.

4. How can the community help?
Give me a piece of advice of what I’m doing wrong.

Thanks in advance,
Alejandro

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[aguida79 (Alejandro Guida)]

It is really strange, because the boolean condition or catch everything or catch nothing, but never only the true condition.

For example, I changed the rule to this one:

rule "Rule Message Crop if is RAW CEF"
when
  starts_with(to_string($message.message), "{")
then
  set_field("original_message", (to_string($message.message)));
  set_field("message", (to_string($message.name)));
end

And appear that the condition is true every time (replace every time message field with name field, and add the field original_message, even if the message field not start with “{”), when is not, as you can see:

Other strange thing is that I tested a similar scenario in other Graylog 4.2.7 deployment (but this time, Docker deployment, not Kubernetes with Kongz helm chart), and is working perfectly, in this case with PA firewall events:

The rule:

rule "Rule Pipeline - Message PA Replace"
when
  starts_with(to_string($message.message), "PAN-OS: [url")
then
  set_field("original_message", (to_string($message.message)));
  set_field("message", (to_string($message.name)));
end

The result, working perfectly:

Maybe the problem is that “{” is a special character, and you cannot evaluate with the condition starts_with? But even with that, for example the condition that I mentioned earlier, contains(to_string($message.message), “CEF”), need to work without any problems…

I don’t know, Im really confused,

[tmacgbay]

Check your Message Processor Configuration , it seems to be catching people as of late: see here. You may want your Message Filter Chain before your Pipeline.

[aguida79 (Alejandro Guida)]

Hi @tmacgbay , thanks for your answer.

My Message Filter Chain is already before my Pipeline Processor:

Saying “catching people as of late”, you are saying that Graylog is processing messages from before the change, and I need to wait for a while to see if my rule is working?

[aguida79 (Alejandro Guida)]

Also, my Graylog Buffers are empty most of the time:

[aguida79 (Alejandro Guida)]

@tmacgbay , additionaly, after I read the post that you told me, I reviewed the “path” of the events on Graylog again, and I think that it cannot be about the log is not “reaching” the pipeline rule, because the logs are entering in the correct stream, and in the same stage of the pipeline, Stage 0, I have two rules, and the first is working right (the first one is the one that do the CEF parsing):

[tmacgbay]

@aguida79

Maybe the problem is that “{” is a special character, and you cannot evaluate with the condition starts_with

I agree, there is likely a “{” at the start of all $message.message that we never see based on what you are seeing.

Can you post what the message looks like coming in? I don’t think CEF inputs and JSON messages mix well but I haven’t used either… You can use the debug() function in your rule to see what is going on in the rule use

tail -f /var/log/graylog-server/server.log

to view the results of debug()

[aguida79 (Alejandro Guida)]

Thanks again for your response @tmacgbay

I put the Debug in the rule:

rule "Rule Message CEF"
when
  starts_with(to_string($message.message), "{")
then
  let debug_message = concat("Debug Message: ", to_string($message.message));
  debug(debug_message);
end

An review the server.log:

The only thing that got my attention is that in the server.log file, the log, when is JSON with CEF inside, is not complete, is not ending with “}”, but I think it is only “cosmetic”, because in Graylog Interface, I see the full message:

You can view that messages like “SSL Alert received” don’t start with “{”, and is being catched by the rule, because you have the Debug Message on logs… Additionally in that situation I don’t know why the Debug Message has " - {}" at the end…

So I really don’t know what is happening, because is like the condition in the pipeline rule, which is really a simple boolean condition, catch everything or catch nothing,

[aguida79 (Alejandro Guida)]

@tmacgbay , I think the basic problem is with the functions on this JSON/CEF messages, because when I do a simple test on other events, work perfectly. For example:

A pipeline rule to change all the Windows Logoff Message, to simply “LOGOFF”:

rule "Test replace Windows logoff"
when
    starts_with(to_string($message.message), "An account was logged off")
then
    set_field("message", "LOGOFF");
end

Message correctly replaced only in the Logoff situation:

I think that I’m going to try to change my collection option (Fluentbit), to get instead of the CEF inside the JSON, only the CEF.

Thanks for all your help @tmacgbay

[tmacgbay]

Let me know if that works!