`pipeline_output` field is missing or blank. Message will be dropped

[aali94 (Ifty)]

Post your pipeline

rules

---

rule “Route Security logs from privatecloud Stream to Staging Stream”
when
contains(to_string($message.message), “hrc-vas-bms-tdb01”)
OR contains(to_string($message.message), “PROC_Limits”)
then
set_field(“pipeline_output”,(to_string($message.message)));
end

---

Error : "pipeline_output field is missing or blank. Message will be dropped"
endhere with comments or how-to documentation.

Dear All,

Getting this error while implementing the above pipeline rule.
The purpose of this rule is to catch the message only if keywords in the pipeline rule matches.

Thanks,

Ifty.

[gsmith]

Hello @aali94 && Welcome

I might be able to help, I see the rule is not completed, need "end’ at the bottom.

Next, Check Message Processors Configuration this is located under System/Configurations → Message Processors Configuration.
The Pipeline Processor should be after Message Filter Chain.

[aali94 (Ifty)]

Hey

Thanks for your response, I missed to add the complete rule while pasting the same in this thread. Rule is working but stops after 3 - 4 minutes. Please have a look at the error message.

Ifty

[gsmith]

Have you tried to u se the debug() function in your pipeline and watch the graylog logs to see what is going on?

[aali94 (Ifty)]

Hey,

The intention is to catch only specific keyword using the pipeline, But as there are many message which doesn’t contain the keyword which pipeline rule is searching. The message which doesn’t contain the keyword defined in pipeline is getting a null value which is populating the below error message

ERROR [PipelineMessageTransformer] Message 5d6cb518-db98-11ec-95d5-005056010db9 pipeline_output field is missing or blank. Message will be dropped.

Just wanted to know how can we add some condition in my pipeline rule to exclude these null value.

Pipeline Rule :

---

rule “Route Security logs from privatecloud Stream to Staging Stream”
when
contains(to_string($message.message), “hrc-vas-bms-tdb01”)
OR contains(to_string($message.message), “PROC_Limits”)
then
set_field(“pipeline_output”,(to_string($message.message)));
end

---

Thank You,

Ifty

[gsmith]

Hello

Have you tired to split them up by creating two rules and attaching them to one pipeline
Example:

rule "Node01"
when
contains(to_string($message.message), "hrc-vas-bms-tdb01")
then
set_field("node01","hrc-vas-bms-tdb01");
end

rule "Node02"
when
contains(to_string($message.message), "PROC_Limits")
then
set_field("node02","PROC_Limits");
end

Something like this.

Or something like this

[aali94 (Ifty)]

Hey @gsmith,

Thanks for your response.

Unfortunately splitting condition is also not working.

Say we have 1000 message getting processed in this pipeline and we have 100 message which contain the keywords which is matching with our condition and the same message is getting filtered. However the remaining 900 messages which doesn’t have the keywords as per the pipeline rule is also getting processed and pipeline assumes these as null and write error in the Graylog app logs.

Just trying to have something like ifelse condition which will discard the remaining 900 messages which doesn’t match the keywords which we are defining in the pipeline.

Thank You,

Ifty.

[gsmith]

@aali94
I just came across this, perhaps it might help

• Pipeline Rule If/else Trick

EDIT:
In my lab I used this. Note I set it to true

rule "Node01"
when
contains(to_string($message.message), "hrc-vas-bms-tdb01",true)
then
set_field("node01","hrc-vas-bms-tdb01");
end

I found how to do this here.

https://archivedocs.graylog.org/en/3.0/pages/pipelines/functions.html#contains

Hope that helps