Pipeline problems

[jendamozna]

Hello,
ive just started with graylog and dont understand the pipelines concept.

My streams:

1. All messages - default index set
2. FW messages - dedicated FW index set, Remove matches from ‘All messages’ stream

Pipelines:

1. “Add recieved timestamp”, connected to “All messages stream” with 1 rule:

rule "set receivedat with now"
when
  true
then
  set_field("receivedat", now());
end

2. “FW cleanup”, connected to “FW messages stream” with two rules:

rule "empty message"
when
  has_field("message")
then
  set_field("message", "");
end
AND
rule "remove level field"
when
  has_field("level")
then
  remove_field("level");
end

Config:
Graylog 2.4.3+2c41897

Order	Processor
1	Pipeline Processor
2	GeoIP Resolver
3	Message Filter Chain

Problems

1. receivedat field is applied also on “FW messages” stream. Im expecting no receivedat on FW messages.
2. No throughput and changes on “FW cleanup”. Simulation reveals only level field change. In fact FW logs returns empty “NOT exists:level” query and message field is still there.

Any ideas?

[jochen]

The assignment of a message to one or many streams happens in the StreamFilter which is part of the “Message Filter Chain”.

So when your pipeline rules are running, the messages haven’t been assigned to any stream yet.

You can either change the order of message processors in your Graylog cluster or explicitly use the route_to_stream() function in a pipeline rule to assign a message to a stream.

[jendamozna]

Processors order changed to:
|1|Message Filter Chain|active|
|2|Pipeline Processor|active|
|3|GeoIP Resolver|disabled|

Problem 1 now solved, receivedat applied only to “All messages stream”. Thank you!

Problem 2 still here. Field “level” is being removed, but “message” field is not emptied. Simulation still reveals only “level” field change.

[jochen]

The “message” field is mandatory and must not be empty or blank (contain only whitespace characters).

[jendamozna]

Thank you jochen, both issues are solved.

This helped:

rule "empty message"
when
  has_field("message")
then
  set_field("message", "empty");
end