Messages not affected by pipeline

[mackan (Markus Faleij)]

Hi,

I’m new to Graylog using version 4.0.8.

I have filebeat running on windows servers fetching logs from console applications writing to disk. The log-files are formatted differently with some using JSON.

There is an input extractor in Graylog parsing JSON messages.

When the messages are processed by a stream I want a pipeline to parse a timestamp from the message row and replace the field timestamp generated by filebeat/graylog.

I am running Graylog in docker.

I have created a pipeline connected to a stream and verified the rule running a simulation with the configuration below. I have added a debug function within the rule.

However, when the messages go through the stream they are not affected by the pipeline/rule.

When I check the logs from the docker container with the command “sudo docker logs 8587d1bee86a -f” I cannot see any error or debug information releated the pipeline.

Can you see anything wrong with the configuration?

Best regards,
Markus

MESSAGE PROCESSORS CONFIGURATION
1 Message Filter Chain active
2 Pipeline Processor active
3 AWS Instance Name Lookup disabled
4 GeoIP Resolver disabled

PIPELINE SIMULATION
Raw message
{“LogVersion”:2,“Timestamp”:“2021-07-08T10:02:56.104”,“Level”:“INFO”,“Message”:“Startar uppdatering”,“Identity”:“ad55c13f-065f-4d1f-ba61-bb399edce3b5”,“ThreadId”:5416}

Message input (optional)
** /Input from Filebeat / Beats

Message codec
Raw String

PIPELINE RULE
rule “prase date”
when
has_field(“message”)
then
let new_date = flex_parse_date(to_string($message.message));
set_field(“timestamp”, new_date);
debug(to_string(new_date));
end

[tmacgbay]

flex_parse_date() requires a date and time value, you can’t inject the entire message.

In this case you could use the key_value() function to pull out the fields,

then:

let new_date = flex_parse_date(to_string($message.Timestamp));

also:

Its easier to spot in the logs with a little text before the results you are looking for:

let debug_message = concat("Results for new_date: ", to_string(new_date));
debug(debug_message);