Extractor fields not appearing when simulating pipeline

[ylmcc (Kyle)]

Hi Guys,

I am quite puzzled with using pipelines, I have created an udp raw input with extractors which sends all messages into “All Messages” with the fields from my extractors added.

When it comes to simulating the pipeline I am unable to see my extractors working on the simulated message. Under “Message Input (optional)” I have selected the input which the message would appear on for codec configuration I have selected “Raw String” would this affect it?

The order my messages are processed are as follows:

Thanks

Edit: Current version of Graylog 3.2.4

[shoothub]

Pipeline simulator can’t process all pipeline rules. I prefer to use debug pipeline function, to check if it works.
https://docs.graylog.org/en/3.2/pages/pipelines/functions.html#debug

let debug_message = concat("Test message from ", to_string($message.source));
debug(debug_message);

After that check your server log in log file /var/log/graylog-server/server.log
using for example tail from command line of graylog server:

tail -f /var/log/graylog-server/server.log

[ylmcc (Kyle)]

I’ll give a go and see what happens. Thanks !

[andrenandes]

Good afternoon.

I am taking advantage of this recently opened thread to look for help, as I have a very similar issue.

I have a JSON message entering Graylog through a Raw/plaintext TCP input, with an associated extractor. This parses my json message just fine, including nested fields. Example:

This goes into a stream called “Azure Devops”, which is correctly configured, I can see all the data in Graylog, including all the extracted fields. Example:

However, I also need to create an additional field (timeToBuild), which is an operation between two fields (more specifically, I want to calculate time difference as epoch between 2 timestamps in the input json: timeToBuild = resource_finishTime - resource_startTime). So I created a Pipeline and a Rule, associated this rule to the pipeline, and connected the pipeline to my “Azure Devops” stream. Example:

My “Message Processing Configuration” is also in the correct order, with “Message Filter Chain” before “Pipeline Processor”. Example:

However, when I try to simulate my pipeline (using my configured input, and the expected raw json message), the entire raw message is contined in the “message” field, as if my extractor didn’t work. Since everything is in a bulk inside the “message” field, my pipeline rule stage 0 condition fails to find any data and stops processing. As a result, my new “timeToBuild” field is not showing up in Graylog. Example:

I guess I am doing something wrong, but after reading a lot of documentation and forum posts I have no idea what. I greatly appreciate any help.

Thank you!

[ylmcc (Kyle)]

This worked for me, I see somewhere it was not possible to drop the “message” field but could be removed in the future.

Thank you shoothub for the debug tip!

[ylmcc (Kyle)]

Posting in a new thread would make this easier to read. However you should use debug(variable_you_are_trying_to_view) and as shoothub suggested

tail -f /var/log/graylog-server/server.log````

[andrenandes]

Good morning @ylmcc

Thank you for your reply.
I will follow your advice and make a new post for this, as it’s probably better suited.

I have also added the debug message as for your suggestion, but so far it hasn’t yielded any useful messages yet (messages have arrived in the meanwhile, and show on Graylog, but my debug message didn’t show up in the server.log file).

Thank you.

[jan (Jan Doberstein)]

he all

you might need to know that the simulator need the message ingested as this comes in to graylog - so that all extractors and every can actually run on that.