Graylog: pipeline rule not considering Input's extractor fields

[andrenandes]

Good morning everyone.

I am having a weird issue regarding pipeline rules and how they behave with my inputs, even after reading a lot of documentation and forum posts I have no clue what the problem is.
Hopefully my post isn’t too dense, as I try to give all available information.

So, I have a JSON message entering Graylog through a Raw/plaintext TCP input, with an associated extractor. This parses my json message just fine, including nested fields. Example:

This goes into a stream called “Azure Devops”, which is correctly configured, I can see all the data in Graylog, including all the extracted fields. Example:

However, I also need to create an additional field (timeToBuild), which is an operation between two fields (more specifically, I want to calculate time difference as epoch between 2 timestamps in the input json: timeToBuild = resource_finishTime - resource_startTime). So I created a Pipeline and a Rule, associated this rule to the pipeline, and connected the pipeline to my “Azure Devops” stream. Example:

My “Message Processing Configuration” is also in the correct order, with “Message Filter Chain” before “Pipeline Processor”. Example:

However, it’s not doing anything, as this field is not showing up in my Graylog results. Also, when I try to simulate my pipeline (using my configured input, and the expected raw json message), the entire raw message is contined in the “message” field, as if my extractor didn’t work. Since everything is in a bulk inside the “message” field, my pipeline rule stage 0 condition fails to find any data and stops processing. As a result, my new “timeToBuild” field is not showing up in Graylog. Example:

I greatly appreciate any help.

Thank you very much!

[andrenandes]

Good morning.

I managed to fix this with a sort of workaround. I leave my solution as it might help someone.

I added 2 additional extractors to my Input, of regex type:
On the first I regex extract resource_startTime and store it on a new field.
On the second I regex extract resource_finishTime and store it on another new field.
The third is my original extractor.

These 2 new fields are then available for me to use on the pipeline rule, and I do the time difference calculation with them (this calculation was also wrong on my previous post, by the way). The route_to_stream also seemed to be unnecessary so I removed it.