Good morning everyone.
I am having a weird issue regarding pipeline rules and how they behave with my inputs, even after reading a lot of documentation and forum posts I have no clue what the problem is.
Hopefully my post isn’t too dense, as I try to give all available information.
So, I have a JSON message entering Graylog through a Raw/plaintext TCP input, with an associated extractor. This parses my json message just fine, including nested fields. Example:
This goes into a stream called “Azure Devops”, which is correctly configured, I can see all the data in Graylog, including all the extracted fields. Example:
However, I also need to create an additional field (timeToBuild), which is an operation between two fields (more specifically, I want to calculate time difference as epoch between 2 timestamps in the input json: timeToBuild = resource_finishTime - resource_startTime). So I created a Pipeline and a Rule, associated this rule to the pipeline, and connected the pipeline to my “Azure Devops” stream. Example:
My “Message Processing Configuration” is also in the correct order, with “Message Filter Chain” before “Pipeline Processor”. Example:
However, it’s not doing anything, as this field is not showing up in my Graylog results. Also, when I try to simulate my pipeline (using my configured input, and the expected raw json message), the entire raw message is contined in the “message” field, as if my extractor didn’t work. Since everything is in a bulk inside the “message” field, my pipeline rule stage 0 condition fails to find any data and stops processing. As a result, my new “timeToBuild” field is not showing up in Graylog. Example:
I greatly appreciate any help.
Thank you very much!