Simple Pipeline Rule

[nateynate (Nathan Robbins)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Trying to work towards an enrichment via a lookup table. However I cannot seem to get a pipeline rule to work at all.

I have a working system, with a stream, and extractors running on the messages to parse the messages into fields… These are syslog-like messages.

2. Describe your environment:

• OS Information:
  Ubuntu 22.04

• Package Version:
  Graylog 5.0.8+4c22532 on sparc-log01 (Eclipse Adoptium 17.0.6 on Linux 5.4.0-150-generic)

• Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
I followed the documentation to create a lookup table, and connect / create a pipeline and a rule .

I can use the test in the lookup table and get desired results.

Basically what I am trying to do, is take a field, field_a that is already present in a log message (created by an extractor), and use it as the key for a lookup table, to then set a new field in the log that contains associated data (the value from the key->value lookup table query).

However I cannot even get the pipeline rule to work… I created the following rule as a test:

rule “Add MemberSEP”
when
1
then
set_field(“membersep”, “unset”);
end

This does not work. I even tried using the debug() function to see if I could get more info but could not get any log messages into the server log.

The real rule would look like this:

rule “Add MemberSEP”
when
has_field(“calix_severity”)
then
** lookup code to add new field with value **
end

I would think the test rule should create the new field in the log message and set it to a value of “unset”

I am stumped,… have scoured the forums and web tyring to find a clue…

4. How can the community help?

Any advice or direction is appreciated.

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[patrickmann]

Sounds like pipeline is running before extractors. You would need to change the ordering of processing stages so pipeline runs after message filter.

[nateynate (Nathan Robbins)]

thought I had done that… This is how its currently configured.

[nateynate (Nathan Robbins)]

Also, I do need the extractors to run first, as they are creating fields in the logs that I need to use to process in the pipeline.

[drewmiranda-gl (Drew Miranda)]

It looks like your processing order is correct. ‘Message Filter Chain’ is the processor that handles static fields and extractors. However, it looks like you have stream rules AFTER pipeline processor. This means that if you are using stream rules to route messages, the pipeline processor will have zero knowledge of what stream the message is in.

You can address this by either using pipelines to route messages to streams, or change the order.

For your test pipeline, you can use true for a condition that is always matched:

rule "test rule"
when
    true
then
    set_field("test_field", "hello world");
end

The pipeline simulator may also be useful:

1. copy any message from the search page using the ‘copy message’ button
2. go to System / Pipelines / Simulator
3. Paste the copied message json
4. Choose stream to suit your needs
5. set message code to gelf

Hope that helps.

[nateynate (Nathan Robbins)]

I am going to try this, I am indeed using stream rules.

I will change the order and test again, as well as investigate changing to using pipelines to route. Any side-effects I need to watch out for when changing the from stream rules to pipelines?

[nateynate (Nathan Robbins)]

Alright, I changed the order, and used the simulator to check a message…

These are the results.

27 μs
Starting message processing
5,110 μs
Message 7bd11900-3579-11ee-b3bf-3a1313377832 running [Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c)] for streams [620c50407ad02700aed067e5]
6,720 μs
Enter Stage 10
6,926 μs
Evaluate Rule ‘Add MemberSEP’ (64cb1f6d214be969cc9e586a) in Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c)
7,116 μs
Evaluation satisfied Rule ‘Add MemberSEP’ (64cb1f6d214be969cc9e586a) in Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c)
7,298 μs
Execute Rule ‘Add MemberSEP’ (64cb1f6d214be969cc9e586a) in Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c)
7,607 μs
Finished execution Rule ‘Add MemberSEP’ (64cb1f6d214be969cc9e586a) in Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c)
7,839 μs
Completed Stage 10 for Pipeline ‘Calix_UPN_Enrichment’ (64cb19be214be969cc9e4c1c), continuing to next stage
7,983 μs
Exit Stage 10
8,056 μs
Finished message processing

However, I do not see that the log entry was modified to include the new field. Would I not see this in a search? Ultimately, the goal here is to use this modified version of the message for a notification sent from Alerts and Events.

This is my pipeline rule: (basically following this simple example: How to Use Graylog Lookup Tables )

rule "Add MemberSEP"
when
    has_field("ont-id")
then
    let sep_source = lookup_value("AccountName", $message."ont-id");
    set_field("membersep", sep_source);
end

I quote the ont-id part of the field, because i get errors when trying to use it otherwise. That could be some of the issue, but changing the defined name of that field is not feasible in our scenario.

[nateynate (Nathan Robbins)]

And nothing I try, (escaping the - or quoting around it, can make it work.

How do you deal with special characters in a field name? I can find all kinds of things for GROK patterns, etc. But none of that works in this case…

The - is fine the the has_field() function, but doesn’t work in the others… ie. in $message.ont-id or sep_source.ont-id

rule "Add MemberSEP"
when
    has_field("ont-id")
then
    let sep_source = lookup_value("smx_sep_lookup_table", to_string($message.ont-id));
    set_field("membersep", sep_source.ont-id);
end

[nateynate (Nathan Robbins)]

I resolved the issue…

It turns out there was some simple errors… The lookup table name was wrong for one…

Corrected that, and then the quotes around the field name worked.

ie:

rule "Add MemberSEP"
when
    has_field("ont-id")
then
    let sep_source = lookup_value("smx_sep_table", $message."ont-id");
    set_field("membersep", sep_source);
end

is working. Sorry for the confusion, and thanks all who helped.