Lookup Table returns proper result; Rule returns proper result; Pipeline using Rule does not produce the proper result

Graylog Central (peer support)
1

1. Describe your incident:
I have configured a Lookup Table and created a rule to take a value from a field and return the corresponding value in the Lookup Table. I’ve tested the Table, I’ve tested the rule; both return the expected result. However, when I apply the rule to a pipeline - which only includes this one rule - it returns either nothing or, if I have configured the Lookup Table to have a default value, the default value.

Rule:

rule "Tailscale User Lookup"
when
  has_field(
    field : "id_orig_h"
  )
then
  let gl2_fragment_extractor_1731100320269 = lookup_value("tailscale", to_string($message."id_orig_h"));
set_field("user_name", gl2_fragment_extractor_1731100320269);
  let output_1 = gl2_fragment_extractor_1731100320269;
end

2. Describe your environment:

  • OS Information:
    Debian 12
  • Package Version:
    Graylog 6.1.4+7528370

Any suggestions? Thank you!

2

Two things come to mind:
Is the message processing order of your cluster correct that the value you think will be there is actually there when this rule runs?
Try and use the debug function in pipeline rule to write that value to_string($message."id_orig_h") into the server.log file, because it may not be exactly in the format you think it is at that point in its life. In the simulator you are testing with messages already stored, so if a value came in with a capital but was lowercased during storage for example your rule wouldn’t work in production, but would in testing.

3

I believe so:

|#|Processor|Status|
| --- | --- | --- |
|1|Message Filter Chain|active|
|2|Stream Rule Processor|active|
|3|Pipeline Processor|active|
|4|GeoIP Resolver|disabled|
|5|AWS Instance Name Lookup|disabled|

Try and use the debug function in pipeline rule to write that value to_string($message."id_orig_h") into the server.log file, because it may not be exactly in the format you think it is at that point in its life. In the simulator you are testing with messages already stored, so if a value came in with a capital but was lowercased during storage for example your rule wouldn’t work in production, but would in testing.

I’ve never used this feature, can you broad-brush the process of writing to server.log?

4

The quotes around the field name seem wrong. Try removing those.

5

The quotes around the field name seem wrong. Try removing those.

Unfortunately that did not resolve the issue.

6

Ultimately I went with a RegEx Input extractor to reproduce the field from the raw message. This has been successful.

I did figure out how to debug to the server log but that was surprisingly unhelpful. All it shows was that the Rule was pulling an empty value from a clearly populated field and then doing a lookup with that empty value.

1 Like
7

The debug output is actually quite illuminating: Now you know that at the time of rule execution, the message field is empty.
You didn’t mention what type of input you were using. That may have simply not been extracting that field. Though then the when has_field() condition should have failed.

1 Like
8

Syslog UDP which then has a JSON Extractor. Can’t say I understand why the rules would work except when placed within a Pipeline but its working now (with the additional RegEx extractor that follows the initial JSON Extractor) so I am not complaining haha.

Thank you for your help!