1. Describe your incident:
I have configured a Lookup Table and created a rule to take a value from a field and return the corresponding value in the Lookup Table. I’ve tested the Table, I’ve tested the rule; both return the expected result. However, when I apply the rule to a pipeline - which only includes this one rule - it returns either nothing or, if I have configured the Lookup Table to have a default value, the default value.
Rule:
rule "Tailscale User Lookup"
when
has_field(
field : "id_orig_h"
)
then
let gl2_fragment_extractor_1731100320269 = lookup_value("tailscale", to_string($message."id_orig_h"));
set_field("user_name", gl2_fragment_extractor_1731100320269);
let output_1 = gl2_fragment_extractor_1731100320269;
end
Two things come to mind:
Is the message processing order of your cluster correct that the value you think will be there is actually there when this rule runs?
Try and use the debug function in pipeline rule to write that value to_string($message."id_orig_h") into the server.log file, because it may not be exactly in the format you think it is at that point in its life. In the simulator you are testing with messages already stored, so if a value came in with a capital but was lowercased during storage for example your rule wouldn’t work in production, but would in testing.
Try and use the debug function in pipeline rule to write that value to_string($message."id_orig_h") into the server.log file, because it may not be exactly in the format you think it is at that point in its life. In the simulator you are testing with messages already stored, so if a value came in with a capital but was lowercased during storage for example your rule wouldn’t work in production, but would in testing.
I’ve never used this feature, can you broad-brush the process of writing to server.log?
Ultimately I went with a RegEx Input extractor to reproduce the field from the raw message. This has been successful.
I did figure out how to debug to the server log but that was surprisingly unhelpful. All it shows was that the Rule was pulling an empty value from a clearly populated field and then doing a lookup with that empty value.
The debug output is actually quite illuminating: Now you know that at the time of rule execution, the message field is empty.
You didn’t mention what type of input you were using. That may have simply not been extracting that field. Though then the when has_field() condition should have failed.
Syslog UDP which then has a JSON Extractor. Can’t say I understand why the rules would work except when placed within a Pipeline but its working now (with the additional RegEx extractor that follows the initial JSON Extractor) so I am not complaining haha.