Hi All,
I have tried out many options to create a pipeline and write rules to stop indexing the data for the non look up table values
To Do: we have a look up table for 10 IP Addresses for corresponding 4 regions. but we are getting 100 IP Address from source. I need to drop the 90 non-look up IP Addresses and need to allow only 10 IP Addresses for indexing.
What I have done:
rule “look up IP Address”
when
is_null(lookup_value(“IP_lookup_table”, $message.ipaddress))
then
drop_message();
end
But this is not working out.
Any suggestions please. All responses appreciated
Did you try first with some rule that is not that destructive?
Like:
rule “look up IP Address”
when
has_field("ipaddress")
then
let LUT = lookup_value(“IP_lookup_table”, to_string($message.ipaddress));
set_field("region", LUT);
end
that would allow you to check if the lookup is working. If that works,
rule "drop message if not region"
when
has_field("ipaddress") AND
is_null(lookup_value(“IP_lookup_table”, to_string($message.ipaddress)))
then
drop_message();
end
this would allow you to debug this a little better.
Hi Jan and all,
The above logic is not working. Can you please suggest any other way to filter out the non look up values keys from indexing…
Options I have tried till date:
rule “look up IP Address”
when
has_field("ipaddress")
then
let LUT = lookup_value(“IP_lookup_table”, to_string($message.ipaddress));
set_field("region", LUT);
end
updated the logic for the below rule
rule "drop message if not region"
when
is_null(to_string($message.region))
then
drop_message();
end
the problem I am facing is it didn’t even crossing the first stage (i.e., stage 0).
please find the trace in the simulator.
it is not clear to me what you try … but I guess you want to drop all messages that are not enriched with the information from the lookup table…
so you first to
Stage 0 Rule 1
rule “look up IP Address”
when
has_field("ipaddress")
then
let LUT = lookup_value(“IP_lookup_table”, to_string($message.ipaddress));
set_field("region", LUT);
end
This will try to enrich the messages. Now you want to drop - after this when it has a ipaddress field but got no regionfield right?
Stage 1 Rule 1
rule "drop if ip but no region"
when
has_field("ipaddress") AND NOT has_field("region")
then
drop_message();
end
The order of rules in a stage is random and if you want to run rules in a specific order take different stages is the best solution.
Hey Jan,
has_field(“ipaddress”) is the culprit, it didn’t allowing to pass to next stage, so I changed to below logic…
rule "lookup-nw-data-lookup-rule"
when
is_not_null(to_string($message.nw_device_name))
then
debug("lookup-ip started");
let lut = lookup_value("lkp-ip-table_03-26-2019", to_string($message.ipaddress));
set_field("region", lut);
debug(lut);
end
stage 1 rule 1
rule "lookup-nw drop message"
when
is_null($message.region)
then
debug("dropping message initiated");
debug($message.region);
drop_message();
debug("dropping message successfull");
end
log file:
At last errors got reduced and passing both stages but the messages are not dropping.
is there are any other functions or logic to drop the non look values ipaddress regions from indexing…
If the message is drop_message(); it will not be written to Elasticsearch and dropped when the processing with the pipelines is finished and the message is not taken back from the dead with another pipeline.
Think of this like the trash_bin in your desktop … it will be cleaned once the processing is finished.
Should the message be saved and you have no other pipeline that restores them it is time to create a bug report over at github.
Here is my understanding from your provided documents and my research.
So dropped message cannot be reflected in the elasticsearch?? though the pipeline passes the dropping function successfully because the messages are reaching the elastic first and pipeline dropping them next.
Can you please provide few more details on this please.
Here are My Hurdles:
In simulator message dropping is shown as succesfull. but when I push the message using the TCP. No messages are getting dropped.
debug function reflecting its results in the log, only during the simulator load message. but not during the pushing of the messages to TCP input.
In simulator while I am debugging, $message.message giving me the whole message in the log, but when I do debug($message.device_name) giving the null value.If so, how can we check the fields??
Though I kept directly “true” in the when condition, eventhough the message is not getting dropped. all 3 pushed messages appearing in the Kibana.
5.Documentation had only basic details. Can you please suggest any other resources to learn pipelines better.
Regarding the creating bug for our issue, what are the details I need to mention in the issue?
