What I would like to do is look up an IP from an existing field and create an additional field that shows the name of the subnet to which it belongs. So, for example, looking up 192.168.1.1 would return ‘office’.
I had originally thought I could do this with a CSV Lookup Table, but if my understanding is correct, I would need to have each individual IP in the lookup table instead of the CIDR definition - is this correct or can a cidr_match be performed on the key of the lookup table?
I know that there is a cidr_match function, but I can not figure out how to work it in to the lookup_value function.
If I were to generate a CSV with each individual IP, I could then look up the subnet name something like this:
rule "lookup: source_ip subnet name"
when
has_field("source_ip")
then
let subnet_name = lookup_value(lookup_table: "subnet_table_name", key: $message.source_ip, default: "unknown");
set_field("subnet_id", to_string(subnet_name));
end
but I am unsure of where cidr_match would fit into this pipeline rule when using the below CSV instead and attempting to get the subnet_name for (for example) 192.168.1.1:
@bruce,
I think (but I’m not sure) the lookup will do 1-1 match. Eg. convert banana -> yellow fruit; carrot -> orange vegetable
I don’t see the source_ip -> subnet (NOT subnet_name) conversion in your code.
If you use only /16 and /24 subnets, maybe you can try to cut the end of it IP, and make a lookup for “192.168.1” if no match you can try “192.168”, etc.
The geoip locator maybe do the subnet conversion. If you can upload personal data to geoip database, maybe it can solve it.
First we should know the lookup function does the ip-> subnet conversion, to do a match a subnet -> subnet_name.
Unfortunately that’s exactly what I’ve had to resort to doing, populated CSV with every single IP to subnet…
The lookup table functionality basically is a 1:1 mapping, i.e. the key needs to match one column, and it pulls in the other column(s) in that row. The cidr_match() function is pretty much moot at that point, as it cannot divine a subnet without knowing the masks of each subnet beforehand; the other side of this is you cannot call a lookup table without a key, and you cannot iterate over the table.
I think this is going to have to be a custom lookup table plugin, purpose built for single column lookups (all the subnets) utilizing a cidr_match() function iterated over all of the rows.
At the very least, I’m happy to hear I’m not alone in this.
EDIT: Actually, after thinking about it, the threat plugin has to be doing this exact same thing to match to the CIDRs from Spamhaus DROP/EDROP. Overall should take comparatively little modification. @jochen? @jan?
If the list of subnets were shorter, I could easily write a pipeline that would match each of the subnets and assign the appropriate value to a field, but this does not scale well.
FYI we have an “our Internet IP addresses to sitename” lookup file. But
unlike you that consists of a mixture of subnets plus individual IP
addresses - so CIDR support doesn’t really help. I bit the bullet and
simply generate a large CSV file with one IP per line. Not really that much
extra work, and can cover all the use-cases.
Ours is 20K entries - graylog happily gobbles it up and works fine
