## Expected Behavior
When creating a lookup table using a Maxmind database ad…apter and lookup table, I expect to be able to use any field available within the Maxmind database and use that lookup value in any given pipeline.
Any field/key available or visible in the test lookup in the Lookup Tables section should also be available in the pipeline rule since the data is included in the JSON output.
## Current Behavior
Currently, it looks as though some fields are not properly available or are not properly being queried for pipeline purposes.
## Possible Solution
Enable lookups for any key in the Maxmind GeoIP databases.
## Steps to Reproduce (for bugs)
In this example pipeline rule, the is_in_european_union variable does not properly set the value because the lookup table is no properly seeing the value.
```
rule "GeoIP Lookup"
when
has_field("ClientIP")
then
let geo = lookup("cloudflare-geo-lookup-table", $message.ClientIP);
set_field("ClientIpGeolocation", geo["coordinates"]);
set_field("ClientIpGeoCountryCode", geo["country"].iso_code);
set_field("ClientIpGeoCountry", geo["country"].names.en);
set_field("ClientIpGeoCountryGdpr", geo["country"].is_in_european_union);
set_field("ClientIpGeoCityCode", geo["city"].names.en);
end
```
DEBUG enabled errors for context:
```
2021-04-30T21:01:30.207Z DEBUG [FieldAccessExpression] Unable to read property is_in_european_union from com.maxmind.geoip2.record.Country [ {"geoname_id":6252001,"is_in_european_union":false,"iso_code":"US","names":{"de":"USA","ru":"США","pt-BR":"Estados Unidos","ja":"アメリカ合衆国","en":"United States","fr":"États-Unis","zh-CN":"美国","es":"Estados Unidos"}} ]
2021-04-30T21:01:30.210Z DEBUG [FieldAccessExpression] Unable to read property isInEuropeanUnion from com.maxmind.geoip2.record.Country [ {"geoname_id":6252001,"is_in_european_union":false,"iso_code":"US","names":{"de":"USA","ru":"США","pt-BR":"Estados Unidos","ja":"アメリカ合衆国","en":"United States","fr":"États-Unis","zh-CN":"美国","es":"Estados Unidos"}} ]
```
## Context
There are myriad reasons to enable all fields from the Maxmind database. The biggest issue is being able to quickly identify countries that need to be anonymized for GDPR compliance.
## Your Environment
* Graylog Version: 4.0.6
* Java Version: openjdk version "1.8.0_282" OpenJDK Runtime Environment (build 1.8.0_282-b08) OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
* Elasticsearch Version: 7.10.2
* MongoDB Version: 4.2.13
* Operating System: Amazon Linux 2
* Browser version: Google Chrome 90.0.4430.93
Test lookup of Maxmind GeoIP City Database using 8.8.8.8 as the test IP.
```
{
"single_value": "37.751,-97.822",
"multi_value": {
"continent": {
"code": "NA",
"geoname_id": 6255149,
"names": {
"de": "Nordamerika",
"ru": "Северная Америка",
"pt-BR": "América do Norte",
"ja": "北アメリカ",
"en": "North America",
"fr": "Amérique du Nord",
"zh-CN": "北美洲",
"es": "Norteamérica"
}
},
"country": {
"confidence": null,
"geoname_id": 6252001,
"is_in_european_union": false,
"iso_code": "US",
"names": {
"de": "USA",
"ru": "США",
"pt-BR": "Estados Unidos",
"ja": "アメリカ合衆国",
"en": "United States",
"fr": "États-Unis",
"zh-CN": "美国",
"es": "Estados Unidos"
}
},
"traits": {
"autonomous_system_number": null,
"autonomous_system_organization": null,
"connection_type": null,
"domain": null,
"ip_address": "8.8.8.8",
"is_anonymous": false,
"is_anonymous_proxy": false,
"is_anonymous_vpn": false,
"is_hosting_provider": false,
"is_legitimate_proxy": false,
"is_public_proxy": false,
"is_satellite_provider": false,
"is_tor_exit_node": false,
"isp": null,
"organization": null,
"user_type": null
},
"city": {
"confidence": null,
"geoname_id": null,
"names": {}
},
"represented_country": {
"confidence": null,
"geoname_id": null,
"is_in_european_union": false,
"iso_code": null,
"names": {},
"type": null
},
"coordinates": "37.751,-97.822",
"location": {
"accuracy_radius": 1000,
"average_income": null,
"latitude": 37.751,
"longitude": -97.822,
"metro_code": null,
"population_density": null,
"time_zone": "America/Chicago"
},
"postal": {
"code": null,
"confidence": null
},
"registered_country": {
"confidence": null,
"geoname_id": 6252001,
"is_in_european_union": false,
"iso_code": "US",
"names": {
"de": "USA",
"ru": "США",
"pt-BR": "Estados Unidos",
"ja": "アメリカ合衆国",
"en": "United States",
"fr": "États-Unis",
"zh-CN": "美国",
"es": "Estados Unidos"
}
},
"subdivisions": []
},
"string_list_value": null,
"has_error": false,
"ttl": 9223372036854776000
}
```
