Issue with AzureAD Pipeline Rule - Client IP for Map

[DrunkMunki]

1. Describe your incident:
I am trying to create a map of IP’s addresses of failed logins.
I followed the guide here to setup GeoIP but the pipeline rule isnt finding the IP Address (ClientIP)

The “Input” Extractor correctly formats the message, per this guide

2. Describe your environment:

• OS Information: Ubuntu 20.04

• Package Version: GrayLog 4.3.9

3. What steps have you already taken to try and solve the problem?
I have tried following steps at 15048

Source Message: (some details changed for privacy)

{"CreationTime": "2022-11-29T19:33:16", "Id": "a8136cf2-ca32-4602-b990-e9b2155a2301", "Operation": "UserLoginFailed", "OrganizationId": "000000-b665-0000-8346-000016c7c39e", "RecordType": 15, "ResultStatus": "Failed", "UserKey": "9496049e-fbc6-494b-8fc7-26b89486412c", "UserType": 0, "Version": 1, "Workload": "AzureActiveDirectory", "ClientIP": "161.97.159.169", "ObjectId": "00000002-0000-0ff1-ce00-000000000000", "UserId": "user@domain.com", "AzureActiveDirectoryEventType": 1, "ExtendedProperties": [{"Name": "ResultStatusDetail", "Value": "UserError"}, {"Name": "UserAgent", "Value": "BAV2ROPC"}, {"Name": "UserAuthenticationMethod", "Value": "16"}, {"Name": "RequestType", "Value": "OAuth2:Token"}], "ModifiedProperties": [], "Actor": [{"ID": "9496049e-fbc6-494b-8fc7-26b89486412c", "Type": 0}, {"ID": "user@domain.com", "Type": 5}], "ActorContextId": "2d04b444-b665-4d7d-8346-728b16c7c39e", "ActorIpAddress": "161.97.159.169", "InterSystemsId": "e34c081c-a363-4a9f-bbcd-2c1391ecbaba", "IntraSystemId": "a8136cf2-ca32-4602-b990-e9b2155a2301", "SupportTicketId": "", "Target": [{"ID": "00000002-0000-0ff1-ce00-000000000000", "Type": 0}], "TargetContextId": "2d04b444-b665-4d7d-8346-728b16c7c39e", "ApplicationId": "00000002-0000-0ff1-ce00-000000000000", "DeviceProperties": [{"Name": "BrowserType", "Value": "Other"}, {"Name": "IsCompliantAndManaged", "Value": "False"}], "ErrorNumber": "50126", "LogonError": "InvalidUserNameOrPassword", "timestamp": "2022-11-29 19:33:16.000"}

Simulation of above message:

Example Message:

Pipeline Rule:

there was mention to add debug to the rule, but not sure what to actually add…

[gsmith]

Heoo @DrunkMunki

Bug rule , Here is mine

rule "GeoIP lookup: srcip"

when

  has_field("srcip")

then

let geo = lookup("geoip", to_string($message.srcip));

 set_field("src_ip_geo_location", geo["coordinates"]);

 set_field("src_ip_geo_country", geo["country"].iso_code);

 set_field("src_ip_geo_city", geo["city"].names.en);
 
 debug(geo);

end

Have you tried testing those IP Address under ClientIP?
I used you ClientIP data in my setup seams to work.

On the left pane on the global search do you see any fields pertaining to GeoIP? I know this was a issue back a couple months ago the naming convention change but they changed it back.

Noticed for some reason its not like my Pipe Rule from above.

[DrunkMunki]

thanks for the super quick reply @gsmith

I have added the debug code but nothing shows in the server.log file when i try and ‘simulate’ the process

from my understanding the below varies on the field data,
has_field("ClientIP")

i did a ‘src’ lookup and got no results

I also looked up ClientIP and got results

I did notice when i tried to click on

I get the errors in the log:
WARN [LookupTableService] Lookup table <watchlist> does not exist

[gsmith]

Ok
that was mine above using src but for you try something like "city_name" , “geo” , “geolocation” or “clientip” see is anything come up. This is just checking for fields incase it was rename or bug issue. Also click on this part, I marked it in red

[gsmith]

@DrunkMunki

I get the errors in the log:
WARN [LookupTableService] Lookup table <watchlist> does not exist

Oh gotcha, let me check
EDIT; @DrunkMunki I post wrong screen shot , I corrected it just now

[DrunkMunki]

Yeah, clicked on both links for the fields, same result.
the Bold text indicates which one was selected.

[gsmith]

Not sure if this was answer , but does this test section in GEO lookup-table work?

I used your ip address from above under ClientIP

EDIT : @DrunkMunki If that doesn’t work then you may have a configuration issue, not sure yet

[DrunkMunki]

No when i click on the Lookup for GeoIP i get the error, which goes into a loop refreshing the page

and error at the bottom

and the log floods with

2022-11-30T15:26:41.619+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:42.621+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:43.619+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:44.620+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:45.621+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:46.617+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:47.615+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:48.610+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:49.613+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:50.624+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist
2022-11-30T15:26:51.625+11:00 WARN  [LookupTableService] Lookup table <watchlist> does not exist

[gsmith]

Yeppers,

I was afraid of that,
Check LIst:

Need to make sure 100% you have everything installed,

sudo apt-get update && sudo apt-get install graylog-server graylog-enterprise-plugins graylog-integrations-plugins graylog-enterprise-integrations-plugins

The status of MongoDb, Graylog & Elasticsearch is Good.

systemctl status graylog-server
systemctl status elasticsearch
systemctl status mongo

Under System/Configuration make sure Pipeline is after message chain

Same section , bottom of that page ensure it looks something like this

EDIT: if all that is correct, try restart Graylog service and tail -f graylog log file look for any errors and warnings

[DrunkMunki]

thanks @gsmith
ok ran the update and was missing: graylog-enterprise-plugins graylog-integrations-plugins graylog-enterprise-integrations-plugins

was missing the MaxMind ASN database, downloaded and put in same location as City one.
restarted graylog service and is showing good logs

2022-11-30T15:42:01.131+11:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/6372e6e5e2c93641eec8752e [@2959cd47] STARTING
2022-11-30T15:42:01.134+11:00 INFO  [LookupTableService] Data Adapter geoip/63868ff2f66a3f67c30b8064 [@2cb3c7ed] STARTING
2022-11-30T15:42:01.137+11:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/6372e6e5e2c93641eec8752e [@2959cd47] RUNNING
2022-11-30T15:42:01.143+11:00 INFO  [LookupTableService] Data Adapter geoip/63868ff2f66a3f67c30b8064 [@2cb3c7ed] RUNNING
2022-11-30T15:42:01.143+11:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/63868ff2f66a3f67c30b8064/@2cb3c7ed> [interval=3600000ms]
2022-11-30T15:42:01.207+11:00 INFO  [LookupTableService] Cache watchlist-cache/6372e6e5e2c93641eec8752c [@1d40a5b2] STARTING
2022-11-30T15:42:01.209+11:00 INFO  [LookupTableService] Cache watchlist-cache/6372e6e5e2c93641eec8752c [@1d40a5b2] RUNNING
2022-11-30T15:42:01.209+11:00 INFO  [LookupTableService] Cache geoip/6386902ef66a3f67c30b80e3 [@bfeb005] STARTING
2022-11-30T15:42:01.209+11:00 INFO  [LookupTableService] Cache geoip/6386902ef66a3f67c30b80e3 [@bfeb005] RUNNING
2022-11-30T15:42:01.221+11:00 INFO  [LookupTableService] Starting lookup table watchlist/6372e6e5e2c93641eec87530 [@53fa8941] using cache watchlist-cache/6372e6e5e2c93641eec8752c [@1d40a5b2], data adapter watchlist-mongo/6372e6e5e2c93641eec8752e [@2959cd47]
2022-11-30T15:42:01.221+11:00 INFO  [LookupTableService] Starting lookup table geoip/6386dd49f66a3f67c30c2258 [@ec1d9e6] using cache geoip/6386902ef66a3f67c30b80e3 [@bfeb005], data adapter geoip/63868ff2f66a3f67c30b8064 [@2cb3c7ed]

and confirmed the country is now being added

Also didnt have the “Geo-Location” Processor enabled…

Also Chrome Cache was affecting the lookup able page, using Firefox i was able to access it, so not sure where in the process this was fixed… but everything 100% now.

thanks again gsmith!

[gsmith]

@DrunkMunki

Happy Logging my friend you got luck, Im heading home from work in a few AHAHA