I am trying to create a rule to mask a social security number and am running into a lot of trouble.
my example log message contains a block as such
“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“121212121”} I want this to look like
“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“***”}
the rule I am trying to use looks like this
rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “(?<=SocialSecurityNumber":")\d{9}”, “***”);
set_field(“message”, message);
end
I tried many other ways of doing this but none worked.
on regex101 I made a regex ("SocialSecurityNumber":")(\d{9})" that matches the string but when I post this in my graylog rule it gives an invalid error.
can someone please help?
much appreciated!
I have hashed IP addresses not SSN, but if you can get the SSN into its own field then hash it from there. This may require two Rules and two stages. Something like this.
Once you extracted the SSN into a field lets say, SocialSecurityNumber then something like this
rule "anonymize-ssn"
when
has_field("SocialSecurityNumber")
then
let ssn= to_string($message.SocialSecurityNumber);
let hash = sha256(SocialSecurityNumber);
set_field("ssn_now", hash);
end
Hello gsmith,
thanks for the welcome.
is this the only way? I just want to mask it with ***
if I extract it as a field then hash it wont this change the way my original log looks
I am close I got a rule that almost works
rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “SocialSecurityNumber":"”, “"SocialSecurityNumber":"***"”);
set_field(“message”, message);
end
the only problem is when I add SocialSecurityNumber":"\d{9}" to remove the digits I get an invalid expression so I need to add an extra \ before the d. but with that being added it does not work
or if you dont want to do that then route all message with SSN into a different stream.
EDIT short answer is no. You extracted the SSN into a field and your only hashing that field you created. If you dont want the original message there is a way to hash it and replace the origianal message
but dont you create the field in one rule then hash it in another or can I create the field out of the message then hash it and save it back in the same message?
maybe its easier if you understand what my current problem is because I am super close to getting it working
rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “SocialSecurityNumber":"\d{9}"”, “"SocialSecurityNumber":"***"”);
set_field(“message”, message);
end
it currently works when I remove \d{9} and replaces everything but the actual digits. not sure why \d{9} just causes everything to break
rule "ipv4"
when
regex(".*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*",to_string($message.message)).matches
then
let pair = regex(".*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*",to_string($message.message),["src","dst"]);
set_field("pipeSrcIPv4",to_ip(pair.src));
set_field("pipeDstIPv4",to_ip(pair.dst));
end
if you go to https://regex101.com/
and use regex (?<=SocialSecurityNumber":").{9} with my log text
2023-03-07 15:47:17,870 [http-nio-8080-exec-7] [e2e9b710385e4931bcd74285417047de] [traceId=] TRACE com.mvsc.vitu.logging.logback.rest.LoggingRequestInterceptor - Received response [200 - OK; Cache-Control: no-cache; Pragma: no-cache; Content-Length: 1301; Content-Type: application/json; charset=utf-8; Expires: -1{“DealerLicenseType”:“vi”,“DealerLicenseNumber”:1003066,“DealerLicenseSuffix”:1,“EmpCustomerNumber”:238635359,“LogNumber”:369449,“DealerLicensePin”:“9355”,“EmployeeType”:“AGT”},“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“111111111”}})]
it highlights the exact value that I need to change why is it not actually making the changes?
rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = regex.replace(to_string($message.message), “(?<=SocialSecurityNumber":").{9}”, “***”);
set_field(“message”, message);
end
I believe you are,
its the finer details Im over looking. Ill post any find here but @tmacgbay is much better at Pipes then I am for an isseu like this.
found the regx that works by grabbing the int for the SSN (\d{3}-\d{2}-\d{4})
Created a pipe
rule "Mask SocialSecurityNumber"
when
has_field("message") AND contains(to_string($message.message), "SocialSecurityNumber")
then
let ssn = regex_replace("(\\d{3}-\\d{2}-\\d{4})", to_string($message.message),"regex");
set_field("message", ssn);
end
you are amazing but this wont work. it will replace all 9 digits. my social security number is just 9 digits with no dashes. I need a way to select the SocialSecuritynumber then change the 9 digits right after
rule "Mask SocialSecurityNumber"
when
has_field("message") AND contains(to_string($message.message), "SocialSecurityNumber")
then
let ssn = regex_replace("(?<=SocialSecurityNumber\":\")(\\d{9})", to_string($message.message),"***");
set_field("message", ssn);
end
this rule should do so but it does not do anything ;/
This may not be the final solution, but you need to double escape your \d. Java regex requires it. I use regexplanet to convert my perl regex into java flavor.
If you can’t rewrite the message field as you desire, you might consider a different approach. If you parse the entire message, you could deal with it as an isolated values and hash it or do whatever you like. You can then rewrite the original message field to something like “this message parsed and message field deleted.” This would satisfy all audit requirements, because you can demonstrate the parsing does not alter the contents of the original message, only the format.
NOTE: For technical reasons in pipelines, its a double escape: \\
NOTE ALSO: The escapes you put in were not shown because you need to use the preformatted text </> forum tool to show them… highlight the text and click that button </> in the editor. I revised your post to show it.
So your regex just need to be adjusted for escapes:
(?<=SocialSecurityNumber\\"\\:\\")(\\d{9})
FYI, without the </> tool, that looks like this:
(?<=SocialSecurityNumber\“\:\”)(\d{9})
ew.
dang it, my slow typing allowed @joe.gross to jump in!! haha!! I like/prefer his solution of parsing the whole message and re-writing the original message to solely denote the actions taken.
