Having an issue with graylog rule

Hi Folks,

I am trying to create a rule to mask a social security number and am running into a lot of trouble.
my example log message contains a block as such
“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“121212121”} I want this to look like
“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“***”}

the rule I am trying to use looks like this

rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “(?<=SocialSecurityNumber":")\d{9}”, “***”);
set_field(“message”, message);
end

I tried many other ways of doing this but none worked.
on regex101 I made a regex ("SocialSecurityNumber":")(\d{9})" that matches the string but when I post this in my graylog rule it gives an invalid error.
can someone please help?
much appreciated!

Hey @agharibk && welcome.

I have hashed IP addresses not SSN, but if you can get the SSN into its own field then hash it from there. This may require two Rules and two stages. Something like this.

Once you extracted the SSN into a field lets say, SocialSecurityNumber then something like this

rule "anonymize-ssn"
when
  has_field("SocialSecurityNumber")
then
  let ssn= to_string($message.SocialSecurityNumber);
  let hash = sha256(SocialSecurityNumber);
  set_field("ssn_now", hash);
end

Hello gsmith,
thanks for the welcome.
is this the only way? I just want to mask it with ***
if I extract it as a field then hash it wont this change the way my original log looks

I am close I got a rule that almost works
rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “SocialSecurityNumber":"”, “"SocialSecurityNumber":"***"”);
set_field(“message”, message);
end

the only problem is when I add SocialSecurityNumber":"\d{9}" to remove the digits I get an invalid expression so I need to add an extra \ before the d. but with that being added it does not work

Message should stay origanal. For example if you use regex and extract it from the full_message/message field

or if you dont want to do that then route all message with SSN into a different stream.

EDIT short answer is no. You extracted the SSN into a field and your only hashing that field you created. If you dont want the original message there is a way to hash it and replace the origianal message

but dont you create the field in one rule then hash it in another or can I create the field out of the message then hash it and save it back in the same message?

maybe its easier if you understand what my current problem is because I am super close to getting it working

rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = replace(to_string($message.message), “SocialSecurityNumber":"\d{9}"”, “"SocialSecurityNumber":"***"”);
set_field(“message”, message);
end

it currently works when I remove \d{9} and replaces everything but the actual digits. not sure why \d{9} just causes everything to break

If your using regex , then yes.
Example:

rule "ipv4"
when
    regex(".*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*",to_string($message.message)).matches
then
    let pair =  regex(".*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3})[^0-9a-zA-Z].*",to_string($message.message),["src","dst"]);
    set_field("pipeSrcIPv4",to_ip(pair.src));
    set_field("pipeDstIPv4",to_ip(pair.dst));
end

if you go to https://regex101.com/
and use regex (?<=SocialSecurityNumber":").{9} with my log text
2023-03-07 15:47:17,870 [http-nio-8080-exec-7] [e2e9b710385e4931bcd74285417047de] [traceId=] TRACE com.mvsc.vitu.logging.logback.rest.LoggingRequestInterceptor - Received response [200 - OK; Cache-Control: no-cache; Pragma: no-cache; Content-Length: 1301; Content-Type: application/json; charset=utf-8; Expires: -1{“DealerLicenseType”:“vi”,“DealerLicenseNumber”:1003066,“DealerLicenseSuffix”:1,“EmpCustomerNumber”:238635359,“LogNumber”:369449,“DealerLicensePin”:“9355”,“EmployeeType”:“AGT”},“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“111111111”}})]

it highlights the exact value that I need to change why is it not actually making the changes?

rule “Mask SocialSecurityNumber”
when
has_field(“message”) AND
contains(to_string($message.message), “SocialSecurityNumber”)
then
let message = regex.replace(to_string($message.message), “(?<=SocialSecurityNumber":").{9}”, “***”);
set_field(“message”, message);
end

Just to make sure I have this right, You do not want to make a new field and Hash the SSN ? Correct?
You want to hash the SSN in the Message field?

correct
I want the message to look like this
2023-03-07 15:47:17,870 [http-nio-8080-exec-7] [e2e9b710385e4931bcd74285417047de] [traceId=] TRACE com.mvsc.vitu.logging.logback.rest.LoggingRequestInterceptor - Received response [200 - OK; Cache-Control: no-cache; Pragma: no-cache; Content-Length: 1301; Content-Type: application/json; charset=utf-8; Expires: -1{“DealerLicenseType”:“vi”,“DealerLicenseNumber”:1003066,“DealerLicenseSuffix”:1,“EmpCustomerNumber”:238635359,“LogNumber”:369449,“DealerLicensePin”:“9355”,“EmployeeType”:“AGT”},“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“***”}})]

Ok,
I think another member here in the forum did something simialr to this. Im looking it up.

Actually, Im going to tag Batman on this on @tmacgbay

thank you so much for being so helpful I really appreciate it.
I have spent all day on this and am sooo close

1 Like

I believe you are,
its the finer details Im over looking. Ill post any find here but @tmacgbay is much better at Pipes then I am for an isseu like this.

1 Like

Hey @agharibk

To brief you on what I did:

I uploaded the log from thei post and made a fake SSN for testing.

2023-03-07 15:47:17,870 [http-nio-8080-exec-7] [e2e9b710385e4931bcd74285417047de] [traceId=] TRACE com.mvsc.vitu.logging.logback.rest.LoggingRequestInterceptor - Received response [200 - OK; Cache-Control: no-cache; Pragma: no-cache; Content-Length: 1301; Content-Type: application/json; charset=utf-8; Expires: {“DealerLicenseType”:“vi”,“DealerLicenseNumber”:1003066,“DealerLicenseSuffix”:1,“EmpCustomerNumber”:238635359,“LogNumber”:369449,“DealerLicensePin”:“9355”,“EmployeeType”:“AGT”},“Value”:{“SearchKey”:3,“SocialSecurityNumber”:“789-56-8189”}})]

found the regx that works by grabbing the int for the SSN (\d{3}-\d{2}-\d{4})

Created a pipe

rule "Mask SocialSecurityNumber"
when
      has_field("message") AND contains(to_string($message.message), "SocialSecurityNumber")
then       
       let ssn = regex_replace("(\\d{3}-\\d{2}-\\d{4})", to_string($message.message),"regex");
       set_field("message", ssn);
end

Results:
Before:

After:

This regex only grabs number formated liek this xxx-xx-xxxx

hope that helps

2 Likes

Well it looks like I am no longer the “pipes” master! Well done with the research and testing there - that’s what I would have done.

1 Like

you are amazing but this wont work. it will replace all 9 digits. my social security number is just 9 digits with no dashes. I need a way to select the SocialSecuritynumber then change the 9 digits right after

rule "Mask SocialSecurityNumber"
when
      has_field("message") AND contains(to_string($message.message), "SocialSecurityNumber")
then       
       let ssn = regex_replace("(?<=SocialSecurityNumber\":\")(\\d{9})", to_string($message.message),"***");
       set_field("message", ssn);
end

this rule should do so but it does not do anything ;/

1 Like

could you please check my last comment maybe you know what I am missing

This may not be the final solution, but you need to double escape your \d. Java regex requires it. I use regexplanet to convert my perl regex into java flavor.

If you can’t rewrite the message field as you desire, you might consider a different approach. If you parse the entire message, you could deal with it as an isolated values and hash it or do whatever you like. You can then rewrite the original message field to something like “this message parsed and message field deleted.” This would satisfy all audit requirements, because you can demonstrate the parsing does not alter the contents of the original message, only the format.

Good luck!

2 Likes

Here are the list of things that need to be escaped \ to mean literal:

& | : \ / + - ! ( ) { } [ ] ^ " ~ * ?

NOTE: For technical reasons in pipelines, its a double escape: \\

NOTE ALSO: The escapes you put in were not shown because you need to use the preformatted text </> forum tool to show them… highlight the text and click that button </> in the editor. I revised your post to show it.

So your regex just need to be adjusted for escapes:

(?<=SocialSecurityNumber\\"\\:\\")(\\d{9})

FYI, without the </> tool, that looks like this:

(?<=SocialSecurityNumber\“\:\”)(\d{9})

ew.

dang it, my slow typing allowed @chris.black-gl to jump in!! haha!! I like/prefer his solution of parsing the whole message and re-writing the original message to solely denote the actions taken.

I may have been first, but I left out the escaping of the literal characters, so yours was more correct! :checkered_flag:

1 Like