# Replace sensitive information

**URL:** <https://community.graylog.org/t/replace-sensitive-information/28691>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [May 5, 2023, 7:46am UTC](https://community.graylog.org/t/replace-sensitive-information/28691 "2023-05-05T07:46:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![JepettoLofgren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jepettolofgren/32/14748_2.png) [@JepettoLofgren](https://community.graylog.org/u/JepettoLofgren)\
**Post date:** [May 5, 2023, 7:46am UTC](https://community.graylog.org/t/replace-sensitive-information/28691/1 "2023-05-05T07:46:39Z")

</div>

Hello!

I am trying to create a Pipeline rule that replaces sensitive information. Im running into issues with using '[^&]\*.

For example i am trying to filter out the following password: ‘#p54\_L35’. It works well if i replace '[^&]\* with ‘#p54\_L35’. So what am i doing wrong here? Is there an alternative way of filtering out sensitive information?

See code below

rule “Hide sensitive information”  
when  
has\_field(“message”)  
then  
let message = to\_string($message.message);  
let filteredMessage = replace(message, “password: '[^&]_'", “password: '[redacted]'”, -1);  
let filteredMessage2 = replace(filteredMessage, "token: ''[^&]_'”, “password: '[redacted]'”, -1);  
let filteredMessage3 = replace(filteredMessage2, “token=[^&]\*”, “token=[redacted]”, -1);  
set\_field(“message”, filteredMessage3);  
end

---

<div class="post-metadata">

**Author:** ![JepettoLofgren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jepettolofgren/32/14748_2.png) [@JepettoLofgren](https://community.graylog.org/u/JepettoLofgren)\
**Post date:** [May 8, 2023, 12:07pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/2 "2023-05-08T12:07:38Z")

</div>

Bump!

I still havent been able to figure this out. Any kind soul out there?

---

<div class="post-metadata">

**Author:** ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)\
**Post date:** [May 8, 2023, 12:52pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/3 "2023-05-08T12:52:03Z")

</div>

could you post one original message, with a faked password?

---

<div class="post-metadata">

**Author:** ![JepettoLofgren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jepettolofgren/32/14748_2.png) [@JepettoLofgren](https://community.graylog.org/u/JepettoLofgren)\
**Post date:** [May 8, 2023, 1:33pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/4 "2023-05-08T13:33:31Z")

</div>

Of course!

Below is a message with fake tokens.

2023-05-08T13:29:20.315Z - e[32minfoe[39m: POST /api/company/basic?token=5a0465asdas987987fds987822ed2ff1e6a0505fba705e879b2a&customerId=15429 method=POST, originalUrl=/api/company/basic?token=5a0465asdas987987fds987822ed2ff1e6a0505fba705e879b2a&customerId=15429, ip=::ffff:10.421.1.123, correlationId=51232349ea-1f234285-446f-923435f-edc5b994eded, token=5a0465asdas987987fds987822ed2ff1e6a0505fba705e879b2a, customerId=9999, q=[{“a”:“orgNumber”,“c”:“eq”,“v”:10001090218}], country=SE, f=[name, cfar, orgNumber, prospectingId], finished=true, time=5, statusCode=200

---

<div class="post-metadata">

**Author:** ![drewmiranda-gl](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/drewmiranda-gl/32/13376_2.png) [@drewmiranda-gl](https://community.graylog.org/u/drewmiranda-gl)\
**Post date:** [May 8, 2023, 5:49pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/5 "2023-05-08T17:49:33Z")

</div>

This recent blog post on this topic may be of interest:

> **[Redacting Message Fields for Privacy Purposes](https://www.graylog.org/post/redacting-message-fields-for-privacy-purposes/)**
>
> Many organizations today have strict data privacy regulations that they must comply with. These privacy regulations can often clash with the requirements of security, application and operations teams who need detailed log information. At Graylog,...

---

<div class="post-metadata">

**Author:** ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)\
**Post date:** [May 9, 2023, 3:11pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/6 "2023-05-09T15:11:49Z")

</div>

those are the Steps I recommend:

1. make your log machine readable in a rule in a pipeline. Parse it into the correct fields.
2. redact all the fields you want to hide in rules
3. delete the message by replacing it with some bogus-content. The message field is mandatory in graylog - you can not delete it.

in detail:

1. parsing:  
Create the following Grok Patterns:  
name: ApacheLogs

```auto
%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}T%{TIME:time}Z - e\[32minfoe\[%{INT:response_time}m: %{DATA_ALL_BUT_SPACE:http_request_method} %{DATA_ALL_BUT_SPACE:path} method=%{DATA_ALL_BUT_SPACE:http_request_method} originalUrl=%{DATA_ALL_BUT_SPACE:http_originalUrl} ip=%{DATA_ALL_BUT_SPACE:source_ip} correlationId=%{DATA_ALL_BUT_SPACE:correlationId}

```

and this one names as “DATA\_ALL\_BUT\_SPACE”:

```auto
[^]+

```

The first one will parse the first part of our log, the rest you can build on your own.

1. create a rule in a pipeline attached to your logs:

```auto
rule "parsing: apache Logs"
when
  true // or better condition if you have
then
  set_fields(
    grok(
      pattern:"^%{ApacheLogs}",
      value:to_string($message.message),
      only_named_captures:true
    )
  );
end

```

This pipeline will set the stuff you need in different fields. There is the pattern for DATA\_ALL\_BUT\_SPACE, you can create a similar one to make the “,” not beeing captured.

1. create rules redacting the fields you don’t want to have with full values:

```auto
rule "redacting"
when
  has_field("secret_values")
then
  set_field(
    field:"secret_values", 
    value:
        abbreviate(
            value:sha256(
                to_string($message.secret_values)
            ),
            width:to_long("10")
        )
    );
end

```

This will replace the field secret\_values with the first 10 signs of the sha256 hash of the value of that field.

and also redacting the full-message:

```auto
rule "redacting"
when
  has_field("message")
then
  set_field("message", "bogus-content");
end

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [May 23, 2023, 3:11pm UTC](https://community.graylog.org/t/replace-sensitive-information/28691/7 "2023-05-23T15:11:51Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.