# IP anonymization

**URL:** <https://community.graylog.org/t/ip-anonymization/16610>\
**Category:** Graylog Central (peer support)\
**Created:** [July 30, 2020, 8:45pm UTC](https://community.graylog.org/t/ip-anonymization/16610 "2020-07-30T20:45:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nandon](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/nandon/32/6969_2.png) [@nandon](https://community.graylog.org/u/nandon)\
**Post date:** [July 30, 2020, 8:45pm UTC](https://community.graylog.org/t/ip-anonymization/16610/1 "2020-07-30T20:45:11Z")

</div>

Hi folks,

I have Graylog running and analyzing logs from different input sources (beats & gelf).  
Due to GDPR reasons I need to make sure IPs are anonymized.  
Currently I use this Plugin [https://github.com/graylog-labs/graylog-plugin-ipanonymizer](https://github.com/graylog-labs/graylog-plugin-ipanonymizer), which replaces the 4th octet of IPv4 with xxx.  
Unfortunately this project is not continued and references to the use of Pipelines.

Is it possible to filter all kind of IPv4 and IPv6 addresses in a similar way that plugin does it, by removing parts of the address before storing it in the MongoDB backend?

```auto
IPV6: ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

IPV4: (?<![0-9])(?:(?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])

```

I tried out the regex in the extractor section, but there I could just replace the whole IP address but not just a part of it.  
It should be possible, that the regex is applied multiple times on the message filed, because it can happen, that IPs are sent at different positions at the message.

Is the usage of a Pipeline the right way to achieve it and ho exactly would I need to formulate a rule to do this?

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [July 31, 2020, 10:03am UTC](https://community.graylog.org/t/ip-anonymization/16610/2 "2020-07-31T10:03:21Z")

</div>

Yes, pipeline function is ideal solutions:

For IPv4, this pipeline function replace last octet with xxx on field message:

```auto
rule "anonymize IPv4"
when
   has_field("message")
then
      let anon_ip = regex_replace(pattern: "(?<![0-9])(?:([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])",
        value: to_string($message.message),
        replacement: "$1.$2.$3.xxx"
    );
    set_field("message", anon_ip);
end

```

---

<div class="post-metadata">

**Author:** ![nandon](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/nandon/32/6969_2.png) [@nandon](https://community.graylog.org/u/nandon)\
**Post date:** [August 2, 2020, 7:59pm UTC](https://community.graylog.org/t/ip-anonymization/16610/3 "2020-08-02T19:59:26Z")

</div>

Hi @shoothub  
thanks for the example.  
It works the way I was expectiong it to work!

The problem was, I had to upgrade my Graylog first from 2.4.6 to 3.3.3 because this “regex\_replace” function has been added at Graylog 3.0.0 …  
It was a bit annoying, because with this update also a few coinfig settings have changed and I needed to modify the start ENV parameters for my Docker container but now it does, what it should.

I will now try the same for IPv6. When I have a working Rule I will post it here.

Best regards!

Edit:

Graylog had some issues with the IPv6 patterns that I wanted to use and because I already replace IPv4 I do not need this part of the regex for IPv6

```
rule "anonymize IPv6 v1"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}|:)",
    value: to_string($message.message),
    replacement: "$1:$2:yyy:yyy:yyy:yyy:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v2"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):(:[0-9A-Fa-f]{1,4}|:)",
    value: to_string($message.message),
    replacement: "$1:$2:yyy:yyy:yyy:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v3"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):(?::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?|:)",
    value: to_string($message.message),
    replacement: "$1:$2:yyy:yyy:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v4"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):(?::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?|:)",
    value: to_string($message.message),
    replacement: "$1:$2:yyy:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v5"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):(?::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?|:)",
    value: to_string($message.message),
    replacement: "$1:yyy:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v6"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):([0-9A-Fa-f]{1,4}):(?::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?|:)",
    value: to_string($message.message),
    replacement: "$1:yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v7"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "([0-9A-Fa-f]{1,4}):(?::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?|:)",
    value: to_string($message.message),
    replacement: "yyy::"
  );
  set_field("message", anonym_ip);
end

```

* * *

```
rule "anonymize IPv6 v8"
when
  has_field("message")
then
  let anonym_ip = regex_replace(
    pattern: "::([0-9A-Fa-f]{1,4})(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?(?::([0-9A-Fa-f]{1,4}))?",
    value: to_string($message.message),
    replacement: "::yyy"
  );
  set_field("message", anonym_ip);
end

```

With all those 9 Rules I filled a Pipeline which I applied to all Streams. Now everything is anonymized

 ![Anonymize IP](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/da55ba01993672cd8a2fb0fa0f3fe136ff32c421.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [August 16, 2020, 7:59pm UTC](https://community.graylog.org/t/ip-anonymization/16610/4 "2020-08-16T19:59:45Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.