# Replace string in field with pipeline rule

**URL:** <https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [April 4, 2019, 3:22pm UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807 "2019-04-04T15:22:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![matemikulic](https://avatars.discourse-cdn.com/v4/letter/m/b5a626/32.png) [@matemikulic](https://community.graylog.org/u/matemikulic)\
**Post date:** [April 4, 2019, 3:22pm UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/1 "2019-04-04T15:22:04Z")

</div>

Hi,  
I have a n conundrum on how to write a pipeline rule to omit password from log.  
Log line looks something like this:  
`Mar 19 12:28:36 localhost haproxy[56341]: 77.243.30.178:12273 [19/Mar/2019:12:28:36.317] api.example.com http-api/BACKEND-1 0/0/1/12/13 200 183 - - ---- 709/707/2/1/0 0/0 {||api.example.com} {189299} "GET /api/command?user=someUser&password=myPassWord&cmd=SOMETHING HTTP/1.1" - -`

And I want to change it to look something like this:  
`Mar 19 12:28:36 localhost haproxy[56341]: 77.243.30.178:12273 [19/Mar/2019:12:28:36.317] api.example.com http-api/BACKEND-1 0/0/1/12/13 200 183 - - ---- 709/707/2/1/0 0/0 {||api.example.com} {189299} "GET /api/command?user=someUser&password= ***** &cmd=SOMETHING HTTP/1.1" - -`

So omitting password is the goal.  
My progress so far is that I’ve created a regext that matches password and incorporated it in a rule.

```
rule "omit passwords"
when
    regex("(?<=password=)[^&]+").matches == true
then

end

```

I’ve connected it to All messages stream. but not all massages will have password part and will not match, and it seems they get unprocessed with all my attempts at configuring “then”.

Prior to pieline, I’ve configured Haproxy extractor and that works fine. So fields that need edit are “http\_request” and “message”.  
Tnx in advance.

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [April 4, 2019, 10:07pm UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/2 "2019-04-04T22:07:56Z")

</div>

There is currently no way to replace text in a string via a regular expression (or otherwise), but you could add it as a feature request on the Graylog github page?

On a side note, passing credentials via query string is… not so good. If it’s an app built in-house, try using headers instead because the problem with credentials in query strings is that the chance of the credentials inadvertantly leaking are larger than I’d personally be comfortable with.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [April 5, 2019, 8:32am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/3 "2019-04-05T08:32:06Z")

</div>

when Graylog 3.0 is used, you would use the `regex_replace` function.

> <https://github.com/Graylog2/graylog2-server/pull/4719/commits/23c5782489ef6ccea6d8d10f13cef3620a7bf02f>

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [April 5, 2019, 9:42am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/4 "2019-04-05T09:42:24Z")

</div>

Huh, I checked the documentation for 3.0 (functions list) and didn’t see regex\_replace in there - but cool that it exists! 😃

---

<div class="post-metadata">

**Author:** ![lindonm](https://avatars.discourse-cdn.com/v4/letter/l/b19c9b/32.png) [@lindonm](https://community.graylog.org/u/lindonm)\
**Post date:** [April 9, 2019, 1:18am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/5 "2019-04-09T01:18:38Z")

</div>

> [@matemikulic](#):
>
> So fields that need edit are “http\_request” and “message”.

Additionally, I believe that it’s not possible to edit the original “message” field? I’m guessing this would need to be done at the log source end, some kind of filebeat processor config? (assuming you’re using filebeat of course)

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [April 9, 2019, 6:34am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/6 "2019-04-09T06:34:44Z")

</div>

> I believe that it’s not possible to edit the original “message” field?

you are able to edit ANY field of a message, even remove and add as many as you like. The only that you need to keep in mind is that Graylog needs `timestamp`, `source` and `message` field to store a message …

---

<div class="post-metadata">

**Author:** ![lindonm](https://avatars.discourse-cdn.com/v4/letter/l/b19c9b/32.png) [@lindonm](https://community.graylog.org/u/lindonm)\
**Post date:** [April 10, 2019, 12:11am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/7 "2019-04-10T00:11:29Z")

</div>

> [@jan](#):
>
> you are able to edit ANY field of a message, even remove and add as many as you like. The only that you need to keep in mind is that Graylog needs `timestamp` , `source` and `message` field to store a message …

Thanks Jan,

I’m a graylog newbie, but I was under the impression that “message” can’t be changed - I think I based this off the fact that “cut” isn’t an option on the message field when creating an extractor.

Apologies up front to the OP for going off-topic, but Jan could you clarify my understanding?

Let’s say I have a message

```auto
ColumnA,ColumB,ColumnC
2019-04-10,Frank,Password123

```

( I believe this is similar to what the OP is mentioning).

So I have my extractor, and after it runs Graylog has 4+ columns;

```auto
Timestamp,Message,UserName,Password

```

But I want to remove password for security reasons - That’s easy, don’t extract it - But it’s still in the “message” field - So you are saying that I can edit the original message field to be

```auto
2019-04-10,Frank

```

?

And if that is the case, in order to save storage as I’ve already extracted all the info I need into separate columns, could I completely remove the “message” field, or completely blank it out?

---

<div class="post-metadata">

**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/benvanstaveren/32/3398_2.png) [@benvanstaveren](https://community.graylog.org/u/benvanstaveren)\
**Post date:** [April 10, 2019, 6:50am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/8 "2019-04-10T06:50:33Z")

</div>

If the message that is sent in to Graylog (let’s say to a raw TCP socket) is in fact `2019-04-10,Frank,Password123` then when Graylog has ingested it, internally the message is now:

`{ "timestamp": "2019-04-10 08:39:00.000Z", "message": "2019-04-10,Frank,Password123", "source": "the-host-that-logged-it"}`

(Plus some additional fields such as the `gl2_*` set that helps Graylog do smart things like being able to show all messages from an input)

Ideally what you would do is set up a Pipeline, not an extractor, something along the lines of this:

```
rule "parse message and discard password"
when
  true
then
   let res = split(",", to_string($message.message));
   set_field("username", res[1]);
   set_field("message", concat(to_string(res[0]), ","));  
   set_field("message", concat(to_string($message.message), to_string(res[1]));
end

```

The reason for the repeated concat is that while a `regex_replace` function exists (which could do it in one line) I haven’t found it’s usage documentation on the Graylog site.

But, in essence, you can slice and dice the message, then put it back in a different form if you wish. I use this all the time on nginx logs - we send them in JSON format, a pipeline parses them and extracts the fields, then re-assembles a message field to look like a standard common log format so that some external legacy tools we use don’t get too upset 🙂

`

---

<div class="post-metadata">

**Author:** ![lindonm](https://avatars.discourse-cdn.com/v4/letter/l/b19c9b/32.png) [@lindonm](https://community.graylog.org/u/lindonm)\
**Post date:** [April 12, 2019, 7:22am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/9 "2019-04-12T07:22:42Z")

</div>

Many thanks Ben 😄

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [April 26, 2019, 7:24am UTC](https://community.graylog.org/t/replace-string-in-field-with-pipeline-rule/9807/10 "2019-04-26T07:24:22Z")

</div>

Closed on April 26, 2019, 7:24am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.