# Regex Matching in Pipeline or Extractor

**URL:** https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207
**Category:** Graylog Central (peer support)
**Created:** [January 10, 2023, 12:13am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207 "2023-01-10T00:13:26Z")
**Posts on this page:** 19
**Page:** 1

<div class="post-metadata">

### Author: ![bluescreenofwin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/bluescreenofwin/32/2859_2.png) [@bluescreenofwin](https://community.graylog.org/u/bluescreenofwin)
#### Post date: [January 10, 2023, 12:13am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/1 "2023-01-10T00:13:26Z")

</div>

Hi there,

I am running Graylog 4.3.3 and I am running into issues with adding a field to a log. I have tried both extraction and a pipeline using 2 different regex patterns.

When testing input extraction, I can test both regex patterns and get the "Matches!’ response yet the field does not get created. I tried moving this workflow to a pipeline rule instead and I get the same result. I can verify I get matches in the pipeline rule yet no field gets created there either.

**First attempt:**

```auto
rule "ESXi Verbosity Matching Hostd"

when
  regex("\\bHostd\\b",to_string($message.message)).matches==true

then
  let verbosity = regex("^.*Hostd: \\s*(\\w+)",to_string($message.message));

set_field("log_verbosity", verbosity);

end

```

**Second attempt:**

```auto
rule "ESXi Verbosity Matching Hostd"

when
  regex("\\bHostd\\b",to_string($message.message)).matches==true

then
  let verbosity = regex("Hostd:\\W*(\\w+)",to_string($message.message));

set_field("log_verbosity", verbosity);

end

```

---

<div class="post-metadata">

### Author: ![bluescreenofwin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/bluescreenofwin/32/2859_2.png) [@bluescreenofwin](https://community.graylog.org/u/bluescreenofwin)
#### Post date: [January 10, 2023, 12:14am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/2 "2023-01-10T00:14:45Z")

</div>

Here is an example log:

```auto
herpa.derpa.corpo.lab Hostd: verbose hostd[2103629] [Originator@6876 sub=Libs opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c] VigorTransport_ClientSendRequest: opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c seq=536204665: Sending GuestStats.SetNotificationTime request.

```

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 10, 2023, 1:08am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/3 "2023-01-10T01:08:36Z")

</div>

Hey @bluescreenofwin

I uploaded you log file to my lab. did a quick regex check/configuration.  
here is my findings.  
Regex extractor config.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/96f26aa33d5ca73b85e8bd510fd58e6846741bed.png)

results.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/520514f9d5916de791ec940b0c19732a544ca864.png)

That being shown, I think you regex is not configured correctly.

---

<div class="post-metadata">

### Author: ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)
#### Post date: [January 10, 2023, 3:04am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/4 "2023-01-10T03:04:50Z")

</div>

Looks like your regex is hunting for the second item (or in the case of regex starting at ‘0’, the 1st item) so tack a `["1"]` on the end…

```auto
rule "ESXi Verbosity Matching Hostd"

when
    regex("\\bHostd\\b",to_string($message.message)).matches==true
then

    let verbosity = regex("^.*Hostd: \\s*(\\w+)",to_string($message.message))["1"];
    set_field("log_verbosity", verbosity);

end

```

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 10, 2023, 3:40am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/5 "2023-01-10T03:40:56Z")

</div>

hey @tmacgbay

I said" Self, give it a try" so Ive been working on this, for some reason i cant even get this to work.

Check System/Configurations was correct.  
Check using different “Stages”

What I have done also was us this piepline.

```auto
rule "batman"
when
    regex("\\bHostd\\b",to_string($message.message)).matches==true
then

    let verbosity = regex("^.*Hostd: \\s*(\\w+)",to_string($message.message))["1"];
    set_field("log_verbosity", verbosity);
    debug(verbosity);
end

```

Results:

```auto
2023-01-09 21:27:48,997 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 21:27:49,153 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 21:27:50,920 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 21:27:50,920 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 21:27:50,920 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 21:32:11,366 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,367 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,374 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,378 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,376 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,374 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,382 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-09 21:32:11,382 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.

```

I used this piepline earlier:

```auto
rule "ESXi Verbosity Matching Hostd"
when
  has_field("message") 
then
  let verbosity = regex("\\[(.*?)]\\)",to_string($message.message));
  set_field("httpd",verbosity["1"]); <--- I also used "0" for testing purposes.
  debug(verbosity);
end

```

Results:

```auto
2023-01-09 20:48:19,929 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:19,929 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:19,929 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:28,034 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:28,034 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:29,681 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:29,681 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:29,682 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:29,683 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:30,041 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:30,041 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:30,041 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:30,065 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:30,503 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:31,127 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:32,837 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:32,838 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:32,838 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:39,774 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:39,774 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:39,776 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:39,776 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:43,036 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:43,036 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:43,039 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:43,039 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-09 20:48:43,040 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}

```

Now Im over here trying to get it to work 😆

---

<div class="post-metadata">

### Author: ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)
#### Post date: [January 10, 2023, 2:32pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/6 "2023-01-10T14:32:26Z")

</div>

Hmmm… There is one “Match” and one “Group” that shows in regex101. I had thought since the Group was after the match that it would naturally pick it up.

You can swap the grouping around and make the first part a Positive Look-Behind `(?<=...)`… Which in reality is what you want… the thing that follows `Hostd:`… then grab the first thing with zero (?) maybe you don’t need that… …this way there is ONLY a match to work with. A regex like such:

`(?<=Hostd:\\s)\\S+`

Which would make the rule:

```auto
rule "ESXi Verbosity Matching Hostd"

when
    regex("\\bHostd\\b",to_string($message.message)).matches==true
then

    let verbosity = regex("(?<=Hostd:\\s)\\S+",to_string($message.message))["0"];
    set_field("log_verbosity", verbosity);

end

```

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 10, 2023, 11:30pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/7 "2023-01-10T23:30:16Z")

</div>

Hey,

I gave it a try, its a "no Go’. I’m wondering if it’s the log file. Dont mind the ERROR, I’ve been try different Input/type out to see if it makes a difference.

Results:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/8/819287336985f166602f6541d1fbd179985332dd.png)

```auto
2023-01-10 16:43:43,886 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,887 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,887 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,887 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,888 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Unable to decode raw message RawMessage{id=3c9383d0-9138-11ed-a56e-0242ac110004, messageQueueId=8623604, codec=gelf, payloadSize=153, timestamp=2023-01-10T22:43:43.885Z, remoteAddress=/10.10.10.10:55136} on input <62db5ffc81f9e61ab57272bc>.
2023-01-10 16:43:43,888 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,888 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,889 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 16:43:43,890 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=3c9383d0-9138-11ed-a56e-0242ac110004, messageQueueId=8623604, codec=gelf, payloadSize=153, timestamp=2023-01-10T22:43:43.885Z, remoteAddress=/10.10.10.10:55136}
java.lang.IllegalArgumentException: GELF message <3c9383d0-9138-11ed-a56e-0242ac110004> (received from <10.10.10.10:55136>) has empty mandatory "short_message" field.
        at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:263) ~[graylog.jar:?]
        at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:141) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:156) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:95) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
2023-01-10 16:43:43,891 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.

```

Not sure whats up, I do know my Regex extractor above works.

---

<div class="post-metadata">

### Author: ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)
#### Post date: [January 11, 2023, 12:40am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/8 "2023-01-11T00:40:33Z")

</div>

What is $message.message coming in as?

The test message is:  
`herpa.derpa.corpo.lab Hostd: verbose hostd[2103629] [Originator@6876 sub=Libs opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c] VigorTransport_ClientSendRequest: opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c seq=536204665: Sending GuestStats.SetNotificationTime request.`

Here it is in regex101: [regex101: build, test, and debug regex](https://regex101.com/r/WfhNLt/1)

To see the message coming in…

```auto
debug(concat("++++message :",$message.message));
debug(concat("++++verbosity:",verbosity));

```

Maybe take the `["0"]` out to see results

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2023, 12:50am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/9 "2023-01-11T00:50:13Z")

</div>

> [@tmacgbay](#):
>
> What is $message.message coming in as?

I uploaded the example logs from this post.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c09c6eaee5cf218dd049cd29e28af1339f485b5b.png)

```auto
message
2023-01-09 20:36:09 herpa.derpa.corpo.lab Hostd: verbose hostd[2103629] [Originator@6876 sub=Libs opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c] VigorTransport_ClientSendRequest: opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c seq=536204665: Sending GuestStats.SetNotificationTime request.

```

> [@tmacgbay](#):
>
> To see the message coming in…
> 
> ```auto
> debug(concat("++++message :",$message.message));
> debug(concat("++++verbosity:",verbosity));
> 
> ```
> 
> Maybe take the `["0"]` out to see results

Ill give that a try, kind of confused why I cant get it to work. I’m using GL Operations 4.4.x with Elasticsearch 7.10, MongoDb 4.4

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2023, 1:21am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/10 "2023-01-11T01:21:43Z")

</div>

@tmacgbay

Lab testing results;  
Log file results:  
debug(concat(“++++message :”,$message.message));

```auto

2023-01-10 19:13:31,930 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: message2023-01-09 20:36:09 herpa.derpa.corpo.lab Hostd: verbose hostd[2103629] [Originator@6876 sub=Libs opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c] VigorTransport_ClientSendRequest: opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c seq=536204665: Sending GuestStats.SetNotificationTime request.
2023-01-10 19:13:31,930 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: message2023-01-09 20:36:09 herpa.derpa.corpo.lab Hostd: verbose hostd[2103629] [Originator@6876 sub=Libs opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c] VigorTransport_ClientSendRequest: opID=laukhmm4-1205666-auto-puar-h5:70342567-86-01-58-fd8c seq=536204665: Sending GuestStats.SetNotificationTime request.

```

debug(concat(“++++verbosity:”,verbosity));

```auto
2023-01-10 19:15:47,973 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,973 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,973 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}
2023-01-10 19:15:47,973 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}
2023-01-10 19:15:47,975 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,975 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}
2023-01-10 19:15:47,974 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: {}
2023-01-10 19:15:47,975 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity{}

```

GUI results:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/30932fc3b30093c8e87333934450fd6df6b55694.png)

**EDIT** : those tests above I have removed `["0"]`.

**EDIT2** : This is with the `["0"]`.

```auto
2023-01-10 19:29:32,005 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,005 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity
2023-01-10 19:29:32,006 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,006 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity
2023-01-10 19:29:32,007 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,007 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity
2023-01-10 19:29:32,007 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,007 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,008 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity
2023-01-10 19:29:32,008 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: Passed value is NULL.
2023-01-10 19:29:32,008 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity
2023-01-10 19:29:32,008 INFO : org.graylog.plugins.pipelineprocessor.ast.functions.Function - PIPELINE DEBUG: log_verbosity

```

**EDIT2.0** :

Ok so something new. The difference between an Extracotor and Pipeline minus the `["0"]` with the same logs/messages. 🤔

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/6/607179d21e370025727c9dc58765f2b04b971ad4.png)

I used this REGEX for the extractor.

`Hostd+\:+(\D+)\s`  
It can also work with `Hostd:(\D+)\s`

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/57f16eab2ab5f088b8581dfbaddbe2236627a7db.png)

---

<div class="post-metadata">

### Author: ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)
#### Post date: [January 11, 2023, 2:28pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/11 "2023-01-11T14:28:05Z")

</div>

`regex("(Hostd:\\s)(\\S+)",to_string($message.message))["1"];`

Tested.

The bones were there. 😑

---

<div class="post-metadata">

### Author: ![bluescreenofwin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/bluescreenofwin/32/2859_2.png) [@bluescreenofwin](https://community.graylog.org/u/bluescreenofwin)
#### Post date: [January 11, 2023, 5:30pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/12 "2023-01-11T17:30:26Z")

</div>

Still catching up on the thread. Appreciate the follow-up gentlefolk.

```auto
regex("(Hostd:\\s)(\\S+)",to_string($message.message))["1"];

```

I’ll throw this into my pipeline and test. Think seperating the different elements was the difference?

---

<div class="post-metadata">

### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)
#### Post date: [January 11, 2023, 11:29pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/13 "2023-01-11T23:29:57Z")

</div>

@tmacgbay Guess what I forgot 😄 during my extensive testing. the field " log\_verbosity" was already create so it was throwing errors, I used a new field calle " hostd",

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d81cc368ff833c3cb633060e1c3ce4cbf1d124a8.png)

Awesome and thanks @tmacgbay ( AKA “Batman”)

---

<div class="post-metadata">

### Author: ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)
#### Post date: [January 12, 2023, 3:57pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/14 "2023-01-12T15:57:42Z")

</div>

The `(...)` is a capturing group which is different than regex matching… A capturing group shows that you specifically want that data returned. I am not exactly clear on how Graylog/java handles regex in this case… you would think what you originally had would have worked… smarter minds than me could explain why. :\

---

<div class="post-metadata">

### Author: ![bluescreenofwin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/bluescreenofwin/32/2859_2.png) [@bluescreenofwin](https://community.graylog.org/u/bluescreenofwin)
#### Post date: [January 12, 2023, 8:09pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/15 "2023-01-12T20:09:32Z")

</div>

Appreciate all the hard work @tmacgbay and @gsmith. Funny enough, the extractor version of the above started working randomly today, but only for some logs, and I was not able to get any of the above to work within the pipelines. I decided just to move the workload to rsyslogd and call it a day. Give that I have another thread open with a similar issue right now… Graylog seems to be kind of buggy when it comes to pipelines/rules and has very questionable regex adoption. 😕

---

<div class="post-metadata">

### Author: ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)
#### Post date: [January 13, 2023, 9:04am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/16 "2023-01-13T09:04:45Z")

</div>

I hate regex as much as I love Grok. Why don’t give a try with Groks?  
Grokpattern needed beforehand:

```auto
DATA_ALL_BUT_SPACE [^]+
DATA_ALL_BUT_CLOSED_CORNERED_BRACKET [^]]+
DATA_ALL_BUT_OPENED_CORNERED_BRACKET	[^[]+

```

Now build a grok saved as myMagic using those to get the string done:  
`%{DATA_ALL_BUT_SPACE:source} Hostd: %{DATA_ALL_BUT_OPENED_CORNERED_BRACKET:host}]%{DATA_ALL_BUT_CLOSED_CORNERED_BRACKET:some_id}]`

and we have those fields:

```auto
{
  "source": "herpa.derpa.corpo.lab",
  "host": "verbose hostd",
  "some_id": "2103629"
}

```

And not build a pipeline parsing it:

```auto
rule "parse_myMagic"
when
  //what ever reason applies
then
  set_fields(
    grok(
      pattern:"^%{myMagic}",
      value:to_string($message.message),
      only_named_captures:true
    )
  );
end

```

and you should be done.

---

<div class="post-metadata">

### Author: ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)
#### Post date: [January 17, 2023, 8:55am UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/17 "2023-01-17T08:55:37Z")

</div>

Hi @bluescreenofwin  
did you try my suggestion? I’d be happy to hear if it’s working for you.

---

<div class="post-metadata">

### Author: ![bluescreenofwin](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/bluescreenofwin/32/2859_2.png) [@bluescreenofwin](https://community.graylog.org/u/bluescreenofwin)
#### Post date: [January 17, 2023, 5:34pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/18 "2023-01-17T17:34:57Z")

</div>

Hi @ihe,

I’ll give it a shot and report back.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)
#### Post date: [January 31, 2023, 5:35pm UTC](https://community.graylog.org/t/regex-matching-in-pipeline-or-extractor/27207/19 "2023-01-31T17:35:08Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.