# GROK Extractor question
**URL:** https://community.graylog.org/t/grok-extractor-question/150
**Category:** Graylog Add-ons
**Tags:** pipeline-rules
**Created:** [February 20, 2017, 2:49pm UTC](https://community.graylog.org/t/grok-extractor-question/150 "2017-02-20T14:49:20Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 2:49pm UTC](https://community.graylog.org/t/grok-extractor-question/150/1 "2017-02-20T14:49:21Z")
Hello,
Loving the new forum setup
Anyway…I have a simple syslog stream coming in from a Maltrail Sensor. I am able to GROK the pattern for the src and dst ip, but how do I tell it to extract the fields that is always after the date and | ?
See below: after this field " 2017-02-19| " is the field that would be great to break out into it’s own extracted field.
All insight is appreciated
TP
```auto
Feb 20 06:09:36 MalTrail CEF:0|Maltrail|sensor|0.10.182|2017-02-19|tor exit node (suspicious)|0|src=192.241.160.32 spt=123 dst=10.0.0.190 dpt=123 trail=192.241.160.32 ref=blutmagie.de (+talosintelligence.com,torproject.org)
```
---
### Author:  [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [February 20, 2017, 3:08pm UTC](https://community.graylog.org/t/grok-extractor-question/150/2 "2017-02-20T15:08:39Z")
Hi Thomas,
you could give the Graylog CEF plugin a try: [https://marketplace.graylog.org/addons/b2c55194-a76e-4fd7-89fd-5421188bf33f](https://marketplace.graylog.org/addons/b2c55194-a76e-4fd7-89fd-5421188bf33f)
Cheers,
Jochen
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 3:35pm UTC](https://community.graylog.org/t/grok-extractor-question/150/3 "2017-02-20T15:35:59Z")
OK…I put it in, and I see messages hitting it…if i send a message, I see the Total change
Throughput / Metrics
1 minute average rate: 0 msg/s
Network IO: 0B 0B (total: 3.2KB 0B )
Empty messages discarded: 0
Yet I hit the Show Massages and it is blank
I’m guessing it’s a Timezone issue, but anything I put in there doesn’t seem to matter.
---
### Author:  [@jochen](https://community.graylog.org/u/jochen)
#### Post date: [February 20, 2017, 3:54pm UTC](https://community.graylog.org/t/grok-extractor-question/150/4 "2017-02-20T15:54:19Z")
Given that there are no error messages in the logs of your Graylog node, you could use an absolute time range and set the end point to some time in the future.
This should show messages with an invalid timestamp or invalid timezone information.
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 4:11pm UTC](https://community.graylog.org/t/grok-extractor-question/150/5 "2017-02-20T16:11:15Z")
Hmmm…tried that too…I event went 3 days into the future…no dice
While I try to figure this out, I would be interested in how to parse out the same on the regular syslog UDP input.
I assume there’s a way to tell it to grab that 6th area (see below again)
I will continue to explore the CEF, but any insight into how to grab that field would be great and something I could use elsewhere
Thanks
TP
Feb 20 10:07:36 MalTrail CEF:0|Maltrail|sensor|0.10.182|2017-02-19|sinkhole conficker (malware)|0|src=10.0.0.137 spt=- dst=136.161.101.53 dpt=- trail=136.161.101.53 ref=(static)
---
### Author:  [@derPhlipsi](https://community.graylog.org/u/derPhlipsi)
#### Post date: [February 20, 2017, 5:18pm UTC](https://community.graylog.org/t/grok-extractor-question/150/6 "2017-02-20T17:18:43Z")
Hey @ThomasPowers,
you could use following **GROK extractor** first:
`%{CEFHEADER:CEFHeader}\|%{DATA:Facility}\|%{DATA:Subfacility}\|%{DATA:version}\|%{DATA:date}\|%{DATA:value1}\|%{DATA:value2}\|%{GREEDYDATA:value3}`
After this add a **Copy-Input extractor** on field value3 (or whatever you rename it to), make it copy/cut to itself and add a **Key=Value converter** to the extractor.
Et voilà, you got your fields 🙂
Greets - Phil
**_PS:_**
I have no idea if the names I gave the fields make sense, so change them if needed 😃
GROK-Pattern for Header (Is it actually a header? Idk 😃 )
`CEFHEADER: %{MONTH:UNWANTED} %{MONTHDAY:UNWANTED} %{TIME:UNWANTED} %{DATA:UNWANTED} %{DATA:UNWANTED}:%{BASE10NUM:UNWANTED}`
---
### Author:  [@lennart](https://community.graylog.org/u/lennart)
#### Post date: [February 20, 2017, 6:10pm UTC](https://community.graylog.org/t/grok-extractor-question/150/7 "2017-02-20T18:10:59Z")
The reason why the CEF input does not work in your case is, that we are expecting a CEF message to be sent via syslog but your example message is not in syslog format. For example, the `PRI` field is missing.
I just reproduced this:
```auto
org.graylog.plugins.cef.parser.CEFParser$ParserException: This message was not recognized as CEF and could not be parsed.
at org.graylog.plugins.cef.parser.CEFParser.parse(CEFParser.java:71)
at org.graylog.plugins.cef.parser.CEFParserTest.testCustomParse1(CEFParserTest.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
```
I’m thinking about shipping a `parse_cef()` pipeline rule to allow more flexible parsing. Would you be willing to give it a try? Should be much easier than using GROK.
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 8:18pm UTC](https://community.graylog.org/t/grok-extractor-question/150/8 "2017-02-20T20:18:32Z")
> [@derPhlipsi](#):
>
> %{CEFHEADER:CEFHeader}|%{DATA:Facility}|%{DATA:Subfacility}|%{DATA:version}|%{DATA:date}|%{DATA:value1}|%{DATA:value2}|%{GREEDYDATA:value3}
Hey…this worked Great! THe splitting of the fields by the | was throwing me off…
Now…On to trying to get the dst= address to hit on the geolocation stuff
Thanks
TP
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 8:19pm UTC](https://community.graylog.org/t/grok-extractor-question/150/9 "2017-02-20T20:19:30Z")
I would try it…sure…but I have no idea what this rule would look like and where it would go
TP
---
### Author:  [@derPhlipsi](https://community.graylog.org/u/derPhlipsi)
#### Post date: [February 20, 2017, 8:25pm UTC](https://community.graylog.org/t/grok-extractor-question/150/10 "2017-02-20T20:25:44Z")
You need to use | because | (pipe) is used as OR-operator when using GROK, so you need to escape it with a backslash 🙂
I can give a little hint there as well 😃
Make sure that the GeoIP Resolver is last in the processing chain 😃
When you have set it up (added the IP-database and enabled it on the configuration page) it will automatically start to geotag every field having an IP-address 🙂
---
### Author:  [@derPhlipsi](https://community.graylog.org/u/derPhlipsi)
#### Post date: [February 20, 2017, 8:27pm UTC](https://community.graylog.org/t/grok-extractor-question/150/11 "2017-02-20T20:27:50Z")
> [@ThomasPowers](#):
>
> I would try it…sure…but I have no idea what this rule would look like and where it would go
Look [here](http://docs.graylog.org/en/2.2/pages/pipelines.html). 🙂
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 8:57pm UTC](https://community.graylog.org/t/grok-extractor-question/150/12 "2017-02-20T20:57:02Z")
The GeoResolver wasn’t working until I moved it to the bottom of the chain…thanks for the heads up on that.
TP
---
### Author:  [@derPhlipsi](https://community.graylog.org/u/derPhlipsi)
#### Post date: [February 20, 2017, 9:02pm UTC](https://community.graylog.org/t/grok-extractor-question/150/13 "2017-02-20T21:02:51Z")
> [@ThomasPowers](#):
>
> The GeoResolver wasn’t working until I moved it to the bottom of the chain…thanks for the heads up on that.
No problem 🙂
The GeoIP Resolver needs to be last because it can only check already existing fields for an IP-address, but if it is first in the processing chain there most certainly won’t be a field containing just an IP-address 😃
Greets - Phil
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 9:43pm UTC](https://community.graylog.org/t/grok-extractor-question/150/14 "2017-02-20T21:43:37Z")
OK Phil…since were on a role…what about when the beginning and end of a field isn’t as obvious as the | we have before
Lets say we want to grab info from here:
Primarily looking for the Alert Received Field, the ET rule in {}, the Source and Destination IPs
Still trying to wrap my head around how this thing defines starts and stops for the fields
Thanks
TP
onion sguil\_alert: 21:13:07 pid(24703) Alert Received: 0 3 misc-activity onion-eth1 {2017-02-20 21:13:07} 3 150963 {ET SCAN Behavioral Unusual Port 139 traffic, Potential Scan or Infection} 10.0.0.17 192.168.1.118 6 49478 139 1 2001579 14 320 320
---
### Author:  [@ThomasPowers](https://community.graylog.org/u/ThomasPowers)
#### Post date: [February 20, 2017, 9:58pm UTC](https://community.graylog.org/t/grok-extractor-question/150/15 "2017-02-20T21:58:22Z")
Actually…I think I got it:
Received: %{DATA:test}{%{DATA:date}} %{DATA:value1}{%{DATA:ETRule}} %{IP:DetSource} %{IP:DetDST}
Then I just UNWANTED what I don’t need
I’m slowly getting there
Thanks
TP
---
### Author:  [@derPhlipsi](https://community.graylog.org/u/derPhlipsi)
#### Post date: [February 20, 2017, 10:12pm UTC](https://community.graylog.org/t/grok-extractor-question/150/16 "2017-02-20T22:12:43Z")
The nice thing about GROK is that as long you have some chars or values that stay constant, you will be able to get that structure.
I guess you can look at the GROK-Pattern and compare it to the String you gave me. Basically you just need to think about using placeholders in the locations where the data that you want is located. But keep in mind that you should be as precise as possible, so don’t use patterns like DATA or GREEDYDATA if you can use more specific ones like IP, DATE, BASE10NUM or whatever 😃
If you dont need the `{2017-02-20 21:13:07} 3 150963` part just change the GROK-pattern of text1 to DATA 🙂
Well, I just saw that you did it yourself 😃
But you made one mistake that makes Graylog refuse the GROK-pattern. You didn’t escape the { } brackets. These are symbols used by GROK, so they need to be escaped when they are actually inside the string. Else Graylog thinks these brackets are part of a pattern it should parse. Your pattern would be this ````Received: %{DATA:test}{%{TIMESTAMP\_ISO8601:date}} %{DATA:value1}{%{DATA:ETRule}} %{IP:DetSource} %{IP:DetDST}```
Notice the `TIMESTAMP_ISO8601` pattern instead of `DATA` on your date field regarding being specific 🙂
And you don’t need to make patterns unwanted if you don’t want them (except they are needed to parse the text-structure correctly) . Just make sure you check Named captures only. Graylog will automatically ignore the rest, see:
Greets - Phil