# Help with extractors

**URL:** <https://community.graylog.org/t/help-with-extractors/8212>\
**Category:** Graylog Central (peer support)\
**Tags:** sidecar, nxlog, winlogbeat, nodatanx\
**Created:** [December 29, 2018, 11:12am UTC](https://community.graylog.org/t/help-with-extractors/8212 "2018-12-29T11:12:33Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [December 29, 2018, 11:12am UTC](https://community.graylog.org/t/help-with-extractors/8212/1 "2018-12-29T11:12:33Z")

</div>

I’ve read the Extractors documentation a couple of times, but I’d appreciate some additional assistance.

I get Windows Eventlogs into Graylog using Windows Event Forwarding, almost everything works fine, but I have one Eventlog where the full\_message contains (such as):

> Time : 19. 05. 1918 19:11  
> Entry Location : HKLM\Software\Classes\*\ShellEx\PropertySheetHandlers  
> Entry : OLE Docfile Property Page  
> Enabled : enabled  
> Category : Explorer  
> Profile : NT AUTHORITY\SYSTEM  
> Description : OLE DocFile Property Page  
> Signer : (Verified) Microsoft Windows  
> Company : Microsoft Corporation  
> Image Path : c:\windows\system32\docprop.dll \<file:///c:/windows/system32/docprop.dll\>  
> Version : 10.0.17134.1  
> Launch String : HKCR\CLSID{3EA48300-8CF6-101B-84FB-666CCB9BCD32}  
> VT detection : 0|70  
> VT permalink : [VirusTotal](https://www.virustotal.com/file/e7b9e572c756fa36ec154401dc5beb319eecd394f051d786d7eb8329b578fc14/analysis/) [https://www.virustotal.com/file/e7b9e572c756fa36ec154401dc5beb319eecd394f051d786d7eb8329b578fc14/analysis/](https://www.virustotal.com/file/e7b9e572c756fa36ec154401dc5beb319eecd394f051d786d7eb8329b578fc14/analysis/)  
> MD5 : BB1729B0AB7912D3E4A2FCF934C79C79  
> SHA-1 : FB6C411009B06DB2F4B0B8711783C005CD8050A5  
> PESHA-1 : 208028A892B5BE9274318602BC6CB9C61C6074FF  
> PESHA-256 : 3DB0EEC32C9F634CCEB2352583C495486B3AE1D4E73485EC6C3BB11CB65471F6  
> SHA-256 : E7B9E572C756FA36EC154401DC5BEB319EECD394F051D786D7EB8329B578FC14  
> IMP : 627AEDCEB4C24CADF889C7BC2C0BD623

I’d ideally like to have each line as a separate field and everything after : as the data.

How, if possible provide the code, example, would I go about doing so?

---

<div class="post-metadata">

**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\
**Post date:** [December 29, 2018, 8:02pm UTC](https://community.graylog.org/t/help-with-extractors/8212/2 "2018-12-29T20:02:56Z")

</div>

Unfortunately, I haven’t solution for this, but please share the Windows and graylog side settings how you do that. I’m also interested in it.  
//It is a modified message? At the first line, the date is so interesting.

---

<div class="post-metadata">

**Author:** ![anmolsharma](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/anmolsharma/32/3456_2.png) [@anmolsharma](https://community.graylog.org/u/anmolsharma)\
**Post date:** [January 1, 2019, 11:12am UTC](https://community.graylog.org/t/help-with-extractors/8212/3 "2019-01-01T11:12:29Z")

</div>

You can use below grok pattern to extract out the required information from message and set them as fields using graylog pipeline [set\_fields](http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#set-fields) function. To test the effect of this grok pattern processing on the message you can use [grokdebug.herokuapp.com](https://grokdebug.herokuapp.com/)

```auto
%{WORD} : %{GREEDYDATA:time}\n%{DATA} : %{GREEDYDATA:entry_location}\n%{WORD} : %{GREEDYDATA:entry}\n%{WORD} : %{WORD:enabled}\n%{WORD} : %{WORD:category}\n%{WORD} : %{DATA:profile}\n%{WORD} : %{DATA:description}\n%{WORD} : %{DATA:signer}\n%{WORD} : %{DATA:company}\n%{DATA} : %{DATA:image_path}\n%{WORD} : %{GREEDYDATA:version}\n%{DATA} : %{GREEDYDATA:launch_string}\n%{DATA} : %{GREEDYDATA:vt_detection}\n%{DATA} : %{GREEDYDATA:vt_permalink}\n%{WORD} : %{DATA:MD5}\n%{DATA} : %{DATA:SHA_1}\n%{DATA} : %{DATA:PESHA_1}\n%{DATA} : %{DATA:PESHA_256}\n%{DATA} : %{DATA:SHA_256}\n%{WORD} : %{DATA:IMP}

```

I hope this helps.

---

<div class="post-metadata">

**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\
**Post date:** [January 1, 2019, 12:06pm UTC](https://community.graylog.org/t/help-with-extractors/8212/4 "2019-01-01T12:06:55Z")

</div>

Hmmm…  
I think for a more general solution, but it should work also.  
But if you use grok just for extract, I suggest use extractor instead of pipeline.

---

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [January 2, 2019, 9:05am UTC](https://community.graylog.org/t/help-with-extractors/8212/5 "2019-01-02T09:05:33Z")

</div>

Thank you both. @anmolsharma, the provided pattern appears to be just what I needed, I tested it using [https://grokdebug.herokuapp.com](https://grokdebug.herokuapp.com) and it seems fine.

I wanted to go with extractors as well, since they seem to be much easier to work with then the pipeline, but receive “We were not able to run the grok extraction. Please check your parameters.”

If I test with just %{WORD} : %{GREEDYDATA:time} it works, if I test with %{WORD} : %{GREEDYDATA:time}\n%{DATA} I get the error.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [January 6, 2019, 12:29pm UTC](https://community.graylog.org/t/help-with-extractors/8212/6 "2019-01-06T12:29:14Z")

</div>

how did you ingest the messages from windows to graylog?

- when using winlogbeat all fields will be already seperated without processing power of Graylog
- when using nxlog with GELF all fields can be already seperated without processing power of Graylog

You might want to rethink how you ingest the messages to save ressources.

---

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [January 6, 2019, 2:46pm UTC](https://community.graylog.org/t/help-with-extractors/8212/7 "2019-01-06T14:46:14Z")

</div>

@jan we’re using Windows Event Forwarding from the endpoints to a central server, then we use nxlog to forward all the messages to Graylog.

---

<div class="post-metadata">

**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\
**Post date:** [January 7, 2019, 6:49am UTC](https://community.graylog.org/t/help-with-extractors/8212/8 "2019-01-07T06:49:35Z")

</div>

> [@CypherBit](#):
>
> I’d ideally like to have each line as a separate field and everything after : as the data

Sounds like you’ll need to build a custom GROK pattern for this… could work, as long as all those lines are part of one and the same “message” field…

**EDIT :**  
NVM, late to the party 😃

> Windows Event Forwarding

@CypherBit, I assume you mean this method, right? →

> **[Use Windows Event Forwarding to help with intrusion detection - Windows Security](https://learn.microsoft.com/en-us/windows/security/threat-protection/use-windows-event-forwarding-to-assist-in-intrusion-detection)**
>
> Learn about an approach to collect events from devices in your organization. This article talks about events in both normal operations and when an intrusion is suspected.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [January 7, 2019, 12:08pm UTC](https://community.graylog.org/t/help-with-extractors/8212/9 "2019-01-07T12:08:25Z")

</div>

what Input did you use in Graylog to receive the data from nxlog?

- Syslog is unstructured and should not be used
- GELF is structured and will create single fields without processing when the data is prepared

* * *

Some Simple configuration with a GELF UDP Input on Graylog might already solve your problems …

```auto
## C:\Program Files (x86)\nxlog\conf\nxlog.conf

## Extensions ##
<Extension _gelf>
    Module xm_gelf
</Extension>

## INPUTS ##
<Input in>
    Module im_msvistalog
    # For windows 2003 and earlier use the following:
    #Module im_mseventlog
</Input>

## OUTPUTS ##
<Output out>
    Module om_udp    
    Host YOUR_GRAYLOG_IP
    Port 12201
    #Exec to_syslog_snare();
    OutputType GELF 
</Output>

## ROUTE ##
<Route 1>
    Path in => out
</Route>

```

---

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [January 9, 2019, 7:06pm UTC](https://community.graylog.org/t/help-with-extractors/8212/10 "2019-01-09T19:06:11Z")

</div>

@jan, yes, that is how we do it with GELF, all other Windows Logs have the correct structure apart from these: [https://github.com/palantir/windows-event-forwarding/tree/master/AutorunsToWinEventLog](https://github.com/palantir/windows-event-forwarding/tree/master/AutorunsToWinEventLog)

---

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [January 16, 2019, 9:16am UTC](https://community.graylog.org/t/help-with-extractors/8212/11 "2019-01-16T09:16:10Z")

</div>

Hello everyone, as mentioned in the original post, I think an extractor and a working GROK pattern would be the way to go. I just can’t get it to work. The one @anmolsharma provided was probably close, but I was getting ““We were not able to run the grok extraction. Please check your parameters.””

---

<div class="post-metadata">

**Author:** ![CypherBit](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@CypherBit](https://community.graylog.org/u/CypherBit)\
**Post date:** [January 24, 2019, 9:46pm UTC](https://community.graylog.org/t/help-with-extractors/8212/12 "2019-01-24T21:46:50Z")

</div>

I’m worried this topic will be closed before I have a working solution or at least something close to it that I can adapt.

Are Grok paterns the way to go, if so, I’d appreciate a bit more assistance, as mentioned @anmolsharma was pretty close.

---

<div class="post-metadata">

**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\
**Post date:** [January 25, 2019, 8:03am UTC](https://community.graylog.org/t/help-with-extractors/8212/13 "2019-01-25T08:03:55Z")

</div>

We won’t solve it, it is your task.  
We helped and showed a working way.  
You have to understand the way and change it to your needs.  
//Or leave it.

You had 2 weeks for debugging, learning, understanding, etc.  
What part of the message what you can’t process?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [February 8, 2019, 8:03am UTC](https://community.graylog.org/t/help-with-extractors/8212/14 "2019-02-08T08:03:55Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.