Help with extractors

Graylog Central (peer support)
, , ,
1

I’ve read the Extractors documentation a couple of times, but I’d appreciate some additional assistance.

I get Windows Eventlogs into Graylog using Windows Event Forwarding, almost everything works fine, but I have one Eventlog where the full_message contains (such as):

Time : 19. 05. 1918 19:11
Entry Location : HKLM\Software\Classes*\ShellEx\PropertySheetHandlers
Entry : OLE Docfile Property Page
Enabled : enabled
Category : Explorer
Profile : NT AUTHORITY\SYSTEM
Description : OLE DocFile Property Page
Signer : (Verified) Microsoft Windows
Company : Microsoft Corporation
Image Path : c:\windows\system32\docprop.dll <file:///c:/windows/system32/docprop.dll>
Version : 10.0.17134.1
Launch String : HKCR\CLSID{3EA48300-8CF6-101B-84FB-666CCB9BCD32}
VT detection : 0|70
VT permalink : VirusTotal https://www.virustotal.com/file/e7b9e572c756fa36ec154401dc5beb319eecd394f051d786d7eb8329b578fc14/analysis/
MD5 : BB1729B0AB7912D3E4A2FCF934C79C79
SHA-1 : FB6C411009B06DB2F4B0B8711783C005CD8050A5
PESHA-1 : 208028A892B5BE9274318602BC6CB9C61C6074FF
PESHA-256 : 3DB0EEC32C9F634CCEB2352583C495486B3AE1D4E73485EC6C3BB11CB65471F6
SHA-256 : E7B9E572C756FA36EC154401DC5BEB319EECD394F051D786D7EB8329B578FC14
IMP : 627AEDCEB4C24CADF889C7BC2C0BD623

I’d ideally like to have each line as a separate field and everything after : as the data.

How, if possible provide the code, example, would I go about doing so?

2

Unfortunately, I haven’t solution for this, but please share the Windows and graylog side settings how you do that. I’m also interested in it.
//It is a modified message? At the first line, the date is so interesting.

3

You can use below grok pattern to extract out the required information from message and set them as fields using graylog pipeline set_fields function. To test the effect of this grok pattern processing on the message you can use grokdebug.herokuapp.com

%{WORD} : %{GREEDYDATA:time}\n%{DATA} : %{GREEDYDATA:entry_location}\n%{WORD} : %{GREEDYDATA:entry}\n%{WORD} : %{WORD:enabled}\n%{WORD} : %{WORD:category}\n%{WORD} : %{DATA:profile}\n%{WORD} : %{DATA:description}\n%{WORD} : %{DATA:signer}\n%{WORD} : %{DATA:company}\n%{DATA} : %{DATA:image_path}\n%{WORD} : %{GREEDYDATA:version}\n%{DATA} : %{GREEDYDATA:launch_string}\n%{DATA} : %{GREEDYDATA:vt_detection}\n%{DATA} : %{GREEDYDATA:vt_permalink}\n%{WORD} : %{DATA:MD5}\n%{DATA} : %{DATA:SHA_1}\n%{DATA} : %{DATA:PESHA_1}\n%{DATA} : %{DATA:PESHA_256}\n%{DATA} : %{DATA:SHA_256}\n%{WORD} : %{DATA:IMP}

I hope this helps.

1 Like
4

Hmmm…
I think for a more general solution, but it should work also.
But if you use grok just for extract, I suggest use extractor instead of pipeline.

5

Thank you both. @anmolsharma, the provided pattern appears to be just what I needed, I tested it using https://grokdebug.herokuapp.com and it seems fine.

I wanted to go with extractors as well, since they seem to be much easier to work with then the pipeline, but receive “We were not able to run the grok extraction. Please check your parameters.”

If I test with just %{WORD} : %{GREEDYDATA:time} it works, if I test with %{WORD} : %{GREEDYDATA:time}\n%{DATA} I get the error.

6

how did you ingest the messages from windows to graylog?

  • when using winlogbeat all fields will be already seperated without processing power of Graylog
  • when using nxlog with GELF all fields can be already seperated without processing power of Graylog

You might want to rethink how you ingest the messages to save ressources.

7

@jan we’re using Windows Event Forwarding from the endpoints to a central server, then we use nxlog to forward all the messages to Graylog.

8

Sounds like you’ll need to build a custom GROK pattern for this… could work, as long as all those lines are part of one and the same “message” field…

EDIT :
NVM, late to the party :smiley:

Windows Event Forwarding

@CypherBit, I assume you mean this method, right? →

9

what Input did you use in Graylog to receive the data from nxlog?

  • Syslog is unstructured and should not be used
  • GELF is structured and will create single fields without processing when the data is prepared

Some Simple configuration with a GELF UDP Input on Graylog might already solve your problems …

## C:\Program Files (x86)\nxlog\conf\nxlog.conf

## Extensions ##
<Extension _gelf>
    Module      xm_gelf
</Extension>

## INPUTS ##
<Input in>
    Module      im_msvistalog
    # For windows 2003 and earlier use the following:
    #Module      im_mseventlog
</Input>

## OUTPUTS ##
<Output out>
    Module      om_udp    
    Host        YOUR_GRAYLOG_IP
    Port        12201
    #Exec       to_syslog_snare();
    OutputType  GELF 
</Output>

## ROUTE  ##
<Route 1>
    Path in => out
</Route>
1 Like
10

@jan, yes, that is how we do it with GELF, all other Windows Logs have the correct structure apart from these: https://github.com/palantir/windows-event-forwarding/tree/master/AutorunsToWinEventLog

11

Hello everyone, as mentioned in the original post, I think an extractor and a working GROK pattern would be the way to go. I just can’t get it to work. The one @anmolsharma provided was probably close, but I was getting ““We were not able to run the grok extraction. Please check your parameters.””

12

I’m worried this topic will be closed before I have a working solution or at least something close to it that I can adapt.

Are Grok paterns the way to go, if so, I’d appreciate a bit more assistance, as mentioned @anmolsharma was pretty close.

13

We won’t solve it, it is your task.
We helped and showed a working way.
You have to understand the way and change it to your needs.
//Or leave it.

You had 2 weeks for debugging, learning, understanding, etc.
What part of the message what you can’t process?

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.