How to apply my custom extractors for incoming messages?


(Blason) #1

Hi Team,

I have messages coming in without being parsed hence I wrote my custom Grok parsers but then wanted to apply in Graylog for those messages.

Can someone help me with this?

TIA
Blason R


(Jan Doberstein) #2

Did you wrote a Grok pattern that does not work? Did you have a pattern that should only match to specific messages?

Your request is not clear!


(Blason) #3

My bad for not being specific.

Let me start, I have below logs coming in Graylog from filebeat
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] connect
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: GET /malware.doc HTTP/1.1
[2018-11-12 08:39:33] [2250] [http_80_tcp 2642] [192.168.5.103:49499] connect
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Host: 192.168.5.43
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Connection: keep-alive
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Upgrade-Insecure-Requests: 1
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: DNT: 1
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Accept-Encoding: gzip, deflate
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] recv: Accept-Language: en-GB,en-US;q=0.9,en;q=0.8,hi;q=0.7
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] info: Request URL: http://192.168.5.43/malware.doc
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] info: Sending fake file configured for extension ‘doc’.
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: HTTP/1.1 200 OK
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: Content-Length: 26624
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: Date: Mon, 12 Nov 2018 03:09:33 GMT
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: Content-Type: application/word
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: Connection: Close
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] send: Server: Microsoft-IIS/8.0
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] info: Sending file: /var/lib/inetsim/http/fakefiles/sample.doc
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] stat: 1 method=GET url=http://192.168.5.43/malware.doc sent=/var/lib/inetsim/http/fakefiles/sample.doc postdata=
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] disconnect

Of course those are not being parsed hence I picked the message which is most relevant to me which is
[2018-11-12 08:39:33] [2250] [http_80_tcp 2641] [192.168.5.103:49498] info: Request URL: http://192.168.5.43/malware.doc

And wrote a parser for this. However wanted to know how do I apply my custom extractor/parser only to this specific log since there are other around 20-25 other lines as well. I save my custom extractor in extractor section and now I wanted to apply that so that can build dashboard on

  1. URI Pattern [http://192.168.5.43/malware.doc]
  2. Client host [[192.168.5.103:49498]
  3. And port client has connected [[http_80_tcp 2641]

TIA
Blason R


(Jan Doberstein) #4

without knowing your setup - but from what i can see I would advice using the processing pipelines to run that only on the given messages.

When formatting your post ( https://community.graylog.org/faq#format-markdown ) in a propper way you make it more easy for other to help you.


(Blason) #5

Surely thanks. Will take a note of it.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.