Graylog - Modsecurity Audit Logs - Grokpattern

[afaqbabar]

Hi all, I want to extract fields from modsecurity audit log, which is multiline log with sections A,B,…Z. I can merge multiline message of a single event using multiline feature in Graylog input. The result is as below:

###############################

message

–12aceb4f-A–
[14/Dec/2018:13:00:02 +0100] XAsbQiwgSSXTTqKkD3L2LQGgAAI ::1 45476 ::1 80
–12abeb4f-B–
GET / HTTP/1.1
Host: localhost
User-Agent: Mozilla/5.0
Accept: application/xml,application/xhtml+xml,text/html
Connection: Close

–12aceb4f-F–
HTTP/1.1 503 Service Temporarily Unavailable
Status: 503 Service Temporarily Unavailable
Retry-After: 3600
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
Content-Length: 1130
Connection: close
Content-Type: text/html; charset=UTF-8

–12aceb4f-E–

#############################

I want to extract fields (such as Host, User-Agent, Status ) of all sections A, B … Z, of a modsecurity log. What is the most preferrable way to extract fields?

[macko003]

You can do on input with extractor, or someone prefer the pipelines.
You are able to use regexp or GROK to extract the fields.

[blason]

This is what my question is as well. I mean once we write extractor and save it in Graylog how do we apply this to incoming messages?

[benvanstaveren]

You don’t, once an extractor is active it will apply itself to any messages coming in on the input you’ve put the extractor on. For pipelines same story, pipeline will apply itself to any messages coming through a stream that the pipeline attaches to.

[macko003]

Have you tried?
http://docs.graylog.org/en/2.5/pages/extractors.html
http://docs.graylog.org/en/2.5/pages/pipelines/pipelines.html

[afaqbabar]

Hi, thanks for your reply. If I apply a Grok pattern %{GREEDYDATA} to the message, I get the following:
"
Extractor preview

GREEDYDATA

–12aceb4f-A–
"
It means only the first line is extracted. How to enable Grok pattern for multiple lines?

[afaqbabar]

Got it. I have to use (?s) to convert multiline into single line and then apply Grok Pattern:

(?s)%{GREEDYDATA}

Thanks