Process IIS Logs with GROK patterns and Extractor (or Pipeline Rule)

[wolfgang.reidlinger]

Hi there,

trying to process IIS logs with GROK patterns.

Found this here in the forum but didn’t really worked.
https://community.graylog.org/t/sidecar-filebeat-and-iis-logs/7974/4

%{TIMESTAMP_ISO8601:datetime} %{IPV4:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} (?:-|"%{URIPATH:cs-uri-query}") %{INT:s-port} %{NOTSPACE:cs-username} %{IPV4:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken:int}

So we did some adoptions and now it works (also including IPv6 addresses).

Version for IPv4 addresses only:

%{TIMESTAMP_ISO8601:datetime} %{IPV4:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{GREEDYDATA:cs-uri-query} %{INT:s-port} %{NOTSPACE:cs-username} %{IPV4:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}

Version for IPv6 (also covering IPv4 addresses):

%{TIMESTAMP_ISO8601:datetime} %{NOTSPACE:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{GREEDYDATA:cs-uri-query} %{INT:s-port} %{NOTSPACE:cs-username} %{NOTSPACE:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}

Just keep in mind, that our IIS Log file has the following fields. If you are using other / additional fields you have to do some more adoptions!

date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken

Currently we are using the Extractor function of Graylog. Its kind of “expensive” because it goes over every incoming message on the Global Input. An alternative method would be an Pipeline with an Pipeline Rule including the mentioned GROK Pattern. Maybe someone got a good Pipeline Rule for parsing IIS Logs.

Many thanks!

[StefanAustin]

Hi, I usually work in a different editor, but maybe something like this could work:

[[rules]]
id = “Your ID”
title = “Your rule name”
description = “”“Description
“””
source = “”"
rule “your_rule_name”
when
true
then
set_fields (
fields: grok (
pattern:“%{TIMESTAMP_ISO8601:datetime} %{NOTSPACE:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{GREEDYDATA:cs-uri-query} %{INT:s-port} %{NOTSPACE:cs-username} %{NOTSPACE:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}”,
value: to_string($message.sepm_fields),
only_named_captures: true
)
);
end
“”"

[wolfgang.reidlinger]

@StefanAustin thanks, but the rule does not work when I’m trying to put a Message trough the Simulator!

• I did choose the right Stream and Message Input

rule "IIS Logs"
when
  true
then
set_fields (
  fields: grok (
    pattern:"%{TIMESTAMP_ISO8601:datetime} %{NOTSPACE:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{GREEDYDATA:cs-uri-query} %{INT:s-port} %{NOTSPACE:cs-username} %{NOTSPACE:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}",
    value: to_string($message.sepm_fields), only_named_captures: true
    )
);
end

I used this sample message (from the message field of the IIS Log input):

2023-02-15 14:55:16 fe80::d87e:7577:e195:4eda%3 POST /owa/auth.owa &CorrelationID=<empty>;&cafeReqId=ac451f55-5b44-4f87-8c52-533a0f956f47;&LogoffReason=NoCookiesGetOrE14AuthPost&encoding=; 443 superadmin fe80::d87e:7577:e195:4eda%3 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/109.0.0.0+Safari/537.36 https://lab007ex.lab.local/owa/auth/logon.aspx?replaceCurrent=1&reason=2&url=https%3a%2f%2flab007ex.lab.local%2fowa 302 0 0 3

[faen]

If I were importing IIS logs, I would use grok patterns in a pipeline rule, so very similar to what you are doing now. Can’t speak to the performance difference, since it’s all still grok-based. If there was a way to configure IIS to write log files with a FieldName:FieldData type format you could just use the Graylog key_value function to parse the data. Thanks again for posting!

[StefanAustin]

If I use Test grok patterns the pattern works, but if you look closely, you see if you copy the message, there are hidden chars in it.

Example: fe80::d87e:7577:e195:4eda%3
you see the %3? That is maybe the issue.

A different reason could be, that you do not route the log to this rule.

I would use %{IP:s-ip}

[wolfgang.reidlinger]

Okey thanks for this hin, I now replaced

%{NOTSPACE:s-ip} %{NOTSPACE:c-ip}

with

%{IP:s-ip} %{IP:c-ip}

So the following GROK pattern did work for IPv4 and IPv6.

%{TIMESTAMP_ISO8601:datetime} %{IP:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{GREEDYDATA:cs-uri-query} %{INT:s-port} %{NOTSPACE:cs-username} %{IP:c-ip} %{NOTSPACE:cs-useragent} %{NOTSPACE:referer} %{NUMBER:sc-status} %{NUMBER:sc-substatus} %{NUMBER:sc-win32-status} %{NUMBER:time-taken}

Used Sample Data:

2023-02-13 12:13:26 10.10.66.7 GET /owa &CorrelationID=<empty>;&cafeReqId=39fc6472-22c9-4a35-88cf-c39b48572181;&LogoffReason=NoCookiesGetOrE14AuthPost&encoding=; 443 - 10.20.10.23 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/109.0.0.0+Safari/537.36+Edg/109.0.1518.78 - 302 0 0 3
2018-07-03 02:26:23 fe80::7c96:1423:9831:a344%3 POST /powershell clientApplication=ActiveMonitor;PSVersion=5.1.14393.1884&sessionID=Version_15.1_(Build_1260.0)=rJqNiZqNgbO+vc/PyLqn0ZOendGTkJyek4HOxsvNz8nNzM7Igc3PzsfSz8jSz8yrz83FzMnFzcs=&CorrelationID=<empty>;&cafeReqId=d5873914-04df-4463-863f-2904a45f13f3; 80 - fe80::7c96:1423:9831:a344%3 Microsoft+WinRM+Client - 200 0 0 1

Is seams that the mentioned %3 is part of the logfile when IPv6 is used…