Strange grok behaviour


(Nick Rosier) #1

I’m trying to extract SSH messages with grok. Found some patterns online but I have problems with “nested” patterns.

I’ve found a simple case that shows the problem; these are the groks patterns:

SSHD_CONN_CLOSE	Connection closed by %{IP:sshd_client_ip}$
SSHD_DISCONNECT	Received disconnect from %{IP:sshd_client_ip} port %{NUMBER:sshd_port}:%{NUMBER:sshd_disconnect_code}: %{GREEDYDATA:sshd_disconnect_status}

SSHD_TEST	(%{SSHD_DISCONNECT}|%{SSHD_CONN_CLOSE})

This is a sample message I try to parse:

Received disconnect from 61.182.226.162 port 59597:11: Normal Shutdown, Thank you for playing [preauth]

When I use the pattern SSHD_DISCONNECT I get following (named for shorter output) expected results:

sshd_client_ip 61.182.226.162
sshd_disconnect_code 11
sshd_disconnect_status Normal Shutdown, Thank you for playing [preauth]
sshd_port 59597

When I use SSHD_TEST I get this:

sshd_disconnect_code 11
sshd_disconnect_status Normal Shutdown, Thank you for playing [preauth]
sshd_port 59597

As you can see sshd_client_ip is missing from the result. The only explanation I can find that it is also used the SSHD_CONN_CLOSE pattern. When I replace this pattern with another one that does not have the sshd_client_ip pattern I get it back as a result but some other information might not get extracted.

Why is this and how do I fix it? I thought once it matches a pattern it should stop and discard the next patterns when you use |


(Konrad Merz) #2

Hi Nick,

can you tell me which version of Graylog you are using? The grok library and we ourself did some changes to the grok behavior in the last versions. To get into your problems the version number is essential.


(Nick Rosier) #3

I’m running the last (stable) version: Graylog 2.5.1+34194da


(Konrad Merz) #4

Hey Nick,

this is a known issue.

Graylog 3.0 the situation will be slightly better, but still not satisfying.

I hope to find some time to fix that in a near future.


(Nick Rosier) #5

Hi Konrad,

thanks for the update; I will see if I can create a test instance with 3.0-RC1. The grok patterns I’m using were designed for Logstash IIRC so I was hoping they would work “out-of-the-box” in Graylog as well.


(Konrad Merz) #6

Hey Nick,

please let me how the 3.0 Grok Patterns work for you. So I can have a better view for future improvements. Since with 3.0 we should handle it like Logstash.


(Nick Rosier) #7

Hi Konrad,

I’ve installed 3.0 RC1; so far it’s looking better. Only “issue” is that null values also get returned:

sshd_client_ip
["xxx.xxx.xxx.xxx",null,null,null,null,null,null]

When you want to investigate the ip-address and click on the “magnification glass” it gets copied as client_ip:xxx.xxx.xxx.xxx,,,,,, which returns nothing. I have to remove the ,'s to get the result I want.


(Konrad Merz) #8

Hi Nick,

Okay thank you for your input. I will try to investigate the problem. Some user mentioned that Logstash would return the same result (with null). Can you confirm that? Otherwise I can better argue with the library maintainer, since there is no Grok Pattern standard it’s quite hard to find anything useful in the internet.

Thanks a lot for your feedback already!


(Nick Rosier) #9

I have no experience with Logstash. Using a commercial SIEM at work but wanted something @ home to collect the logs of all my servers and devices. Graylog seemed the best solution for me. But I find lots of grok-pattens written for Logstash so I try to recycle those for Graylog :smile:


(system) closed #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.