Order of GROK patterns changes result

aehdings · November 8, 2017, 5:57pm

Hi there,

given two stored patterns and if they are linked with “|”, one of the stored patterns matches a certain string and the other stored pattern does partly. Then the result depends on the order of the stored patterns. The result is only right if the fully matching pattern is mentioned last. Which also means its wrong in every other case. AND there is no “right order” as order breaks it for other inputs. Maybe I’m missing something here but if not; this would mean Graylog is no longer an option for my team as we depend on working with syslogs in this manner.

The main Question is: Does chaining nested GROK patterns with " | " not work with the current Graylog GROK engine or am I misunderstanding something.

Example :

imap-login: proxy(testuser): disconnecting 8.8.8.8 (Disconnected by client): user=<testuser>, method=PLAIN, rip=8.8.8.8, lip=127.0.0.1, TLS, session=<zlgOH3xdHQBScWJe>

Pattern = Result:
%{DOVECOT_PROXY1} = Nothing, no match (correct)
%{DOVECOT_PROXY2} = Full match. All fields filled (correct)
(%{DOVECOT_PROXY1}|%{DOVECOT_PROXY2}) = Full match. All fields filled (correct)
(%{DOVECOT_PROXY2}|%{DOVECOT_PROXY1}) = Broken / Quirky result

It is fully reproducible in the extractor setup/test if you take the following data and patterns

Full syslog message:
<22>1 2017-11-08T18:42:22+01:00 dovecot-proxy dovecot - - - imap-login: proxy(testuser): disconnecting 8.8.8.8 (Disconnected by client): user=, method=PLAIN, rip=8.8.8.8, lip=127.0.0.1, TLS, session=<zlgOH3xdHQBScWJe>

GROK Patterns:
(Taken from github)
DOVECOT_PROXY1 %{WORD:proto}-login: %{WORD:proxy}\(%{USERNAME}\): started %{WORD:proxy_start} to %{IPORHOST:proxyto_host}:%{POSINT:proxyto_port}: user=<(%{USERNAME}(@%{HOSTNAME})?)?>, method=%{WORD:method}, rip=%{IP:rip}, lip=%{IP:lip}(, %{WORD:crypto})?, session=<%{DATA:session}>

DOVECOT_PROXY2 %{WORD:proto}-login: %{WORD:proxy}\(%{USERNAME}\): %{WORD:conn_status} %{IPORHOST} \(%{DATA:status_message}\): user=<(%{USERNAME}(@%{HOSTNAME})?)?>, method=%{WORD:method}, rip=%{IP:rip}, lip=%{IP:lip}(, %{WORD:crypto})?, session=<%{DATA:session}>

jan · November 9, 2017, 7:08am

@aehdings

what is your exact question now?

aehdings · November 13, 2017, 11:01am

Sorry @jan . Edited and clarified the question

jan · November 13, 2017, 1:03pm

you might already notice that chained GROK Patterns need to be covered by brackets, as in the original GROK Source at Github can be seen:

github.com

matejzero/logstash-grok-patterns/blob/master/dovecot.grok#L67


# 27  imap-login: proxy(username): started proxying to 2.2.2.2:143: user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, TLS, session=<LGL0EwwQOQBOmTSo>
DOVECOT_PROXY1 %{WORD:proto}-login: %{WORD:proxy}\(%{USEROREMAIL}\): started %{WORD:proxy_start} to %{IPORHOST:proxyto_host}:%{POSINT:proxyto_port}: user=<(%{USERNAME}(@%{HOSTNAME})?)?>, method=%{WORD:method}, rip=%{IP:rip}, lip=%{IP:lip}(, %{WORD:crypto})?, session=<%{DATA:session}>
# Disconnecting
# 28  pop3-login: proxy(username): disconnecting 2.2.2.2 (Disconnected by server): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, TLS, session=<gg7JEwwQ6QDBTZ2t>
# 29  pop3-login: proxy(username): disconnecting 2.2.2.2 (Disconnected by server): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, session=<9J/3EwwQFwDZSF8F>
# 30  imap-login: proxy(username): disconnecting 2.2.2.2 (Disconnected by server): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, secured, session=<GKEBFAwQMgDBAgFf>
# 31  imap-login: proxy(username): disconnecting 2.2.2.2 (Disconnected by client: Connection reset by peer): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, TLS, session=<tk+T3O4PowDULq55>
# 32  pop3-login: proxy(username@example.com): disconnecting 2.2.2.2 (Disconnected by server): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, session=<9J/3EwwQFwDZSF8F>
DOVECOT_PROXY2 %{WORD:proto}-login: %{WORD:proxy}\(%{USEROREMAIL}\): %{WORD:conn_status} %{IPORHOST} \(%{DATA:status_message}\): user=<(%{USERNAME}(@%{HOSTNAME})?)?>, method=%{WORD:method}, rip=%{IP:rip}, lip=%{IP:lip}(, (session=<%{DATA:session}>|%{WORD:crypto}, session=<%{DATA:session}>|%{WORD:crypto}))?
DOVECOT_PROXY3 %{WORD:proto}-login: %{WORD:proxy}\(%{USEROREMAIL}\): %{WORD:conn_status} %{IP:rip}
DOVECOT_PROXY (%{DOVECOT_PROXY1}|%{DOVECOT_PROXY2}|%{DOVECOT_PROXY3})


### EXCEEDED
# Max number of connections is exceeded
#  33 imap-login: Maximum number of connections from user+IP exceeded (mail_max_userip_connections=50): user=<username>, method=PLAIN, rip=2.2.2.2, lip=5.5.5.5, secured, session=<at1XQPAPJABUFPIj>
DOVECOT_EXCEEDED %{WORD:proto}-login: %{DATA:conn_status} \(%{DATA:status_message}\): user=<(%{USERNAME:user})?>, method=%{WORD:method}, rip=%{IP:rip}, lip=%{IP:lip}(, (session=<%{DATA:session}>|%{WORD:crypto}, session=<%{DATA:session}>|%{WORD:crypto}))?


### LMTP logs
# 34  lmtp(32352): Disconnect from local: Successful quit
# 35  lmtp(32347): Connect from local
# 36  lmtp(username): iUi8BBUI2FRbfgAAA15QOA: msgid=<E1YKcnl-0001q3-UM@example.com>: saved mail to INBOX

aehdings · November 13, 2017, 2:10pm

@jan
I tried that already as i was comming from the “largest” pattern in the file you mentioned %{DOVECOT}.
If this is supposed to be the right syntax, it is not working as i described.

jan · November 13, 2017, 2:30pm

see line 67 of the document.

that is working only this way and in every implementation of GROK the same.

aehdings · November 13, 2017, 3:10pm

@jan
Seems we’re not on the same page. I was referring to line 84 of the document which includes line 67 (once again nested). The problem is the same

jan · November 13, 2017, 4:05pm

To make things easy to debug:

This is actually the http://grokdebug.herokuapp.com but that shows you the issue. How should grok know which is the correct pattern, if one already matches?

That type of patterns can not work in current GROK implementations. It might be in the past - but not in current ones.

system · November 27, 2017, 4:05pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Chained grok pattern issue Graylog Central (peer support)	1	979	December 15, 2017
Extractors - Grok with Regex get first matching string Graylog Central (peer support)	3	2134	August 28, 2017
Rules execution order in stages and changes visibility? Graylog Central (peer support)	4	1661	February 14, 2022
Nested Grok issues in a syslog extractor Graylog Central (peer support)	1	560	August 30, 2019
Grok multiple pattern parsing issue Graylog Central (peer support)	5	1798	March 29, 2019

Order of GROK patterns changes result

Related topics