Grok pattern problem with Windows DNS Log

Graylog Central (peer support)
1

Hi,

I have a problem with my Grok pattern in order to translate my windows 2008R2 DNS log.

This is an example of log entries
07/07/2020 16:58:59 0DA0 PACKET 0000000002F84B90 UDP Rcv 10.0.25.139 3522 Q [0001 D NOERROR] A .clients.l.google.com.

When i’m applying this grok pattern, I have a error message :cry:
%{WINDNS_TIME:Time} +%{WINDNS_THREADID:ThreadID} +%{WINDNS_NOTSPACE:Context} +%{WINDNS_BASE16NUM:InternalID} +%{WINDNS_PROTOCOL:Protocol} +%{WINDNS_SNDRCV:SndRcv} +%{WINDNS_IP:IP} +%{WINDNS_XID:XID}%{WINDNS_QUERYRESP:QueryResp}%{WINDNS_OPCODE:Opcode} +[%{WINDNS_FLAGSHEX:FlagsHex}%{WINDNS_FLAGSCHAR:FlagsChar}%{WINDNS_RESPONSE:Response}]%{WINDNS_QTYPE:QType}%{WINDNS_NAME:Name}

-> Attention
We were not able to run the grok extraction. Please check your parameters.

Can U help me please ?

2

This is grok pattern ::

WINDNS_BASE16NUM (?<![0-9A-Fa-f])(?:[±]?(?:0x)?(?:[0-9A-Fa-f]+))

WINDNS_FLAGSCHAR (\s+[A T D R]{1,4}\s+) (\s+[A T D R]{1,4}\s+[A T D R]{1,4}\s+) (\s+[A T D R]{1,4}\s+[A T D R]{1,4}\s+[A T D R]{1,4}\s+) \s+

WINDNS_FLAGSHEX ([0-9]+)

WINDNS_IP (?<![0-9])(?:(?:25[0-5] 2[0-4][0-9] [0-1]?[0-9]{1,2})[.](?:25[0-5] 2[0-4][0-9] [0-1]?[0-9]{1,2})[.](?:25[0-5] 2[0-4][0-9] [0-1]?[0-9]{1,2})[.](?:25[0-5] 2[0-4][0-9] [0-1]?[0-9]{1,2}))(?![0-9]) ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4} :)) (([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4} ((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2}) :((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3}) :)) (([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3}) ((:[0-9A-Fa-f]{1,4})?:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4}) ((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5}) ((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6}) ((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)) (:(((:[0-9A-Fa-f]{1,4}){1,7}) ((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)(.(25[0-5] 2[0-4]\d 1\d\d [1-9]?\d)){3})) :)))(%.+)?

WINDNS_NAME (?:\s+.+ )

WINDNS_NOTSPACE \S+

WINDNS_OPCODE ([A-Z]{1})

WINDNS_PROTOCOL (UDP TCP)

WINDNS_QTYPE (?:\s\S+ )

WINDNS_QUERYRESP (\s+R\s+ \s+)

WINDNS_RESPONSE ([A-Z]+)

WINDNS_SNDRCV (Snd Rcv)

WINDNS_THREADID [a-zA-Z0-9]{4}

WINDNS_TIME (?:0?[1-9] 1[0-2])[/-](?:(?:0[1-9]) (?:[12][0-9]) (?:3[01]) [1-9])/-{1,2}\s(?!<[0-9])(?:2[0123] [01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9] 60)(?:[:.,][0-9]+)?)\s(A P)M

WINDNS_XID ([a-z0-9]{4})

3

Ok My WINDNS_TIME isn’t correct.
I have modify it wtith this syntax ::
(?:0?[1-9]|1[0-2])/-/-{1,2}\s(?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)

But it’s the same error message : Cannot add converters to Grok pattern extractors…

What is the problem ?!

4
  1. First, read guideline, how to paste content, especially code using ``` code ```
  2. You use grok which is in my opinion too specific, it’s better to use grok patterns already included in graylog:
WINDNS_BASE16NUM - is not necessary, use rather BASE16NUM
WINDNS_PROTOCOL (UDP|TCP) - correct one with | between options
WINDNS_SNDRCV (Snd|Rcv) - correct one with | between options
WINDNS_IP - is not necessary, use rather IP
  1. I edited your grok a bit:
    %{WINDNS_TIME:Time} +%{WINDNS_THREADID:ThreadID} +%{NOTSPACE:Context} +%{BASE16NUM:InternalID} +%{WINDNS_PROTOCOL:Protocol} +%{WINDNS_SNDRCV:SndRcv} +%{IP:IP} +%{WINDNS_XID:XID} %{WINDNS_OPCODE:Opcode} +\[%{WINDNS_FLAGSHEX:FlagsHex}%{WINDNS_FLAGSCHAR:FlagsChar}%{WINDNS_RESPONSE:Response}]%{WINDNS_QTYPE:QType}%{GREEDYDATA:dns_query_name}
  2. I created simplified version of your grok:
    %{WINDNS_TIME:Time} %{WORD:ThreadID} %{WORD:Context}%{SPACE}%{WORD:InternalID} %{WORD:Protocol} %{WORD:SndRcv} %{IP:IP}%{SPACE}%{WORD:XID}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}\[%{GREEDYDATA:FlagsHex}%{SPACE}%{WORD:Response}\]%{SPACE}%{WORD:QType}%{SPACE}%{GREEDYDATA:Name}
  3. Best way to debug grok is to use either graylog (one pattern at once and add) or onine grok editor:
    http://grokdebug.herokuapp.com/
1 Like
5

Thank You very much for your help.

But I think that my problem is in ES.

When I use your simplified version : I have the same issue on log :

2020-07-08T14:27:38.981+02:00 WARN [Messages] Failed to index message: index=<graylog_0> id=<68470eca-c116-11ea-a30a-005056bf4905> error=<{“type”:“mapper_parsing_exception”,“reason”:“failed to parse field [ThreadID] of type [long] in document with id ‘68470eca-c116-11ea-a30a-005056bf4905’”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:"For input string: “0DA0"”}}>

Why my field ThreatID is considered by a Long Type ?

I’m tying to modify this with this command :

curl -XPUT -H “Content-Type: application/json” localhost:9200/_template/graylog -d ’

{

“template”:“graylog*”,

“settings”:{

“index.refresh_interval”:“30s”

},

“mappings”:{

“message”:{

“properties”:{

“ThreadID”:{

“index”:“true”,

“type”:“keyword”

}

}

}

}

}’

And when I’m checking after that :

curl localhost:9200/_template/graylog

{“graylog”:{“order”:0,“index_patterns”:[“graylog*”],“settings”:{“index”:{“refresh_interval”:“30s”}},“mappings”:{“message”:{“properties”:{“ThreadID”:{“index”:“true”,“type”:“keyword”}}}},“aliases”:{}}}

The type of field is “keyword” …. ?!

I don’t understand why ES find this fied as “Long” type ?!

1 Like
6

You can specify type in grok pattern like this:
%{WORD:ThreadID;string}

https://docs.graylog.org/en/3.3/pages/extractors.html#using-grok-patterns-to-extract-data

7

Already try this (force type into grok pattern) but same problem ;’-(

%{WINDNS_TIME:Time} %{WORD:ThreadID;string} %{WORD:Context}%{SPACE}%{WORD:InternalID} %{WORD:Protocol} %{WORD:SndRcv} %{IP:IP}%{SPACE}%{WORD:XID}%{SPACE}(?:Q|R|U) ?(Q|R|U)?%{SPACE}[%{GREEDYDATA:FlagsHex}%{SPACE}%{WORD:Response}]%{SPACE}%{WORD:QType}%{WINDNS_NAME:Name}

Result :

2020-07-08T14:54:41.922+02:00 WARN [Messages] Failed to index message: index=<graylog_0> id=<2fd489f8-c11a-11ea-a30a-005056bf4905> error=<{“type”:“mapper_parsing_exception”,“reason”:“failed to parse field [ThreadID] of type [long] in document with id ‘2fd489f8-c11a-11ea-a30a-005056bf4905’”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:"For input string: “0DA4"”}}>

And ES filter :

{“graylog”:{“order”:0,“index_patterns”:[“graylog*”],“settings”:{“index”:{“refresh_interval”:“30s”}},“mappings”:{“message”:{“properties”:{“ThreadID”:{“index”:“true”,“type”:“keyword”}}}},“aliases”:{}}}

8

Try to rename field from ThreadID to something differect if it works…

9

It works !!

Thank you very much :wink:

10

Elastic guestimates the field type by the first value that comes in and will hold onto a fields type until you force a type change (in Grok or custom fields in elastic…)and then rotate the index. Naming a new typed field to the data shortcuts all that for now and future data.

Its faster than creating custom fields and rotating the index… and WAY faster than creating custom fields and re-feeding the data to a new index so it is historically typed properly. I mention this because it’s possible should someone reading this want to do it.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.