GROK Extractor Timestamp Question

[poisedforflight]

I am trying to extract the timestamp from a message, but the %{TIMESTAMP_ISO8601} pattern returns a lot of data I do not need. I only need the timestamp field, how do I remove the unnecessary data.

Also, is it possible to skip this field altogether?

Please see the screenshots.

[shoothub]

Yes, there is a option Named captures only in extractor, or pipeline function grok(), parameter only_named_captures, so it will extract only named, in your case only timestamp.

https://docs.graylog.org/en/3.3/pages/pipelines/functions.html#grok

[poisedforflight]

Thank you, that worked. Is there a way to completely skip fields?

[shoothub]

Yes of course, use UNWANTED as a name in grok pattern, for example:
%{BASE10NUM:UNWANTED}

Check docs:
https://docs.graylog.org/en/3.3/pages/extractors.html#using-grok-patterns-to-extract-data

[poisedforflight]

@shoothub

When I try this I get an error that Graylog cannot parse:

[shoothub]

1. GROK as a result generate regular expression, so it’s only simplified form of regex. So you need to include grok patterns for part of text.
2. If you want to skip some text you can use %{GREEDYDATA:UNWANTED}. For example to extract fw field use:
   %{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:UNWANTED}fw=%{IP:fw_ip}
3. If you want to extract all field, check for key-value extractor. Your message is great candidate for it.
   https://docs.graylog.org/en/3.3/pages/extractors.html#automatically-extract-all-key-value-pairs

[poisedforflight]

I’m trying that, as well as some other patterns like INT, NUMBER, WORD, and a few others and Graylog throws an error no matter what I try. Ideally I’d like to strip other fields out as well.

[shoothub]

You have space between %{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:UNWANTED}
If you check original message, there is " after timestamp, and not only space, so you need to include " in your grok, so:

1. Either use grok without space (which will also include " and space and another strings):
   %{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:UNWANTED}
2. Or use grok with " after timestamp:
   %{TIMESTAMP_ISO8601:timestamp}" %{GREEDYDATA:UNWANTED}

[poisedforflight]

@shoothub Thanks to you I’ve made some major progress. I have another issue I need to overcome though.

There is a field in the log that contains the line:

msg=“NWC23465: VPN Tunneling: Session ended for user with IPv4 address 192.168.110.10”

I would like to break this into two fields:

evID = NWC23465 (without colon)
msg = VPN Tunneling: Session ended for user with IPv4 address 192.168.110.10

[shoothub]

Why not use something like:
msg="%{WORD:evID}: %{GREEDYDATA:msg}"

[poisedforflight]

@shoothub

msg=“%{WORD:evID}: %{GREEDYDATA:msg}”

I would like to buy you a drink sometime. THANK YOU!

[shoothub]

Check this fine online grok debugger, it parses grok patterns online, so can help with grok generation.
http://grokdebug.herokuapp.com/

[poisedforflight]

I have been. It does not like the {EMAILADDRESS} for some reason. I had to use {GREEDYDATA} to pull the Username on that site.

[shoothub]

Maybe graylog have more grok patterns, than this page, but you can still add your own, using Add custom patterns

[poisedforflight]

I will need to use a pipeline to change the order of the fields in the message? It appears to default to alphabetical order.

[shoothub]

If you mean how graylog displays extracted fields, yes it’s in alphabetical order, so you can’t change it.