Help with GROK extractor

[aqx]

Hi all, i’m trying to “normalize” this output i get from an input stream:

fe2 magento-system 2017-09-28T07:26:48+00:00 CRIT (2): Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

with this pattern:

%{WORD:loglevel} (%{BASE10NUM:UNWANTED}):%{GREEDYDATA:message}

and when i try it it looks like:

loglevel
CRIT
message
Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

good!

But then, when i go and see the messages received from the input i don’t see any change in the “message”.
Why my extractor isn’t applied to the message?
Sorry if this question looks nonsense, i’m quite new to graylog

Thanks!

[jochen]

Does it work if you assign a different field name to the %{GREEDYDATA} match?

[aqx]

no, it doesn’t…
maybe i’m missing something. now i removed any extractor from the input.
the original message is:

2017-09-28 12:29:47.000	fe1
fe1 magento-system 2017-09-28T08:29:45+00:00 CRIT (2): Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

and i want it to be like:

2017-09-28 12:29:47.000	fe1
Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

from what i understand, extractors are the way to go (and GROK looks like a familiar way to do it).
is it correct to use

\(%{NUMBER:UNWANTED}\)\:%{GREEDYDATA:message}

in order to overwrite “message” with ony the strings i’m interested into
is the above correct?

thanks!

[jochen]

Are you sure you’re looking at the correct messages?

Seeing the message timestamp and the timestamp in the message itself, it looks like there is some offset.
Try search with an absolute time range and set the end to some time in the future (e. g. +4 hours).

[aqx]

Really, i didn’t know what was happening.
I unflagged “allow timestamp override” in input syslog stream and now i see extractor working correctly.

That’s all, for now

thanks indeed for your support!

[aqx]

Hmm, there’s something wrong with the timestamp, still…
i deleted all extractors again, and i’m using a TCP rsyslog input.
i see very different timestamps (now is 15:05, UTC +1):

• the log on the server is -2 Hrs:

2017-09-28T13:05:55+00:00 CRIT (2): Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

• the log in graylog is +2 Hrs:

2017-09-28 17:05:44.000 fe2
fe2 magento-system 2017-09-28T13:05:44+00:00 CRIT (2): Not valid template file:frontend/base/default/template/catalog/product/view/spptierprices.phtml

• in server.conf i have:

root_timezone = Europe/Rome

• allow override timestamp is not checked in input

what’s wrong?
thanks

[jochen]

How are you ingesting logs? Is it a Syslog input?
If so, then try storing the raw message in the “full_message” field for further inspection (it’s a setting for the Syslog inputs).

[aqx]

Here is a screenshot with all the timestamps and full_message:

thanks

[jochen]

Try to configure your syslog daemon to include the (correct) timezone in the syslog timestamp.

See https://github.com/Graylog2/graylog-guide-syslog-linux/blob/master/README.md for rsyslog and syslog-ng.

[aqx]

Ok, i did add the RSYSLOG_SyslogProtocol23Format i forgot about.
I still have Store full message? checked and Allow overriding date? unchecked

Let’s see how it behaves after some time.
thanks for now