At my organization, we created extractors using Grok patterns in our Graylog instance to parse our logs into fields.
I’ve been noticing in Graylog if an extractor doesn’t successfully parse a log (because the format doesn’t match the Grok pattern) the log doesn’t get written to an index. Instead, it generates an indexing failure.
This means that if a log format changes we’ll lose all our logs until we update our extractors.
Is there a way to avoid this?
@clarson you should not loose a message if extractors are not matching the messages that are coming in.
But without knowing what you are doing exact this is hard to tell why your Installation behave this way.
regards
Jan
Thanks for taking a look at my question.
This happened when we upgraded from Graylog 1.3 to 2.2. We’ve been using the below Grok patterns from the marketplace to parse our firewall logs. After the upgrade all of our firewall logs stopped indexing. They wouldn’t show up in any of our dashboards, streams or searches.
I looked in the Indexing Failures page and saw that all of our firewall logs were generating failures. Changing the Grok patterns fixed this.
Just to make sure I understand, if the conditional regular expression for a extractor matches a log, but the Grok pattern does not, does the log not get indexed? See the below screenshot for an example condition we used for one of the extractors:
@clarson how did you update?
the linked content pack might not work with 2.2 as the author did not update the readme regarding that.
it looks like you are running in the same problem like that user over here - and some additional that are also had posted in the Forum.
We already provide some detailed helped here for example.
I updated the .repo files and updated with yum according to these instructions:
https://www.lisenet.com/2016/graylog-server-upgrade-from-1-3-x-to-2-0-x-on-centos-6/
You’re right that the content pack didn’t work with 2.2. We managed to get that part working by updating the Grok pattern.
But I’m curious if there’s something we can do so that if an extractor isn’t matching correctly, the logs still get stored in an unparsed format.
please read the linked help from the forum.
switch over to a RAW Input - as written in the linked thread!
Oh now I understand.
I wasn’t sure if it was related the RAW input or something in the extractor configuration.
I just changed our input to RAW TCP and tested the extractors and they behaved the way you described. This fixes our problem.
Thanks so much!
