We just upgraded from 7.0.4 to 7.1.3, and are now seeing a large number of “Message Errors” on one of our Beats inputs, relative to the total number of messages being received. This input receives Filebeat logs from sidecars on several Windows servers, and it was working fine prior to the upgrade. All of the errors seem to be in message processing, for example:
Message Error at Input: 0
Message failed to process: 4289
Message failed to index: 0
There are a handful of extractors on this input which all seem fine, and there is no pipeline that handles anything from this input. Even with all logging set to DEBUG, nothing relevant to this gets logged.
I would really like to track down the cause of this. Some messages are working fine on this input, but the number of failures represents ~10% of total messages received and is always climbing. Any ideas or suggestions are greatly appreciated.
Thanks!
Hey @d_a_parker,
There were some issues raised and subsequently fixed in 7.1.3 with regards to extractors, perhaps something has gone unaccounted for.
Do the messages still get processed by the extractors, can you pinpoint exactly what messages the error applies to?
While this would not fix the problem, this could be used as an opportunity to migrate the extractors to rules and a pipeline. Feel free to post them here if you want a hand with that.
Thanks @Wine_Merchant - that’s good to know.
Unfortunately, there’s no good way I’ve found to see which messages, extractors, or pipelines are involved. The tools to see that info appear to be enterprise-only features. But after watching this for a few days, I now see that the Message failed to process number is not a total. It goes up and down over time, although I’m not sure what the timeframe for this number is. I don’t know if this is the number of failures just in the last few seconds, or over a longer (but sliding) window of time.
Also, not every extractor on this input will match every message that comes in. I’ve been wondering if this is just a count of normal extractor “misses” in the last n seconds, which was not surfaced in 7.0.4 but is now visible in 7.1.3. This isn’t happening with any other inputs, though, and we definitely have other extractors which will also fail to match a lot of messages that pass through them.
I appreciate the offer to convert our extractors into rules. I’m trying to minimize the changes made in Graylog while I troubleshoot this, but may decide to make that switch later. I had planned to do it when we upgraded from 5 to 7 because I thought extractors were deprecated in 6, but they still seem to be supported and ours have been working fine.
Thanks!
