Has anyone faced this issue that message gets stuck in processing buffer, unable to be indexed by elastic search? How did you deal with it?
I did some digging - custom index mapping is one way, but that involves messing with indexes on console level, directly via REST API, would be nice to avoid that.
Also in configuration there is this “UI analysis disabled for fields”, contains full message by default - I cannot find anywhere in documentation what this does exactly.
its “Disabled analysis fields” - A ‘,’ separated list of message fields for which analysis features like QuickValues will be disabled in the web UI. - does this mean elastic will not index fields listed here or its just for GUI ?
When I had a similar issue it was a poorly formed GROK statement. Do you have a sense of what is locking up the processing buffer? Can you give more detail such as the message and maybe the extractor or pipeline rule that it’s getting hung at? (obfuscated and formatted nicely with the forum tool </>) Creating a custom mapping on the elastic side is not that hard, it’s slightly daunting, but there are posts in here on how to do it if you search… I have put out one or two.
Process buffer is locked by 2500 lines of data or jason, when whole real payload is logged instead of informative message. One of fields obviously exceeds 32 KB and thats that.
Navigate to System/Configurations, then under Search Configuration click the “Update” button.
At the very bottom Surrounding search filter fields add what you want.( i.e. full_message, message, etc…)
So its just for GUI then and does not affect indexing ? All messages coming into graylog are indexed unless there is custom index mapping on console level - that’s my current understanding.
Elastic indexes all the fields, just some are treated differently. Custom indexes are usually done to adjust the field type - From the Graylog docs:
Also note that message , full_message , and source are the only fields that are being analyzed by default.While wildcard searches (using * and ? ) work on all indexed fields, analyzed fields will behave a little bit different.See wildcard and regexp queries for details.
(The included link is old but still relevant. I will submit to have the link checked/updated.)
