within our GrayLog installation we’re currently having the problem, that ElasticSearch isn’t able to store messaged because of Strings that ElasticSearch is trying to place in a “Long” field. I already know that I have to fix this by creating a custom mapping index (Elasticsearch - Configuring Graylog).
But I guess that this would simply be a workaround, as some of our applications is trying to use the “level” field for text. So my idea was to identify the system, which is sending these messages but so far I was unable to do so.
As the message didn’t got stored by ElasticSearch, I’m unable to search for the message (At least a search with gl2_message_id & the message ID gave me zero results).
So what can be done, so that I’m able to identify the source of these messages?
This is a hard one, If elasticsearch did not index these messages then you might not be able to find them.
Not sure what version/s you have or how GrayLog is configured. But a bit of advice. If this environment has different devices ( Firewalls, Switches ,etc…) highly suggest to create an Input for each type of device/s.
When this issues happens at least it can be narrowed down.
Not necessarily, If there was a device trying to use same field/s that elasticsearch default/dynamically created then with separate INPUTs it would be easy to create a new Index template attach it to Index set and re-route that device/input to the new INDEX set. Another option would be to use a pipeline and convert that message but I understand you need to find that device.
Another suggestion would be is to stop sending logs to Graylog and add one at a time until you find your device. Or perhaps remove any new device that was added prior to this issue.
Last , If by chance your using Graylog 4.x or greater, there is an index set called "Graylog Message Failures " you can look in there.
thank you for the detailed reply and sorry for my late response. I was able to identify the system which was responsible for this.
Also the “faulty” messages got stored the whole time, I was simply unable to find them with the provided information’s. Within the GrayLog Dialogue (Version 4.1 btw) the system provides a “letter id” of the message where Elasticsearch ran into an Exception. But I failed to use this “letter id” for a lookup. In my previous message I’ve already noted that I’ve searched for a message where the gl2_message_id equals this letter id but this resulted in zero results. Also a simple search for this “letter id” gave me zero results.
I fixed my issue with simply luck, but I guess it might help others how they can lookup a message by this letter id. Within this board I found some posts that the “letter id” equals the id that is displayed when you click on a log entry. But I didn’t found an information how you can search for this ID.
