I took over Graylog server after one of my colleagues and as usually, issues started happen Therefore I have two questions.
Few days ago out of the blue one team noticed that their messages are not being processed by Graylog anymore. They found out that the one of the fields “job_id” changed from string to long and Graylog was saying that “There were 93,640 failed indexing attempts in the last 24 hours.” We fix this issue by creating separate index for this team and now messages are being processed. I read there is something called “Dynamic field mapping” - is there any possibility it will change the field type depending on the majority of the data in that field or its setup once for the index and then not touched at all?
With this another question arrised. We currently have three main indexes 7 days retention, 15 days and 30 days. These three are used by whole company to store the data (therefore issue in first question could happen?). It this something that is OK or better will be that each team with have their own index? I read that shards should be only one since we are running one instance, right? Anyway, one picture for thousands words
Some tech details. We have one instance running as a VMware VM on Linux (runing on Docker), version is 4.3.5 (we are planning to upgrade to v5).
Thank you for your time and guidance!