Graylog 7.1.x and broken shards daily

General
1

We setup graylog in august, and found that we needed more servers to support the load, so I setup two more nodes. they all have a graylog node, graylog data node and a mongodb replica. The setup seems to be working, but I find that pretty much everyday I end up with one or more servers with a broken shard. Now I recently update to 7.1.11 in hopes that it would fix the problem but I don’t think it has. One thing that I have noticed is when I setup the new nodes I used ext4 for the shards and since then have changed one of the new nodes to use XFS like the original node that seems to have made little difference. The logs seem to suggest an io wait but they don’t line up with when I get broken shards and the Hyper-V hosts don’t seem to share this same issue infact they report everything is fine and super fast. I’m not the best when it comes to linux, more of a windows admin, so I’m at a loss as to what to check or change to fix this issue. Node 1 the original node has never had this issue only the two new nodes. This would suggest a configuration difference but the only real difference is the EXT4 vs XFS that I can see. Node 1 and Node 3 are on the same cluster of two hosts with one flash array for storage. Node 2 is on a separate hyper-v cluster on the same LAN, using a different flash array. The network link between the two clusters is 10gb. If there is a piece of information that I have not provided that would be needed to help solve this just ask.

2

Hey @aarnold-sd68,

Would you be able to provide the server and data node logs?

Just an advisory but generally the Graylog Server and Data Node should be installed to separate hosts to avoid resource contention.

With that in mind, what are the resources available per node and what do you have set as the heap settings for both Graylog service and Data Node?

3

I can upload the logs, but I have one question is there anything in them I should blank out? Security being what it is and all. The VM’s have 8 processors with 64gb memory. and they are all on separate hosts for the most part, I have them configured to be separated so they can only share if they need to be, maintenance mostly. Just FYI I can make them stable sometimes, like right now they seem stable, and they have even balanced themselves to with in 1 shard so I have made progress. The hosts are intel xeon’s with 64 logical cpu count at 2.39 ghz and 1,500gb of memory for node 1 and 3, and xeon’s 2.3 with again 64 logical cpu’s and again 1,500gb of memory and that is where node 2 sits. So you think I should separate the data node and the graylog node’s for my setup. Interesting I go the impression from the documentation that my setup was acceptable..

4

For MongoDB and Graylog, it is recommended that those services are installed to the same host in smaller clusters but the indexers should be separated. The amount of resources dedicated to each node are calculated based on use cases and the average expected daily ingest into the cluster. These are just best practice, no need to change anything until you get the bottom of the issue.

No worries on uploading the logs if you’re concerned they may contain identifying information. The next time a shard fails, you can query the reason behind the failure with the below commands. You would need to craft a curl specific for your Graylog Data Nodes and also generate a client cert via the ui to use with the curl command. Check the timestamp for the specific failure and search the Data Node logs for that point in time to hopefully get a more detailed explanation. You can post any findings back here, anonymising where required.

All shards
GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason,node&s=state

target specific shard

GET _cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": true
}
5

Thank you, I kind of already know the reason. The shard gets created but it missing all the files, except for wire.lock and one other that I can’t remember of the top of my head. It’s like it started to create the shard and then just stopped before it finished created all the needed files. but I’ve actually had two days now with no broken shards, which is new. I did just upgrade to 7.1.11-1 so maybe some thing was bugged there. As for moving the indexers, would it just be as simple as creating a new set of indexers putting them behind a load balancer and joining them to the cluster? How would I move the current data to the new indexers? is there a document somewhere?

6

Good to hear it has stopped, if you find it happens again then make sure you check pore through the logs for clues.

There is no need for a load balancer between Graylog and the indexers, distribution is handled by Graylog/Opensearch.

You could spin up new nodes and have those nodes join the cluster, then drain the old nodes of shards and stop the service.

Below is the command that excludes nodes from shard routing, if you add all the IP’s of the old nodes and the 3 new nodes are available then the data should be transfered. While this happens the cluster will be red and the transfer could take some time.

   PUT _cluster/settings
   {
     "persistent": {
       "cluster.routing.allocation.exclude._ip": "10.0.0.1,10.0.0.2,10.0.0.3"
     }
   }

After the data has moved and the cluster appears as green and data is searchable, you can stop the data node service on the old nodes one by one and then uninstall the package.