1. Describe your incident:
My graylog often doesn’t consume data after some time. It was running normally for the past year, but this problem suddenly occurred recently. After restarting gralog-server, it will return to normal, but after a while, the problem of not consuming data will appear again, maybe a few hours or a day.
2. Describe your environment:
-
OS Information:CentOS Linux release 7.4.1708
-
Package Version:Graylog 5.1.4
-
Service logs, configurations, and environment variables:
2023-09-01T06:52:48.935+08:00 INFO [CreateNewSingleIndexRangeJob] Created ranges for index g4-service_14267.
2023-09-01T06:52:48.941+08:00 INFO [SystemJobManager] SystemJob <0882e152-4851-11ee-91b8-78ac443b71f8> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 2198ms.
2023-09-01T07:02:25.805+08:00 INFO [SystemJobManager] SystemJob <1ad3c0e0-4851-11ee-91b8-78ac443b71f8> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 578334ms.
2023-09-01T07:31:10.442+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-1, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:10.444+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-0, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:10.578+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-2, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:10.718+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60853dadb3dda129103843e9-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:10.729+08:00 INFO [ConsumerCoordinator] [Consumer clientId=gl2-5312e0e0-60853dadb3dda129103843e9-0, groupId=hkgraylog] Revoke previously assigned partitions g3-app-log-2
2023-09-01T07:31:10.730+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60853dadb3dda129103843e9-0, groupId=hkgraylog] (Re-)joining group
2023-09-01T07:31:10.740+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-1, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:10.741+08:00 INFO [ConsumerCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-1, groupId=hkgraylog] Revoke previously assigned partitions
2023-09-01T07:31:10.741+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-1, groupId=hkgraylog] (Re-)joining group
2023-09-01T07:31:11.142+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6087ae5fb3dda129103aec0c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.143+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.167+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60877967b3dda129103ab20c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.189+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6089084db3dda129103c6404-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.469+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.524+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.536+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-2, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.538+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-0, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.569+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-2, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:11.579+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-0, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:13.469+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-1, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:13.475+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-0, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:13.587+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-2, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.145+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6087ae5fb3dda129103aec0c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.145+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.170+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60877967b3dda129103ab20c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.192+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6089084db3dda129103c6404-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.471+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.527+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.539+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-2, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.540+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-0, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.571+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-2, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:14.584+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-0, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:16.474+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-1, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:16.479+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-0, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:16.592+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-611b252a807b003fc22702e6-2, groupId=graylog2] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.147+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60891b6db3dda129103c7902-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.147+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6087ae5fb3dda129103aec0c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.172+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-60877967b3dda129103ab20c-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.194+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-6089084db3dda129103c6404-0, groupId=hkgraylog] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.473+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b9459d79e0b75aa0c4865-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
2023-09-01T07:31:17.532+08:00 INFO [AbstractCoordinator] [Consumer clientId=gl2-5312e0e0-610b94e5d79e0b75aa0c492e-1, groupId=g4app] Attempt to heartbeat failed since group is rebalancing
3. What steps have you already taken to try and solve the problem?
1.upgrade graylog-server version to 5.1.4.
2.upgrade kafka version.
4. How can the community help?
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]