May 9, 2023, 2:41pm
1. Describe your incident:
We had a problem with mongoDB that made a minor upgrade and stopped to work. We downgraded the mongoDB and it started to work again. The problem is: our Graylog have 3 indices: the default (graylog_X) and two more duck and cow.
All messages from the default index are appearing normally after this, but messages from
duck and cow are not, graylog know that message are there but is not exhibiting it:
And OpenSearch is Green
2. Describe your environment:
Kubernetes running Graylog 4.3, Opensearch 1.3.5 and Mongo 5.0.6.
3. What steps have you already taken to try and solve the problem?
Tried to recalculate index range, and checked index status, but i’m kind of out of ideas and I don’t see ERRORS in logs.
Do you have any direction to point?
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]
@gasc, have you rotated the active write index for each of them?
May 12, 2023, 1:01am
Hi Chris, yes I tried this with one of them, but no success.
May 16, 2023, 12:05pm
Did you try to recalculate index ranges for these index ?
May 18, 2023, 1:54pm
yes i recalculated all indices
May 23, 2023, 10:39am
your streamid got changed?
Try to check if the stream id is retain as-is from old to new
May 31, 2023, 10:44am
gocoolsg thanks man, this is the problem.
do you have any ideia how to fix this?
i’ll try to change it directly on mongo in my test environment and check if i can recover it.
June 7, 2023, 4:01pm
I chanted the stream ID from the object directly on MongoDB but messages still not appearing. When I recalculated the index ranges to check if works, Graylog rollback the ID for the former stream.
So I think that Graylog has some reference for the old stream. I just don’t know where.
June 7, 2023, 5:07pm
not the ideal solution, but I will stop with this: instead of migrate old messages to the new stream I recreated the old stream directly on mongo