Thank you for your help! It indeed seems that the GitHub issue describes the same observations and after verifying this in my use case, it is indeed also the case that the duplicates occur only for indexes which are not the most recent, i.e. after the index rotation window.
I do have some questions about the shunting example you posted, let alone whether the code actually works:
- How would I make sure that exclusively messages from the shunting pipeline are processed within the Stream and that no messages are processed from other origins (e.g. from the process that is currently in place)?
- Are you able to confirm that the described problem does not occur when the shunting is applied?
- So to rephrase your shunting example in my setting: I would make the All stream the shunting stream, which receives all messages. I then add a pipeline to this Stream with a Stage for each additional Stream which shunts the messages based on additional conditions. e.g. for the Anomalies stream I would create a stage which forwards all messages with a level lower than 6 to the Anomalies stream. Does this way of phrasing it sound correct to you?