I have my system set up so that anything being worked on is removed from the all messages stream, that way I know if something shows up there that it didn’t get captured in a stream/pipeline properly. Its a personal choice, the system doesn’t require it. Streams are a representation of data flowing through Graylog that you can apply a pipeline to… once it is finished with a stream the data is sent to the connected index… which is a searchable storage area on Elasticsearch/OpenSearch. Here is a thread where a bunch of us discussed and drew up message path through Graylog. Hopefully that will clarify a bit.
I am confused about the separate streams you describe? You can have one input going to one stream and apply multiple pipelines, stages, rules to it. it wouldn’t be efficient to create a separate stream and/or index for each machine… You can have streams associated with indexes that have different retention time/counts and use the pipeline function route_to_stream() to route messages to a different stream (retention) when needed… I feel like the is something missing to the explanation but I can’t put my finger on it…