Differents between steams and searches

I try to understand right way to use streams - i read documentation but it difficult to understand what is better?

For example we get logs from nginx(huge amount of logs) and logs from vmware.

Now there is two streams - default stream and nginx log stream.
As i understand message from nginx stores in default+nginx stream so i have question.

  1. Is such storing method write 2 message in database(space used twice?) or message stores ones and its binded to different streams?
  2. What best way to use streams - when its nessesary to use and when it helps?

Thanks a lot


Hello @urt

The log will only be duplicated if the streams are associated with separate indices, if the streams share an index then the log will not be duplicated.

Streams are a way to logically separate your logs, this is useful when you think about searching, dashboards, pipelines and permissions.

As an example you may only want a given user to be able to see logs from your nginx hosts and in this case you simply share with that use the nginx stream and nothing else.

Another example, when utilising pipelines and rules this is done per stream. If you wanted to only enrich the logs coming from your nginx hosts that is made easier assuming all those logs can be found under a single stream, a pipeline can then be associated with this stream.