Only message from the last 7 days are beeing displayed / Where are the old messages? Index/Streams Question

[seroal]

Hi all,

yesterday I observed in my Graylog Dashboard, that I only have messages available, that are max. 7 days old… It looks like that:

Because I have different indices, the main ones based on size, I started looking what could be the issue.
This is the default index set, at which a few weeks ago I changed the default rotation strategy to Timebased / 7 days. A few unimportant messages are routed into this default index.

This is a size based index, which gets it´s input by a stream, routing messages into it.

But the oldest messages, that are beeing routed into this index by this stream, are only 7 days old. And as you can see, the maximum size of the index set is not yet met.

So for me it looks like, as if the changed setting to the “default index set” also has an impact to the other index starting with “co”.

Also, the messages are routed into the correct stream:

But as the index starting with “co” is still 240GB big, where are all the messages older than 7 days? There must be all the old data somewhere! How can I access it again?

Any ideas, where the problem could be?

Systeminfo:
Type: packet installation
Graylog 4.0.5+d95b909 on XXX (Debian 11.0.9.1 on Linux 4.19.0-13-amd64)
elasticsearch-oss 7.10.2
mongodb-org-server 4.2.13

Package: graylog-server
    Versions:
    4.0.5-1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages) (/var/lib/dpkg/status)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.4-1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.3-1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.2-1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.1-1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-8 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-7.rc.2 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-5.rc.1 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-4.beta.4 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-3.beta.3 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f

    4.0.0-2.beta.2 (/var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages)
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f
     Description Language:
                     File: /var/lib/apt/lists/packages.graylog2.org_repo_debian_dists_stable_4.0_binary-amd64_Packages
                      MD5: bc2d4f8a10fe2ef75ffd64454742166f


    Reverse Depends:
      graylog-enterprise-plugins,graylog-server 4.0.3-1
      graylog-integrations-plugins,graylog-server 4.0.5-1
      graylog-integrations-plugins,graylog-server 4.0.4-1
      graylog-integrations-plugins,graylog-server 4.0.3-1
      graylog-integrations-plugins,graylog-server 4.0.2-1
      graylog-integrations-plugins,graylog-server 4.0.1-1
      graylog-integrations-plugins,graylog-server 4.0.0-8
      graylog-integrations-plugins,graylog-server 4.0.0-7.rc.2
      graylog-integrations-plugins,graylog-server 4.0.0-5.rc.1
      graylog-integrations-plugins,graylog-server 4.0.0-4.beta.4
      graylog-integrations-plugins,graylog-server 4.0.0-3.beta.3
      graylog-integrations-plugins,graylog-server 4.0.0-2.beta.2
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise,graylog-server
      graylog-enterprise-plugins,graylog-server 4.0.5-1
      graylog-enterprise-plugins,graylog-server 4.0.4-1
      graylog-enterprise-plugins,graylog-server 4.0.0-7.rc.2
      graylog-enterprise-plugins,graylog-server 4.0.2-1
      graylog-enterprise-plugins,graylog-server 4.0.1-1
      graylog-enterprise-plugins,graylog-server 4.0.0-8
      graylog-enterprise-plugins,graylog-server 4.0.0-5.rc.1
      graylog-enterprise-plugins,graylog-server 4.0.0-4.beta.4
      graylog-enterprise-plugins,graylog-server 4.0.0-3.beta.3
      graylog-enterprise-plugins,graylog-server 4.0.0-2.beta.2
    Dependencies:
    4.0.5-1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.4-1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.3-1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.2-1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.1-1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-8 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-7.rc.2 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-5.rc.1 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-4.beta.4 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-3.beta.3 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    4.0.0-2.beta.2 - uuid-runtime (0 (null)) graylog2-web (0 (null)) graylog-web (0 (null)) graylog (0 (null)) graylog2-server (0 (null))
    Provides:
    4.0.5-1 -
    4.0.4-1 -
    4.0.3-1 -
    4.0.2-1 -
    4.0.1-1 -
    4.0.0-8 -
    4.0.0-7.rc.2 -
    4.0.0-5.rc.1 -
    4.0.0-4.beta.4 -
    4.0.0-3.beta.3 -
    4.0.0-2.beta.2 -
    Reverse Provides:

Any help appreciated.

Have a good day,
Sebastian

[seroal]

I think I should add the following, maybe it is important here:

In the past, I set another Index as the “Default index set”, this was because of a thought how to route data to different destinations based on the customer to keep the data seperate (to be able to set permissions accordingly and also be able to delete data per customer).

So I actually set the default index to another one, but then I observed, that the stream “All messages” was responsible, that the messages were still routed into: “index set Default index set”. It is also not possible to change or delete this “All messages” stream.

So can this behaviour be a cause of the changes I made ?

[gsmith]

@seroal

It is also not possible to change or delete this “All messages” stream

Maybe this might help with your question.

Is it possible to edit the default stream (all messages) via the rest api?

No, that’s not possible. But it’s possible to add outputs to the “All messages” stream via the Graylog REST API. You can also manage outputs in the Graylog web interface via Streams/Manage outputs.

[seroal]

@Is it possible to edit the default stream (all messages) via the rest api?

ut it’s possible to add outputs to the “All messages” stream via the Graylog REST API. You can also manage outputs in the Graylog web inte

I mean, that answer is especially about the Rest API. So I don´t know, what would be possible outside API to do with the all messages Stream.

But that was also a question to provide with the main question, how it comes, that I only have data for the last 7 days…

[seroal]

Now I set back the default index setting to the standard “default index set”. I will monitor the situation… But anyways, would be interesting to get to know from somebody, how the system behaves like it does.

[seroal]

Hi there.

Today I took another look at my messages. I went into my stream, that sends messages into the index using index rotation by index size. Despite of that, I still have only 7 days of messages.

The 7 days setting is only applied to the default index set.

Is the setting somehow inherited to my other index? What are the dependencies between default index set and newly ones? How can this be fixed?

[gsmith]

Hello,

@seroal

that sends messages into the index using index rotation by index size.

Have you tried to increase the size of that index (10GB) to see if you can have more then 7 Days?
Like make it 12 GB etc…

That index created by Size (i.e. 10GB ) Did you test to see if there are current messaged being applied to that index?

@seroal

Is the setting somehow inherited to my other index?

I dont believe so.

Here is my index set’s.

I can go to those stream/s and see messages being applied
I hope that helps

[seroal]

Hi!

@gsmith

@seroal

that sends messages into the index using index rotation by index size.

Have you tried to increase the size of that index (10GB) to see if you can have more then 7 Days?
Like make it 12 GB etc…

That index created by Size (i.e. 10GB ) Did you test to see if there are current messaged being applied to that index?

I didn´t. I think this will not change, because there is enough space left for more than 7 days. Rotation will occur after 10GB and maximum number of indices is 40. So 400GB are available. Currently 244 are used. And interestingly, the size is actually staying at arround 244GB. So it looks, as if the Index rotation does work as expected… Could this be, that the system thinks, that it is already using 400GB? How can I troubleshoot this?

@gsmith

That index created by Size (i.e. 10GB ) Did you test to see if there are current messaged being applied to that index?

Yes, I actually take a look at the message (see also my screenshots above) and in the message, the stream and index is correct (starting with “co”), so they seem to be routed into the correct destination.

@gsmith

I can go to those stream/s and see messages being applied
I hope that helps

I also can see it. Looks normal, but on the end, I only have 7 days available.

What I today did is, that I created a new index-set and now I route some of my messages in this new stream. The indexset “2co” is configured identical with my indexset “co”. Now I will wait for a few days to see, if the messages in the new index are also only max 7 days.

Lets see.

Thanks so far.

[gsmith]

Sounds good, keep us update.

[seroal]

Hi there. Today I took another look into the system. What should I say. For some reason, it is now working as expected again. Have a look at the data available since now 29.03.21. Thats strange to me. I can only imagine, that this is caused by changing the default index set. And then because of some reason in the default message stream, this all does not work as expected any more. Maybe someone can explain please…?!

But to doublecheck, the next days I will switch back the setting in streams and route all messages again into index “co*”. But I want to observe a few more days.

[gsmith]

@seroal
Sorry to here that this is still a problem. What changes and/or configuration have you done different this time compared to last time?

NOTE: “The definition of insanity is doing the same thing over and over again and expecting different results.”