Mismatch between reported traffic and data in indicies

Hi
We are trying out the Graylog to see if this is something we can use. We have a setup with graylog in docker running graylog 4.2.3. We have only setup 2 systems to forward logs to graylog. And we where surprised how much data graylog says that have been logged. If we go to the page System/Overview and looking at the section: Graylog cluster - Outgoing traffic the overview says that
we have logged:
oct. 9th: 6.896G
oct. 8th: 4.543G
oct. 7th: 4.668G
But looking at the indecies there are much less data.
1 index have a retention period of one week. Total 245 MiB
1 index with a retention period of one day.
index for oct. 9.: Contains messages from a day ago up to 5 hours ago (1.0GiB / 3,141,974 messages)
index for oct. 8.: Contains messages from 2 days ago up to a day ago (670.7MiB / 2,092,509 messages)
index for oct. 7.: Contains messages from 3 days ago up to 2 days ago (571.1MiB / 1,757,799 messages)

Using the REST API (System/ClusterTraffic : Cluster traffic stats) I get these traffic stats:

Date Incomming Decoded Outgoing
07-10-2022 50.444.158.011 48.888.754.197 4.670.274.864
08-10-2022 36.922.528.760 35.787.946.659 4.545.440.223
09-10-2022 38.958.743.534 37.749.931.628 7.128.138.631
10-10-2022 8.950.238.443 8.672.468.000 1.611.908.984

There are messages which we drops in a pipeline, which explains why only about 20% of the messages are outgoing. So why are the overview reporting 6 times as much data??
Which data/traffic counter is the license depending on. Data in the indexes or the outgoing traffic from the system overview, which reports 6x as much data???
Are data counts as new traffic in every stage in the pipelines?

kind regards
Jens M. Kofoed

I think Elastic/Opensearch does some kind of magic compression in the database. Therefore the indices are smaller. There are amazing algorithms for the compression of strings.
Did you do a barchart with the sum of the field gl2_accounted_messagesize? It should be the same as the barchart for the licensing.

Yes, I have been looking at the gl2_accounted_messagesize, which is the same as the graph in the system/overview.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.