# Outgoing traffic -vs- message size **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [September 30, 2019, 2:29pm UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166 "2019-09-30T14:29:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![james.naish](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@james.naish](https://community.graylog.org/u/james.naish)\ **Post date:** [September 30, 2019, 2:29pm UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/1 "2019-09-30T14:29:59Z")
Hi all, Something I’m keen to understand is the difference between outgoing traffic and the size of messages in Elasticsearch. I’ve been trying out adding a new service to Graylog and this has significantly increased the outgoing traffic. When I export all the log messages that this new service has pushed to Graylog, the total message size is in the region of 80 MB. However, this appears to correlate to approximately 20 GB of additional outgoing traffic over the same period. We also calculated the message size for ALL messages over a given period. The message size was 3 GB, but total outgoing traffic for the same period was around 14 GB. - There is only one Index Set configured and all streams are set to use this On investigation, I came across this post: > [@Aggregate on message size across multiple events](https://community.graylog.org/t/aggregate-on-message-size-across-multiple-events/5996): > > Hi Folks, Is it possible to create dashboard on daily (24hrs) log data ingestion size for multiple events? The overview shows cumulative data ingested into graylog per day however it doesn’t provide the breakdown, for example which logs made up the most data and from which sources. is there a way to do so using search and charts? Thanks, Navdeep Which contains the cryptic line " be warned, that the size of the messages in Elasticsearch **is not equal** to the outgoing traffic in Graylog (on the System page)." Two questions: 1. Any thoughts on what might explain the discrepancy we are seeing? 2. Can anyone elaborate on the difference between message size and outgoing traffic? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [September 30, 2019, 2:52pm UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/2 "2019-09-30T14:52:34Z")
The Graylog Documentation explains about the outgoing traffic: [http://docs.graylog.org/en/3.1/pages/enterprise/setup.html#details-on-licensed-traffic](http://docs.graylog.org/en/3.1/pages/enterprise/setup.html#details-on-licensed-traffic) The size on disk depends on the data, the field type that is used and similar. ---
**Author:** ![james.naish](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@james.naish](https://community.graylog.org/u/james.naish)\ **Post date:** [October 1, 2019, 7:27am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/3 "2019-10-01T07:27:03Z")
Thanks @jan. We had read this, but this doesn’t actually explain why there would be a discrepancy between the size of the message coming into Graylog and the size of the message going out to Elasticsearch which is what this question is asking. The link you provided defines outgoing traffic as: “what is written to Elasticsearch after all processing is done” We have already confirmed there is only one Index Set and all streams use this. This raises the question as to what additional processing might take place to enrich messages between GL and ES? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [October 1, 2019, 8:45am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/4 "2019-10-01T08:45:59Z")
ok, I think I got it. If not, let me know. Messages are ingested into Graylog as one single string. That is after that processed in different ways: - by the codec of the input (Syslog, GELF, CEF or any other) - extractors that are defined for the input - processing pipelines for the stream the messages are in The codec will at first make a first seperation into different fields, extractors or pipeline can split that into more specific fields or enrich the data from external sources. That would include geo data for IPs, DNS lookups or external databases/systems that are queried via REST APIs or a simple CSV Lookup. It could also be that information from the original message are removed or anonymized. I have seen environment where the original message was removed completely and only a new generated message with some information from the original message plus information’s form a lookup are saved in elasticsearch. I have myself in my environment some noisy messages that I can’t remove because the 3rd party vendor is either chatty or gives no information. So I remove the chatty messages to have the only important ones. But that means I remove 19 messages out of 20 to get the one important message. This drop is done in Graylog so only one message is saved to elasticsearch all other are gone. Does that explain enough? ---
**Author:** ![james.naish](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@james.naish](https://community.graylog.org/u/james.naish)\ **Post date:** [October 1, 2019, 9:13am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/5 "2019-10-01T09:13:43Z")
Thanks @jan - this is all really helpful and gives some good pointers. So one form of processing would be to filter out fields/messages so we only have the useful stuff, but in this case we would expect Input \> Output. However, we are finding that the messages are getting substantially bigger, not smaller, at the point of storage. So as an example, we looked at messages over a weekend and found that we generated 3 GB of messages going into GL, but this ended up generating 14 GB of outgoing traffic. We see consistently that the volume of outgoing traffic is substantially larger than the size on disk of the input messages. Any thoughts? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [October 1, 2019, 9:48am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/6 "2019-10-01T09:48:02Z")
> Any thoughts? Many but without knowing anything about your Environment, what you ingest what you process and how it is all just guessing what have happened. ---
**Author:** ![james.naish](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@james.naish](https://community.graylog.org/u/james.naish)\ **Post date:** [October 1, 2019, 10:24am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/7 "2019-10-01T10:24:48Z")
We have one input which is ingesting messages via GELF. These messages hit three streams: - Application Messages - Errors - Notices and Warnings All three streams use the same Index Set. We don’t have any processing pipelines set up for the input. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [October 15, 2019, 10:24am UTC](https://community.graylog.org/t/outgoing-traffic-vs-message-size/12166/8 "2019-10-15T10:24:53Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.