# Graylog as Aggregator and ETL tool **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [January 4, 2019, 12:10am UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249 "2019-01-04T00:10:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![raghu999](https://avatars.discourse-cdn.com/v4/letter/r/c4cdca/32.png) [@raghu999](https://community.graylog.org/u/raghu999)\ **Post date:** [January 4, 2019, 12:10am UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/1 "2019-01-04T00:10:46Z")
Hi all is there a way to configure a stream not to send data to ES index and use Graylog as aggregator like fluentd? We are doing a POC on sampling log data and We want to send data to multiple destinations after processing for example to splunk ,crateDB, ES and s3 but currently because of the underlying single ES dependency to all the streams we are seeing data drop and huge latencies seeing logs in splunk as the journal is getting filled quickly, we tried increasing the Journal buffers but that is adding more latency in seeing logs in splunk or s3. What are we trying to do? In our setup the Data being ingested to graylog directly is roughly around 70TB but we only want to send around 4.5TB to Splunk via pipelines after processing and around 10TB to ES and all the 70TB to S3. Current Behavior Graylog is able to buffer all the incoming logs but most of the data is getting dropped because we have only dedicated 10TB resources to ES per DC. Below is detailed description of our POC setup: We are running Graylog in three DC’s…Our current setup Per DC with heavy TCP tunings. **Graylog** (7 masters 33 agents) Deafult journal Buffers running as Containers on k8’s **Mongo** (HA 4 Nodes) **Elasticsearch** (10 Nodes 1-master,1-balancer,8-Datanodes) **Instance Details** All nodes are running on I-Flavor(Designed for Data workloads) with 16CPU and 64GB RAM **Operating System** Centos 7.2 Willing to contribute back if its a viable solution: We are in process of implementing custom stream plugin and need guidance or suggestions if we can use Graylog as aggregator and ETL tool without single ES dependency and also Is implementing custom stream plugin the right approach or are there any other alternatives? Here is an open issue which i raised which also relates to ES Federation and using **Graylog** as **Aggregator** [https://github.com/Graylog2/graylog2-server/issues/4199](https://github.com/Graylog2/graylog2-server/issues/4199) ---
**Author:** ![raghu999](https://avatars.discourse-cdn.com/v4/letter/r/c4cdca/32.png) [@raghu999](https://community.graylog.org/u/raghu999)\ **Post date:** [January 4, 2019, 12:21am UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/2 "2019-01-04T00:21:09Z")
Current Throughput Per DC ![08%20PM](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/34a33e4c5ef018edaabeb58921a8d042f5f0d25e.png) ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [January 4, 2019, 6:50am UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/3 "2019-01-04T06:50:04Z")
So many questions 🙂 This setup is waaaay over my head, so I’m pretty much not-very-helpful here. I just have to ask: what is “_I-Flavor_”? Googling for it with some terms added to the query doesn’t give my useful results. ---
**Author:** ![macko003](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/macko003/32/3175_2.png) [@macko003](https://community.graylog.org/u/macko003)\ **Post date:** [January 4, 2019, 11:20am UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/4 "2019-01-04T11:20:58Z")
“just forward” - no it is not possible, but I think it could be a workaround. You do a new index set, with little size based indices. If you use this index set the GL will store the messages, but only a few mins/hours. Maybe if you set ES to don’t index the index set, don’t store replica, etc…, it could be add some performance for this forward index set. You can increase the output\_batch\_size parameter, and increase the ES http max size. Change it parallel, it can cause problems the wrong sizes. If you have time, we collect the big clusters’ information. As I see you handle a lot of data, so you can tell something new for us. If you would like, we can ask an admin to reopen the topic. > [@Users feedbacks / Guides for heavy load graylog Cluster](https://community.graylog.org/t/users-feedbacks-guides-for-heavy-load-graylog-cluster/7794): > > Hi everyones, i create this topic to have informations about sizing for a “heavy load” graylog cluster. Specially for the sizing of ressources (CPU / IO / RAM) of each nodes (graylog / elasticsearch), heap memory configuration …tricky options … I think it would be interesting for the community to have generals advises. Inside this Topic : @macko003 (great performance / sizing review) : [Users feedbacks / Guides for heavy load graylog Cluster](https://community.graylog.org/t/users-feedbacks-for-heavy-load-graylog-cluster/7794/4)@benvanstaveren, review of his graylog cluster … ---
**Author:** ![raghu999](https://avatars.discourse-cdn.com/v4/letter/r/c4cdca/32.png) [@raghu999](https://community.graylog.org/u/raghu999)\ **Post date:** [January 4, 2019, 2:20pm UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/5 "2019-01-04T14:20:45Z")
I family instances are used for data intense workloads more details can be found here [Storage Optimized Instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html) ---
**Author:** ![Totally\_Not\_A\_Robot](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/totally_not_a_robot/32/3353_2.png) [@Totally\_Not\_A\_Robot](https://community.graylog.org/u/Totally_Not_A_Robot)\ **Post date:** [January 4, 2019, 3:05pm UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/6 "2019-01-04T15:05:37Z")
> [@raghu999](#): > > I family instances Ahhh! I get it now! K versus I instances on AWS 🙂 And here I thought there was a whole IBM architecture that I had somehow missed out on 😃 Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [January 18, 2019, 3:05pm UTC](https://community.graylog.org/t/graylog-as-aggregator-and-etl-tool/8249/7 "2019-01-18T15:05:37Z")
Closed on January 18, 2019, 3:05pm UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.