# Configuring logs from certain inputs to other disks/elasticsearch node?

**URL:** <https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418>\
**Category:** Graylog Central (peer support)\
**Created:** [August 14, 2018, 2:14pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418 "2018-08-14T14:14:19Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![cbrhue](https://avatars.discourse-cdn.com/v4/letter/c/f6c823/32.png) [@cbrhue](https://community.graylog.org/u/cbrhue)\
**Post date:** [August 14, 2018, 2:14pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/1 "2018-08-14T14:14:19Z")

</div>

I’ve been using a simple graylog config for a few months (elastic,mongo, graylog all on one server).  
Now I want to add some more inputs which will require more storage space.

I wanted to store these logs on a separate physical disk but didn’t see how to configure that on the single graylog server.  
Then I thought if i created a new elasticsearch node, I could add the disk there and configure graylog to store those inputs on only that node.

I created the elasticsearch node and it is communicating properly with the one on the graylog box. But I still don’t see any option to configure a specific input to store it’s data on only the new elasticsearch node.

What I am seeing is that graylog duplicated the already existing data onto the new elastic node.

Is what I am trying to do possible with graylog? If it is, where are the configuration settings?

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [August 15, 2018, 8:51am UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/2 "2018-08-15T08:51:35Z")

</div>

You can not easily give Graylog the configuration to store specific inputs on specific Servers or Disks. In addition it is not really possible to give Elasticsearch the configuration to store specific messages on specific disks. It will always create a big bucket out of the available storage.

What you have done is scaling Elasticsearch and with the given index configuration it scales now to the two configured nodes.

Are you happy with the current setup or did you want to change something?

---

<div class="post-metadata">

**Author:** ![cbrhue](https://avatars.discourse-cdn.com/v4/letter/c/f6c823/32.png) [@cbrhue](https://community.graylog.org/u/cbrhue)\
**Post date:** [August 15, 2018, 1:02pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/3 "2018-08-15T13:02:48Z")

</div>

Then, ignoring servers and disks, can I:

1. Configure one input in its own index
2. Configure that index to use only one elasticsearch node

I guess the answer should be here: [http://docs.graylog.org/en/2.4/pages/configuration/index\_model.html?highlight=index](http://docs.graylog.org/en/2.4/pages/configuration/index_model.html?highlight=index)

but I’m having trouble understanding if what I want do is mentioned there or not.

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [August 15, 2018, 2:06pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/4 "2018-08-15T14:06:51Z")

</div>

1. create a new [index set](http://docs.graylog.org/en/2.4/pages/configuration/index_model.html#index-set-configuration) and [create a Stream](http://docs.graylog.org/en/2.4/pages/streams.html#how-do-i-create-a-stream) that takes all messages from that input.
2. basically
  - give the ES Node one [attribute](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/allocation-awareness.html)
  - create [index template](http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings) that is assigned to the previous created index set

---

<div class="post-metadata">

**Author:** ![cbrhue](https://avatars.discourse-cdn.com/v4/letter/c/f6c823/32.png) [@cbrhue](https://community.graylog.org/u/cbrhue)\
**Post date:** [August 16, 2018, 4:57pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/5 "2018-08-16T16:57:18Z")

</div>

For #1:  
I created a new index set called ‘EXT test’.  
Then created a stream called ‘EXT stream’.  
Created a rule so the stream sends it to the ‘EXT test’ index.  
I can see in graylog this is working as intended - the data is being stored in the new index set.

For #2:  
I haven’t done it yet but I think I get how to give the node an attribute so that the data can be forced into one ‘zone’.  
But I am having trouble understanding how to properly create a new index template.  
It isn’t clear to me from the documentation how that can assign the data to the index set.  
The example given shows how to specify new fields but nothing about assigning data to a new index set.

Could you show an example of what index template I would need to add?  
And this index template - it would need to be added on both nodes or just one?  
Even then, how does this make it so the new index set is storing data only on the ‘zone’ I choose?

---

<div class="post-metadata">

**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\
**Post date:** [August 17, 2018, 8:05am UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/6 "2018-08-17T08:05:54Z")

</div>

This is how you route indices “direct” between different nodes:

[https://www.elastic.co/guide/en/elasticsearch/reference/5.6/shard-allocation-filtering.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/shard-allocation-filtering.html)

As starting point the following will place all indices of the nodes with the attribute `box_type` and the value of that set to `hot` and it will set the refresh interval to 30 seconds.

```auto
{
  "order": 2,
  "template": "*",
  "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "30s",
  },
  "mappings": {},
  "aliases": {}
}

```

---

<div class="post-metadata">

**Author:** ![cbrhue](https://avatars.discourse-cdn.com/v4/letter/c/f6c823/32.png) [@cbrhue](https://community.graylog.org/u/cbrhue)\
**Post date:** [August 22, 2018, 4:44pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/7 "2018-08-22T16:44:02Z")

</div>

Thanks! Had to go over things a few times to finally understand this but finally got it working the way I wanted.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [September 5, 2018, 4:44pm UTC](https://community.graylog.org/t/configuring-logs-from-certain-inputs-to-other-disks-elasticsearch-node/6418/8 "2018-09-05T16:44:03Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.