Configuring logs from certain inputs to other disks/elasticsearch node?

Graylog Central (peer support)
1

I’ve been using a simple graylog config for a few months (elastic,mongo, graylog all on one server).
Now I want to add some more inputs which will require more storage space.

I wanted to store these logs on a separate physical disk but didn’t see how to configure that on the single graylog server.
Then I thought if i created a new elasticsearch node, I could add the disk there and configure graylog to store those inputs on only that node.

I created the elasticsearch node and it is communicating properly with the one on the graylog box. But I still don’t see any option to configure a specific input to store it’s data on only the new elasticsearch node.

What I am seeing is that graylog duplicated the already existing data onto the new elastic node.

Is what I am trying to do possible with graylog? If it is, where are the configuration settings?

2

You can not easily give Graylog the configuration to store specific inputs on specific Servers or Disks. In addition it is not really possible to give Elasticsearch the configuration to store specific messages on specific disks. It will always create a big bucket out of the available storage.

What you have done is scaling Elasticsearch and with the given index configuration it scales now to the two configured nodes.

Are you happy with the current setup or did you want to change something?

3

Then, ignoring servers and disks, can I:

  1. Configure one input in its own index
  2. Configure that index to use only one elasticsearch node

I guess the answer should be here: http://docs.graylog.org/en/2.4/pages/configuration/index_model.html?highlight=index

but I’m having trouble understanding if what I want do is mentioned there or not.

4
  1. create a new index set and create a Stream that takes all messages from that input.
  2. basically
5

For #1:
I created a new index set called ‘EXT test’.
Then created a stream called ‘EXT stream’.
Created a rule so the stream sends it to the ‘EXT test’ index.
I can see in graylog this is working as intended - the data is being stored in the new index set.

For #2:
I haven’t done it yet but I think I get how to give the node an attribute so that the data can be forced into one ‘zone’.
But I am having trouble understanding how to properly create a new index template.
It isn’t clear to me from the documentation how that can assign the data to the index set.
The example given shows how to specify new fields but nothing about assigning data to a new index set.

Could you show an example of what index template I would need to add?
And this index template - it would need to be added on both nodes or just one?
Even then, how does this make it so the new index set is storing data only on the ‘zone’ I choose?

6

This is how you route indices “direct” between different nodes:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/shard-allocation-filtering.html

As starting point the following will place all indices of the nodes with the attribute box_type and the value of that set to hot and it will set the refresh interval to 30 seconds.

{
  "order": 2,
  "template": "*",
  "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        }
      },
      "refresh_interval": "30s",
  },
  "mappings": {},
  "aliases": {}
}
7

Thanks! Had to go over things a few times to finally understand this but finally got it working the way I wanted.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.