Rotation of indexe using Curator

Graylog Central (peer support)
1

Hello,
I know there’s an archiving option with Graylog Enterprise, but how do you do that for a company that doesn’t have the budget?
I read that there is a method of archiving indexes using Curator (Graylog with curator?) but there are a lot of things I don’t understand.
What is a noop rotation strategy? And is what you really need to do as many conf files as specify here:

And then, my server is almost saturated in disk space, will I have to start a recalculation of all my index ranges?

Thank you

2

To repeat my question, I just saw that Curator must be used with a HOT/WARM architecture, however my hard drive is already almost saturated

image

Can I adapt my architecture without deleting data?

Thank you

3

Hi @Cruciani_Elliot I had a similar problem a while ago.

Here’s what I did to solve it:

4

Hi @reimlima, thank you for the anwser,

what you sent me is going to be very useful to me, however I do not believe that you are talking in the processing of indexes that are already created and that is what may cause me problem since I would like to archive them without deleting them

Thank you

5

Hi @Cruciani_Elliot,

Yes, this is about created indexes, I just forgot to mention that you have to configure index rotation in your Graylog UI (sorry about that).

To solve the problem I had to choose “Rotation period: P1D (1 day, a day)” as an option.

You can run all this process manually, so you don’t have to wait until next day to see if it works.

6

thank you @reimlima

I have a last question: I didn’t understood the action:allocation on your curator config

7

Hi,

so, “allocation” is one type of actions Curator can do, it tells Curator to migrate your index from one node to another based on filters you define.

See my configuration as an example:

action: allocation
    description: "Apply shard allocation filtering rules to the hot log"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: True
      max_wait: 3600
      timeout_override:
      continue_if_exception: False
      disable_action: False
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: graylog_
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 1

A little of explaining, basically it will migrate indexes named like “graylog_” and older than 1 day from hot nodes to warm nodes.

Curator will know the nodes based on the key “box_type” with value “warm”.

You must tag your nodes using the option in the elasticsearch.yml file:

node.attr.box_type: hot # to hot nodes

And

node.attr.box_type: warm # to, you guess it, warm nodes

It strongly recomended that you take a moment to read the Curator’s doc to understand how to configure and run it.

1 Like
8

Hi @reimlima, thank you for the anwsers, I’ll try to do that

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.