Rotation of indexe using Curator

Hello,
I know there’s an archiving option with Graylog Enterprise, but how do you do that for a company that doesn’t have the budget?
I read that there is a method of archiving indexes using Curator (Graylog with curator?) but there are a lot of things I don’t understand.
What is a noop rotation strategy? And is what you really need to do as many conf files as specify here:

And then, my server is almost saturated in disk space, will I have to start a recalculation of all my index ranges?

Thank you

To repeat my question, I just saw that Curator must be used with a HOT/WARM architecture, however my hard drive is already almost saturated

image

Can I adapt my architecture without deleting data?

Thank you

Hi @Cruciani_Elliot I had a similar problem a while ago.

Here’s what I did to solve it:

Hi @reimlima, thank you for the anwser,

what you sent me is going to be very useful to me, however I do not believe that you are talking in the processing of indexes that are already created and that is what may cause me problem since I would like to archive them without deleting them

Thank you

Hi @Cruciani_Elliot,

Yes, this is about created indexes, I just forgot to mention that you have to configure index rotation in your Graylog UI (sorry about that).

To solve the problem I had to choose “Rotation period: P1D (1 day, a day)” as an option.

You can run all this process manually, so you don’t have to wait until next day to see if it works.

thank you @reimlima

I have a last question: I didn’t understood the action:allocation on your curator config

Hi,

so, “allocation” is one type of actions Curator can do, it tells Curator to migrate your index from one node to another based on filters you define.

See my configuration as an example:

action: allocation
    description: "Apply shard allocation filtering rules to the hot log"
    options:
      key: box_type
      value: warm
      allocation_type: require
      wait_for_completion: True
      max_wait: 3600
      timeout_override:
      continue_if_exception: False
      disable_action: False
      allow_ilm_indices: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: graylog_
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        unit_count: 1

A little of explaining, basically it will migrate indexes named like “graylog_” and older than 1 day from hot nodes to warm nodes.

Curator will know the nodes based on the key “box_type” with value “warm”.

You must tag your nodes using the option in the elasticsearch.yml file:

node.attr.box_type: hot # to hot nodes

And

node.attr.box_type: warm # to, you guess it, warm nodes

It strongly recomended that you take a moment to read the Curator’s doc to understand how to configure and run it.

1 Like

Hi @reimlima, thank you for the anwsers, I’ll try to do that

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.