Graylog with curator?


(Jason Gould) #1

I know that graylog supports archiving in the commercial solution. But I’m curious how difficult it would be to use curator to perform this archive/retention behavior for a small shop without the budget.

Thanks.


(Jochen) #2

You can use an external program such as Curator to rotate and delete/archive indices in Graylog.

Just remember to use a rotation strategy which doesn’t rotate indices by itself, e. g. a message count based strategy with a really high number of messages or the “noop” rotation strategy once it’s been merged and released:
https://github.com/Graylog2/graylog2-server/pull/4207

You also have to trigger a recalculation of index ranges if you’ve rotated indices with an external program.


(Jason Gould) #3

Thanks. If I’m understanding you correctly, basically you are saying to set the config in Graylog so that it doesn’t interfere with your external program (ie. Curator). To do that currently you would change the rotation strategy to a really high message number (if you plan to handle it X days could you change the strategy in graylog to X+1 days?). Eventually the “noop” rotation strategy will be added to graylog so we can set it to that instead.

With that in place just run Curator as normal. Then trigger a recalculation so graylog is aware of the rotated indices. Is this correct (from the docs);

curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild

Thanks


(Jochen) #4

This would recalculate the index ranges of all indices, which might not be necessary.

You can also rebuild the index ranges of a specific index set or even a specific index using these resources in the Graylog REST API:

/system/indices/ranges/index_set/{indexSetId}/rebuild
/system/indices/ranges/{indexName}/rebuild

(John Buchanan) #5

Hopefully this is still deemed on-topic, but I’m using Curator to close indices older than 90 days, to manage heap space on my Elasticsearch data nodes. Also, to manage overall disk space I calculated that we could have approximately 315 indices on disk and stay at about 75% usage on the data nodes, so I reconfigured my Graylog index rotation with a max number of indices of 315 and rotation strategy of delete (I’m also using a rotation strategy of Time - P1D).

A week or so ago we reached and then exceeded an index count of 315, but Graylog did not begin deleting the oldest. When I realized this we were at 322 indices. Would this be due to my closing the indices > 90 days? Will Graylog only act upon open indices?

My solution was to add another action to Curator to delete older than 315, which is working fine, but seeing this other reference to Curator I figured I’d ask.

John


(Jochen) #6

Yes, Graylog will only include open indices in its calculation for rotation and retention.


(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.