Multiple nodes or cluster, different retention periods

[sgarciam (Stephen)]

Hi All,

Version: 2.2.3+7adc951, codename Stiegl

So Im wanting to add a couple servers to my single graylog server setup and wanted some input on what the right way to approach it would be

This is the picture:

(new) Graylog server on the DMZ receiving logs from hosts outside our premises.
It would retain logs for maybe an hour. Main thing is it would have GELF outputs to our main server.

(current) Main server receiving all logs. Retaining for about two weeks.

(new) Long term server. We would determine what streams on the main server need a longer retention period, say file auditing for example. We would have GELF outputs for those pointing to this one. This server will retain for 6 months to a year, depending on compliance requirements.

Would this be considered a multi node cluster or just 3 servers talking to each other ( I haven’t messed with graylog clusters before ). What would be the best approach to this? Is this approach even a good idea? Thoughts?

Thanks,
Stephen

[jtkarvo]

You can have different retention periods with a single server. Just create a new index set and make pipeline rules to sort message to the right index sets.

That would probably be more robust than forwarding logs between graylog servers, although I see your point with the separate DMZ server.

You could make a graylog cluster, too, but that would be a different thing. Making a cluster setup, I would recommend starting by making the Elasticsearch servers a separate cluster with 3 servers (and later more if you have a lot of load). This has the added benefit of more resilience: you can then set the number of replicas to 1 in the index sets page.