We create and destroy dozens or hundreds of servers a day.
With the old collector-sidecar, it used to be the case that we just wrote tags into the collector-sidecar configuration and that was it. As soon as it was installed and working, it logged all the files it should.
So far so good. However it seems that with the new graylog-collector we now have to use an API call to associate nodes with configurations? That would require an entirely new control plane software waiting for nodes to come online, having an internal mapping which node is supposed to get which configuration, then set it via the API… That seems like a huge regression and complication. Am I missing something here? Surely there must be a different way but to develop an entire software ourselves?
thanks for your question. I noticed that your question wasn’t get a number of expert eyes on it since you posted it last week. I’ve moved to our Daily Challenges where you may be able to responses from the community.
I am writing the custom management software now and it seems we require admin permission to pull sidecar registration status? Is there a way to have a more fine grained permission structure? A lot of companies won’t be able to do that due to compliance reasons, thus won’t be able to use collectors.
In order to assign sidecars to configurations, you need to pass in an entire list of sidecars that are supposed to have that configuration. This is a non-solvable issue when you want the nodes themselves to register, because they cannot know what other nodes are supposed to have that config and might not be able to cleanly remove themselves. Instead, it is easier for us to just manage filebeat config ourselves.
In case a developer reads this:
graylog-sidecar is much more work intensive and much less useful than graylog-collector-sidecar. I would go so far as to say that it is useless. The previous solution was completely self-contained and just worked, as all you needed was to give nodes tags, which could be done very easily in config management.
The current solution is unusable in modern IT, because:
requires manual steps to add nodes
adding nodes requires admin privileges
So in the end, it is actually easier to simply manage the filebeat config yourself, as the collector at no point reduces the amount of work required. You need to set up the config anyway, better to just do it once in config management, than doing it in collector and then manually managing nodes.
As a freelancer, I recommended graylog to a lot of companies, but will not do so anymore. The reason is that you seem to have lost touch with your customer base. I see important core functionality just removed / changed. The only way I can explain things like quick values or tags for collectors being removed is that you didn’t actually ask customers about it beforehand.
