Nxlog sidecar collector and local nxlog service ARE able to run at the same time

[cesq]

Looking at the documentation for setting up sidecar collectors and the docs state that before launching nxlog by graylog-sidecar you have to stop, disable and unconfigure the local nxlog instance:

Because the Sidecar takes control of stopping and starting NXlog, it is necessary to stop all running instances of NXlog and unconfigure the default system service

I’ve only now seen this part of the docs, after having completed almost dozen of migrations from local nxlog to nxlog via graylog-sidecar on countless Linux and Windows hosts.

During which I’ve had an approach of setting up nxlog via graylog-sidecar but NOT disabling local nxlog instances.
Simply because after having added all the necessary hosts to sidecar I wanted to compare wether the messages arriving from both sources were exactly the same. Only then I disabled the locally running nxlog service. I did this because sometimes I may have assigned the wrong configurations or messed them up and wanted to be 100% sure that every host sent every log that it should have. Graylog servers on which I’ve done the migrations were heavily utilised for production purposes and this way of doing things was a lot safer, more comfortable and less stressful. For the duration of the switch I had a temporary index that I deleted after the change in order to get rid of the duplicates.

But I digress, the thing I wanted to say is that despite what the docs say, you can run two instances of nxlog - of which one is local and one is handled by sidecar.

As to how?: I have no idea. looking at the nxlog logs, it looks like the the program just straight up launches a second instance and calls it a day. No warnings, no nothing. The local instance just keeps running.

The only thing to be mindful of is Membuffers and Diskbuffers. If you use them in your configuration, you have to name them differently in the sidecar config as it doesn’t have permissions to access the local nxlog’s buffers.

Out of almost 1000 different Linux, Windows hosts, distros, versions, the only host on which it hasn’t worked was an archaic Centos 6 host running Kernel 2.6.

So is it safe to say that the docs can be updated with this info? It’s a really great feature to have and saves you a lot of trouble.

[Joel_Duffield]

I think this has been left out of the docs on purpose to avoid confusion, most people only want to be running a single instance, and yes because of the way sidecar runs it causes people to be running two agents and sending in duplicate logs without knowing it, causing all kinds of problems.

[cesq]

@Joel_Duffield Okay, gotcha. But we can we at least get a mention of it? Had I looked at the docs first I probably wouldn’t bother trying to run two instances and would have way more work to do with getting the migrations approved, potentionally missing out on important logs. Overall the change would be way more serious and disruptive to the production. I just feel like someone (though granted, you are right, most users don’t do this) might be in a similliar situation.

Maybe a little mention would be appropriate? Something like:

Though it is not reccommended nor oficially supported, it is possible in most cases to run two concurrent instances of NXlog, one via sidecar and one local.

While keeping all the other content of this documentation page.