NXLOG troubles with sidecar

[LeMeow]

Hello everyone,

Something is bothering me for quite a while now, I can’t make the sidecar work on Debian 8 with NXLOG, I had no problems with filebeat or syslog whatsoever.
The current issue is the sidecar launching NXLOG via the path /usr/bin/nxlog as configured in the sidecar.yml but it doesn’t seem to register that NXLOG has launched, therefore the logs from the sidecar :

$timestamp : level=info msg="[nxlog] Starting (exec driver)"
$timestamp : level=error msg="[nxlog] Backend finished unexpectedly, trying to restart 3/3."
$timestamp : level=info msg="[nxlog] Starting (exec driver)"
$timestamp : level=error msg="[nxlog] Unable to start collector after 3 tries, giving up!"
$timestamp : level=error msg="[nxlog] Collector output: $timestamp ERROR Another instance is already running (pid 4851);Resource temporarily unavailable\n"

I tried to reinstall following the documentation to be sure, and still have the same error, I also tried on a Ubuntu 18.04.5 LTS with the same results.
I’ve checked if the nxlog service is really disable and it is following the update-rc.d command.
I’ve also checked the PID file to check if it matched the running process, it does.
And the last check that makes me doubt of nxlog working with the sidecar is the fact that when doing a systemctl status I see that the PID that is “already running” is in fact in the same cgroup as the sidecar, therefor launched by the sidecar.

My guess is that the sidecar does launch Nxlog but fails to monitor it as in the early versions of the sidecar. Does anyone uses nxlog on Linux and have encountered the same issue ? I had no issues on the Windows side.

NXLOG Conf :

######################################
# Global Directives                  #
######################################

User nxlog
Group nxlog

define ROOT /etc/nxlog

LogFile /var/log/nxlog.log
LogLevel INFO
CacheDir %ROOT%/cache_dir
SpoolDir %ROOT%/spool_dir

######################################
# Modules                            #
######################################

<Extension _syslog>
    Module	xm_syslog
</Extension>

<Extension json>
    Module	xm_json
</Extension>

<Input local>
    Module	im_file
    File	"/var/log/*"
    SavePos	TRUE
    ReadFromLast TRUE
</Input>

<Output graylog>
    Module	om_tcp
    Host	$IP
    Port	$PORT
    <Exec>
	parse_syslog();
	$Hostname ="Test";
	delete($EventReceivedTime);
	delete($SourceModuleName);
	delete($SourceModuleType);
	delete($SyslogFacility);
	delete($ProcessID);
	delete($SeverityValue);
	delete($SyslogFacilityValue);
	delete($SyslogSeverityValue);
	to_json();
    </Exec>
</Output>

######################################
# Routes                             #
######################################

<Route 1>
    Path	local => graylog
</Route>

Sidecar Conf :

server_url: $URL
server_api_token: $TOKEN
node_id: "file:/etc/graylog/sidecar/node-id"
update_interval: 30
tls_skip_verify: false
send_status: true
log_path "/var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
list_log_files:
    - /var/log
collector_binaries_whitelist:
    - "/usr/bin/nxlog"
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"