Error Starting Sidecar Service on Windows

Trying to finish migrating my sidecars and I’ve installed the latest sidecar exe. I’ve installed the service and issued the command to start the service. The service never starts and I don’t see the sidecar show up in my active view in the overview. I start the sidecar interactively and all works like a champ. I’ve uninstalled the service and reinstalled it, but still no luck getting it running. Going into services.msc and trying to start it as well yields error 1067. Found someone else that was experiencing that issue and they were able to resolve with a reboot. I can start the sidecar interactively but I get error messages in the logs:

time="2020-07-31T14:09:13-05:00" level=info msg="Starting signal distributor" 
time="2020-07-31T14:09:23-05:00" level=info msg="Adding process runner for: winlogbeat" 
time="2020-07-31T14:09:23-05:00" level=info msg="[winlogbeat] Configuration change detected, rewriting configuration file." 
time="2020-07-31T14:09:24-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:09:33-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:09:33-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:09:43-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:09:43-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:09:53-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:09:53-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:03-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:03-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:13-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:13-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:23-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:23-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:33-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:33-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:43-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:43-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:10:53-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:10:53-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:11:03-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:11:03-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:11:13-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:11:13-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:11:23-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal" 
time="2020-07-31T14:11:23-05:00" level=info msg="[winlogbeat] Starting (svc driver)" 
time="2020-07-31T14:11:28-05:00" level=info msg="Stopping signal distributor" 
time="2020-07-31T14:11:28-05:00" level=info msg="[winlogbeat] Stopping" 

Starting within services.msc gives me the visual error 1067 (See below).

graylog1067

Service install from powershell or command line works properly, but starting the service from command line or powershell looks to work properly, but the service never starts. Rebooting the machine does not correct this. I’ve tried this with both version 1.0.2, 1.0.1 and 1.0.0.

Any suggestions are appreciated!

Hey @jfarr2008,

If you can start it interactively but not as a service that suggests to me a permissions issue. If you set the service to use your credentials does it start as expected?

What is your sidecar YML configuration in C:\Program Files\Graylog\sidecar\sidecar.yml

Let me give that a try.

server_url: "http://graylogserverurl.local:9000/api"

server_api_token: "my-api-token"

node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"

node_name: ""

update_interval: 10

tls_skip_verify: false

send_status: true

Will not start even after I set it to use my credentials and still wouldn’t after setting it to use a service account.

seems a bit slim… here is the working one from my setup:

server_url: http://graylog:9000/api/
server_api_token: "<SuperSecretCodeHere>" 
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
collector_id: file:C:\Program Files\Graylog\sidecar\collector-id
cache_path: C:\Program Files\Graylog\sidecar\cache
log_path: C:\Program Files\Graylog\sidecar\logs
log_rotation_time: 86400
log_max_age: 604800
tags: [windows]
collector_binaries_whitelist: []
backends:
    - name: nxlog
      enabled: false
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\nxlog.conf
    - name: winlogbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\winlogbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\winlogbeat.yml
    - name: filebeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\filebeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\filebeat.yml
    - name: auditbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\auditbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\auditbeat.yml
1 Like

Which version are you running?

Graylog 3.3.3 but this is the same config as the 2.x days. The sidecar version is a mix of 1.0.1 and 1.0.2

Fell back to my 2.x config as you mentioned. Same issues persist. It seems like a permission issue or something as @ttsandrew mentioned.

If it turns out to be permissions post what you changed to fix it - if it helps I can post up permissions in out environment (they are default)

Will do, I’m digging into this today to see what I can find out. I’ve never had to mess with the permissions much in previous versions.

You could try uninstalling, then make sure the services are removed with regedit. If you delete them yourself, ALWAYS make a backup of it first.

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\graylog-sidecar
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\graylog-collector-winlogbeat

If you have to delete the keys yourself, you will likely need to reboot.

In case it is just adjusting settings here are what mine currently show:


2 Likes

Awesome. Thank you for the tip. I’ll give this a shot now.

1 Like

Registry is clean, no remnants of those keys…

Is there any fix for this? We are running into the exact same issue—runs fine from the command line, Error 1067 from Services.

I’m having the same issue

If it makes any difference—we are having the same problem with the deprecated collector-sidecar: runs from CLI, 1067 from services. This happened when we upgraded to Windows Server 2019.

Our issue was resolved when we whitelisted the sidecar directory in our antivirus application.

1 Like

I don’t see any evidence that our AV, Carbon Black Defense, has blocked anything.