jfarr2008
July 31, 2020, 7:20pm
1
Trying to finish migrating my sidecars and I’ve installed the latest sidecar exe. I’ve installed the service and issued the command to start the service. The service never starts and I don’t see the sidecar show up in my active view in the overview. I start the sidecar interactively and all works like a champ. I’ve uninstalled the service and reinstalled it, but still no luck getting it running. Going into services.msc and trying to start it as well yields error 1067. Found someone else that was experiencing that issue and they were able to resolve with a reboot. I can start the sidecar interactively but I get error messages in the logs:
time="2020-07-31T14:09:13-05:00" level=info msg="Starting signal distributor"
time="2020-07-31T14:09:23-05:00" level=info msg="Adding process runner for: winlogbeat"
time="2020-07-31T14:09:23-05:00" level=info msg="[winlogbeat] Configuration change detected, rewriting configuration file."
time="2020-07-31T14:09:24-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:09:33-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:09:33-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:09:43-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:09:43-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:09:53-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:09:53-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:03-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:03-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:13-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:13-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:23-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:23-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:33-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:33-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:43-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:43-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:10:53-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:10:53-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:11:03-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:11:03-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:11:13-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:11:13-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:11:23-05:00" level=error msg="[winlogbeat] Backend finished unexpectedly, sending restart signal"
time="2020-07-31T14:11:23-05:00" level=info msg="[winlogbeat] Starting (svc driver)"
time="2020-07-31T14:11:28-05:00" level=info msg="Stopping signal distributor"
time="2020-07-31T14:11:28-05:00" level=info msg="[winlogbeat] Stopping"
Starting within services.msc gives me the visual error 1067 (See below).
Service install from powershell or command line works properly, but starting the service from command line or powershell looks to work properly, but the service never starts. Rebooting the machine does not correct this. I’ve tried this with both version 1.0.2, 1.0.1 and 1.0.0.
Any suggestions are appreciated!
Hey @jfarr2008 ,
If you can start it interactively but not as a service that suggests to me a permissions issue. If you set the service to use your credentials does it start as expected?
tmacgbay
July 31, 2020, 8:21pm
3
What is your sidecar YML configuration in C:\Program Files\Graylog\sidecar\sidecar.yml
jfarr2008
July 31, 2020, 9:35pm
5
server_url: "http://graylogserverurl.local:9000/api"
server_api_token: "my-api-token"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_name: ""
update_interval: 10
tls_skip_verify: false
send_status: true
jfarr2008
July 31, 2020, 9:51pm
6
Will not start even after I set it to use my credentials and still wouldn’t after setting it to use a service account.
tmacgbay
August 1, 2020, 1:50am
7
seems a bit slim… here is the working one from my setup:
server_url: http://graylog:9000/api/
server_api_token: "<SuperSecretCodeHere>"
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
collector_id: file:C:\Program Files\Graylog\sidecar\collector-id
cache_path: C:\Program Files\Graylog\sidecar\cache
log_path: C:\Program Files\Graylog\sidecar\logs
log_rotation_time: 86400
log_max_age: 604800
tags: [windows]
collector_binaries_whitelist: []
backends:
- name: nxlog
enabled: false
binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
configuration_path: C:\Program Files\Graylog\sidecar\generated\nxlog.conf
- name: winlogbeat
enabled: true
binary_path: C:\Program Files\Graylog\sidecar\winlogbeat.exe
configuration_path: C:\Program Files\Graylog\sidecar\generated\winlogbeat.yml
- name: filebeat
enabled: true
binary_path: C:\Program Files\Graylog\sidecar\filebeat.exe
configuration_path: C:\Program Files\Graylog\sidecar\generated\filebeat.yml
- name: auditbeat
enabled: true
binary_path: C:\Program Files\Graylog\sidecar\auditbeat.exe
configuration_path: C:\Program Files\Graylog\sidecar\generated\auditbeat.yml
1 Like
jfarr2008
August 1, 2020, 10:57am
8
Which version are you running?
tmacgbay
August 1, 2020, 4:26pm
9
Graylog 3.3.3 but this is the same config as the 2.x days. The sidecar version is a mix of 1.0.1 and 1.0.2
jfarr2008
August 3, 2020, 2:49pm
10
Fell back to my 2.x config as you mentioned. Same issues persist. It seems like a permission issue or something as @ttsandrew mentioned.
tmacgbay
August 3, 2020, 4:37pm
11
If it turns out to be permissions post what you changed to fix it - if it helps I can post up permissions in out environment (they are default)
jfarr2008
August 3, 2020, 5:07pm
12
Will do, I’m digging into this today to see what I can find out. I’ve never had to mess with the permissions much in previous versions.
tmacgbay
August 3, 2020, 7:28pm
13
You could try uninstalling, then make sure the services are removed with regedit. If you delete them yourself, ALWAYS make a backup of it first.
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\graylog-sidecar
If you have to delete the keys yourself, you will likely need to reboot.
In case it is just adjusting settings here are what mine currently show:
2 Likes
jfarr2008
August 3, 2020, 7:30pm
14
Awesome. Thank you for the tip. I’ll give this a shot now.
1 Like
jfarr2008
August 4, 2020, 3:14pm
15
Registry is clean, no remnants of those keys…
tgarons
August 14, 2020, 4:33pm
16
Is there any fix for this? We are running into the exact same issue—runs fine from the command line, Error 1067 from Services.
I’m having the same issue
tgarons
August 15, 2020, 5:01pm
18
If it makes any difference—we are having the same problem with the deprecated collector-sidecar: runs from CLI, 1067 from services. This happened when we upgraded to Windows Server 2019.
Our issue was resolved when we whitelisted the sidecar directory in our antivirus application.
1 Like
tgarons
August 17, 2020, 7:51pm
20
I don’t see any evidence that our AV, Carbon Black Defense, has blocked anything.
