Windows Sidecar (Winlogbeat) Service Problem - "Received no ping signal from sidecar"

[greavette]

Graylog Server version 4.1
Graylog Sidecar Version (windows): 1.1.0

Hello Graylog Forum,

I’ve successfully installed sidecar on my graylog server and configured it to talk to sidecar’s setup on my Windows and Linux Servers. I’ve also successfully installed the Windows Sidecar on a Windows 10 and Windows 2019 server and following the instructions on this page: Graylog Sidecar
I issued these commands in an elevated command prompt admin window to install the service:

“C:\Program Files\graylog\sidecar\graylog-sidecar.exe” - service install

The service does install and I set it to start automatically and the service is running. But in my Graylog > sidecar console I don’t see the server. The server status is Unknown and the error is that I’ve “Received no ping signal from sidecar”.

So looking at the documentation I see that I can start the service from the elevated command prompt so I issued this command:

“C:\Program Files\graylog\sidecar\graylog-sidecar.exe” - service start

In my Graylog console the server now shows up as running and logs are collected. BUT…the elevated command prompt does not close on my windows computer. The service was started. If I close the command prompt the service stops and my Graylog console again shows the windows computer as Unknown. So I went to my Services in the windows computer and started the service. The service shows as running but again my Graylog console shows the windows computer as Uknown.

So it appears the only way I can get my Windows sidecar services to connect to my Graylog server is to start the service with elevated rights. It seems I have a permissions or rights issue with how my windows sidecar service is run. I installed the windows sidecar with an account that is in the Administrator’s group. My windows service uses the local system account.

Any ideas from this community on where I need to make a change on my windows computers to get my service running properly and stay connected to my Graylog server?

Thank you in advance for any help you can provide me.

[tmacgbay]

What does your sidecar.yml look like on the machine you are having problems with? For instance below is mine: Feel free to post your but obfuscate the secret parts and use the forum tools like </> to make it readable.

server_url: http://FatGraylogServer:9000/api/
server_api_token: "Too_Secret" 
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
collector_id: file:C:\Program Files\Graylog\sidecar\collector-id
cache_path: C:\Program Files\Graylog\sidecar\cache
log_path: C:\Program Files\Graylog\sidecar\logs
log_rotation_time: 86400
log_max_age: 604800
tags: [windows]
collector_binaries_whitelist: []
backends:
    - name: nxlog
      enabled: false
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\nxlog.conf
    - name: winlogbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\winlogbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\winlogbeat.yml
    - name: filebeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\filebeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\filebeat.yml
    - name: auditbeat
      enabled: false
      binary_path: C:\Program Files\Graylog\sidecar\auditbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\auditbeat.yml

[greavette]

Thanks very much @tmacgbay for your reply to my post. Here is my sidecar.yml file for your review.

server_url: “http://mysecretip:9000/api”
server_api_token: “mysecrettoken”
node_id: “file:C:\Program Files\Graylog\sidecar\node-id”
node_name: “WindowsVM1”
update_interval: 10
tls_skip_verify: false
send_status: true

I’ve only included the lines in my .yml file that were not commented out.

Let me know if you need anything further.

Thank you.

[tmacgbay]

My suggestion is that you adjust your sidecar.yml based on the template design of my sidecar.yml. Not all config lines are required but it should be reasonably clear which are . For instance, you haven’t described your backends (The backends: section is sensitive to spacing…) are you using beats? nxlog?. Other things of note - you are using node_name which is fine but you should know that by default sidecar will use the hostname of the machine it’s running on.

[greavette]

Hello @tmacgbay,

I’ve tried the configuration you have provided. I then stopped and then started my Graylog collector service from services.msc and my Windows computer did not report into Graylog > Sidecars.

So I tried the windows command prompt (elevated) and it started my Graylog service and immediately my Graylog server could see the Windows computer. I did this test twice with the .yml config you suggested and I went back to the .yml config I had initially used. Both configs work but only if I issue the service start from the windows command prompt.

Here is what I see in the Windows command prompt when I issue the start command:

C:\Program Files\Graylog\sidecar>graylog-sidecar.exe - service start
time=“2021-11-10T17:39:26-05:00” level=info msg=“Using node-id: c5813ead-5559-4c74-9f66-bd6f9c7c8c5f”
time=“2021-11-10T17:39:26-05:00” level=warning msg=“collector_binaries_whitelist is deprecated. Migrate your configuration to collector_binaries_accesslist.”
time=“2021-11-10T17:39:26-05:00” level=info msg=“Starting signal distributor”
time=“2021-11-10T17:39:36-05:00” level=info msg=“Adding process runner for: winlogbeat”
time=“2021-11-10T17:39:36-05:00” level=info msg=“[winlogbeat] Configuration change detected, rewriting configuration file.”
time=“2021-11-10T17:39:37-05:00” level=info msg=“[winlogbeat] Starting (svc driver)”

The command does not end and bring me back to the “C:\Program Files\Graylog\sidecar>” prompt so I can exit out of the windows command prompt. I’m stuck with this window open. If I forcefully close the window my service stops and doesn’t connect to my Graylog server.

It’s something in the way Windows allows this service to start? From the command prompt it works fine…but from the services.msc it does not work even though the Graylog service is running.

Any ideas what I’m doing wrong?

Thank you…

[tmacgbay]

My guess is permissions - try changing the service to run as the account that you are using in the command window.

[greavette]

I’ve logged in as Administrator and uninstalled graylog sidecar on this Windows 10 computer. I’ve the installed fresh the windows sidecar and configured it to point to my graylog server. Once again the command prompt instruction I use will start the service but it will not exit so my command prompt window is opened and cannot be closed or else the graylog service will stop. Here is a screenshot of what I see. You will notice that it has not return from the command and I cannot close the window:

Is there any ideas of how I install this from the command prompt and have me manage it from the services.msc so it works to connect to my graylog server?

Thank you.

[tmacgbay]

It seems odd that the docs have the “-” disconnected from the “service”… I recall that it was more like this:

& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install
& "C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start

I don’t know if that will make a difference but you could try. There is also log files of what sidecar is doing locally located at C:\Program Files\Graylog\sidecar\logs that might give you a better idea of what is going on. Post the relevant parts of them if you need help figuring them out… though I find them pretty easy to understand. (use the forum tool </> in the text window to make the log file readable)

[greavette]

Hi @tmacgbay , Wow! That was the issue all along for me! once I removed the space between the hyphen and the word service, the command would finish successfully and I can close my command prompt. Thank you very much for helping me solve this issue.

I have a couple new questions for you related to this work I did to get my services working if you have time to review.

I had re-installed one of my windows sidecar with winlogbeats and it seems my re-installing on my Windows Server has now added two sidecar’s showing up in my Graylog > sidecar > Overview with one being in failed status and one in running status. I’m guessing the one in failed status was the one I uninstalled. I can’t find a way to delete this from my Graylog Server. Is there a command I can issue to purge this sidecar? It doesn’t show up as Active but I’d like to keep it clean and remove it.

I also have on another Windows 2019 server my Graylog Sidecar Client status showing up in the Graylog > sidecar > Overview as Failing where the error on my winlogbeat says:

SetupEventLogSource() failed: SYSTEM\CurrentControlSet\Services\EventLog\Application\graylog-collector-winlogbeat registry key already exists

What would be the fix to remove this error so my client sidecar status shows as running and healthy.

Thank you.

[tmacgbay]

Usually best to open a new topic for further questions to make things more searchable for future readers.

Unless it has changed since last I asked that question, the old sidecar with a failed status will roll off in a week or so and there is no provision to remove it… short of monkeying around in MongoDB… which I didn’t deem worth figuring out…

For the 2019 server I would do an uninstall of it all, check the registry item and remove it if it still exists… then reinstall.

[greavette]

Very good advice @tmacgbay . I will open a new topic for further questions. But thank you for your input into my post. Really appreciate your help and for helping me find root cause for my initial question about installing/running the windows service properly.

Cheers!