Connecting Graylog Sidecar installed on SQL Server to Graylog

Description of your problem

I am trying to connect Graylog Sidecar (version 1.1.0-1) on an Azure SQL Managed Instance server to our Graylog server. I followed the instructions (Graylog Sidecar - Graylog Sidecar) but when I attempt to run “C:\Program Files\graylog\sidecar\graylog-sidecar.exe” -service install` I receive one of the following errors:
If public IP address is used:
‘time=“2021-11-16T14:16:14Z” level=error msg="[UpdateRegistration] Failed to report collector status to server: Put “http://[public ip]:9000/api/sidecars/86a2c07e-9b74-426f-af30-8dbedd7e9bc3”: dial tcp [public ip]:9000: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."’

If private IP address is used:
‘time=“2021-11-16T14:13:32Z” level=error msg="[UpdateRegistration] Failed to report collector status to server: Put “http://[private ip]:9000/api/sidecars/86a2c07e-9b74-426f-af30-8dbedd7e9bc3”: EOF"’

At this point I’m stumped on what to try next to resolve this issue. I’ve looked into other Graylog Community threads on this exact issue but found no solution to this.

Below is the sidecar.yml file:

# The URL to the Graylog server API.
# Default: "http://127.0.0.1:9000/api/"
server_url: "http://[private ip]:9000/api/"

# The API token to use to authenticate against the Graylog server API.
# Default: none
server_api_token: "mso53nr3vi1id42ege46pmgggff5u6qjqcr24160gemn7e0bkm0"

# The node ID of the sidecar. This can be a path to a file or an ID string.
# If set to a file and the file doesn't exist, the sidecar will generate an
# unique ID and writes it to the configured path.
#
# Example file path: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
# Example ID string: "6033137e-d56b-47fc-9762-cd699c11a5a9"
#
# ATTENTION: Every sidecar instance needs a unique ID!
#
# Default: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
#node_id: "b20c8480-0cd9-43b7-afcf-9212d536ac47"

# The node name of the sidecar. If this is empty, the sidecar will use the
# hostname of the host it is running on.
# Default: ""
node_name: ""

# The update interval in secods. This configures how often the sidecar will
# contact the Graylog server for keep-alive and configuration update requests.
# Default: 10
update_interval: 10

# This configures if the sidecar should skip the verification of TLS connections.
# Default: false
tls_skip_verify: false

# This enables/disables the transmission of detailed sidecar information like
# collector statues, metrics and log file lists. It can be disabled to reduce
# load on the Graylog server if needed. (disables some features in the server UI)
# Default: true
send_status: true

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
list_log_files: 
  - "C:\\Program Files\\Microsoft SQL Server\\MSSQL13.MSSQLSERVER\\MSSQL\\Log"

# Directory where the sidecar stores internal data.
#cache_path: "C:\\Program Files\\Graylog\\sidecar\\cache"

# Directory where the sidecar stores logs for collectors and the sidecar itself.
#log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"

# The maximum size of the log file before it gets rotated.
#log_rotate_max_file_size: "10MiB"

# The maximum number of old log files to retain.
#log_rotate_keep_files: 10

# Directory where the sidecar generates configurations for collectors.
#collector_configuration_directory: "C:\\Program Files\\Graylog\\sidecar\\generated"

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the access list feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_accesslist:
#       - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#       - "C:\\Program Files\\Filebeat\\filebeat.exe"
#
# Example disable access listing:
#     collector_binaries_accesslist: []
#
# Default:
collector_binaries_accesslist:
#  - "C:\\Program Files\\Graylog\\sidecar\\filebeat.exe"
  - "C:\\Program Files\\Graylog\\sidecar\\winlogbeat.exe"
#  - "C:\\Program Files\\Filebeat\\filebeat.exe"
#  - "C:\\Program Files\\Packetbeat\\packetbeat.exe"
#  - "C:\\Program Files\\Metricbeat\\metricbeat.exe"
#  - "C:\\Program Files\\Heartbeat\\heartbeat.exe"
#  - "C:\\Program Files\\Auditbeat\\auditbeat.exe"
#  - "C:\\Program Files (x86)\\nxlog\\nxlog.exe"

The sidecar is not showing up on the Graylog web console so something is stopping it connecting but I can’t see what.
If anyone could recommend some next steps it would really help!

Description of steps you’ve taken to attempt to solve the issue

  • Tried switching server url to be both private and public server ip.
  • Tried generating new server api token incase previous had a problem with it.
  • Tried inputting node-id myself instead of using given default.
  • Added firewall rule to SQL Server to allow outbound ports 5044 and 9000 and checked Graylog server allows inbound same ports.
  • Can ping/telnet between servers so uncertain whether it could be networking issue
  • Tried jumping ahead of this step by using ‘graylog_sidecar_installer_1.1.0-1.exe /S -SERVERURL=http://[private ip]:9000/api -APITOKEN=mso53nr3vi1id42ege46pmgggff5u6qjqcr24160gemn7e0bkm0’ and then starting the graylog-sidecar service, but it just fills sidecar logs with the same above errors.

Operating system information

  • Linux (redhat 7.9)

Package versions

  • Graylog v4.1.6
  • Elasticsearch v6.8.18

I believe if you run this at an elevated command prompt (without the -service install) it will attempt to run interactively… and may give you more information. also - what do you see in the logs at:

C:\Program Files\Graylog\sidecar\logs

I ran the previous commands in an elevated command prompt. Running it without -service install yields the same results as running with:

C:\Program Files\Graylog\sidecar>"C:\Program Files\graylog\sidecar\graylog-sidecar.exe"
time="2021-11-16T16:41:03Z" level=info msg="Using node-id: 86a2c07e-9b74-426f-af30-8dbedd7e9bc3"
time="2021-11-16T16:41:03Z" level=info msg="No node name was configured, falling back to hostname"
time="2021-11-16T16:41:03Z" level=info msg="Starting signal distributor"
time="2021-11-16T16:41:34Z" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"http://[public ip]:9000/api/sidecars/86a2c07e-9b74-426f-af30-8dbedd7e9bc3\": dial tcp 40.69.45.130:9000: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."

Currently the sidecar.yml has the public IP address added to server url so it yielded the connection attempt failed response.

The sidecar logs are just a repeat of the console output (either the EOF error or failed to connect, depending on if the yml has the public/private ip). There are no further details in the messages.

I was looking up the in other community posts where there was a proxy involved… also there was a post earlier that I was replying to where I questioned having the sidecar.yml config having node_name: "" you may want to comment that out, sidecar will then fallback to hostname. Not sure if that worked because the other person did not mark an answer.

Just tried commenting out node_name: "" but I’m still getting the same errors I’m afraid.
I know the server with sidecar and the graylog server can see one another as they’re able to ping between one another, but I wonder if perhaps there’s another step I’ve missed to ensure that sidecar can communicate to the server so it becomes available on the web interface?

Below is my working windows server sidecar.yml to compare with. One thing to note… yml is highly sensitive to spacing and indentation. There is no proxy or any network FW etc that might be blocking port/protocol/app-id?

server_url: http://BHS-02:9000/api/
server_api_token: "<secret>" 
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files:
collector_id: file:C:\Program Files\Graylog\sidecar\collector-id
cache_path: C:\Program Files\Graylog\sidecar\cache
log_path: C:\Program Files\Graylog\sidecar\logs
log_rotation_time: 86400
log_max_age: 604800
tags: [windows]
collector_binaries_whitelist: []
backends:
    - name: nxlog
      enabled: false
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\nxlog.conf
    - name: winlogbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\winlogbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\winlogbeat.yml
    - name: filebeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\filebeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\filebeat.yml
    - name: auditbeat
      enabled: true
      binary_path: C:\Program Files\Graylog\sidecar\auditbeat.exe
      configuration_path: C:\Program Files\Graylog\sidecar\generated\auditbeat.yml
1 Like

What version of sidecar do you use, as I was wondering what the differences would be where your yml differs from mine like collector_binaries_whitelist / collector_binaries_accesslist and yours has a “binaries” area?

I’ve checked our Firewall and seeing no traffic relating to this, and I can ping between the boxes so I’m uncertain what to try next.

You can change from whitelist to accesslist, they both work for now… I do need to update mine. It does feel like you have a firewall issue, perhaps you haven’t opened port 9000 for traffic? That would be a different rule setting than ping. You should see something at the firewall, either passing through or getting denied… Hard to envision from afar… :slight_smile:

After reading this post I’m leaning towards what @tmacgbay is suggesting about firewall.
I’m not sure if you tried to test the IP Address /w Port number.
Below is how you can do this. Test the connection from Windows Device to Graylog server using port 9000. Run this command

 PS C:\Program Files\Graylog\sidecar> Test-NetConnection -ComputerName 8.8.8.8  -Port 9000

You should see something like this on the output.

ComputerName     : 8.8.8.8
RemoteAddress    : 8.8.8.8
RemotePort       : 9000
InterfaceAlias   : Ethernet
SourceAddress    : 4.2.2.1
TcpTestSucceeded : True

Just an FYI ping command uses ICMP protocol and it really doesn’t tell if the port is open. Just tells you there is a device with that IP address and is reachable.
I normally use PING command with -T flag to check my DNS server /w FQDN and/or Packet loss during a connection.

I’m kind of confused between your command you executed and your error shown.
What I don’t understand is if your trying to install the service and you receive the error stated above

The command you executed.

The error you stated.

Installing a service has nothing to do with a network connection UNLESS you START that service. Then I could see that ERROR happening.
For example if you executed this command below using PowerShell (and may I say these are happy years since I get to use BASH in PowerShell) with administrator privileges.

.\graylog-sidecar.exe" -service start

Then the configurations you made in GL sidecar EXE file is trying to execute the GL URL (i.e. “http://[private ip]:9000/api/”)

Best way to find a service installed on that windows device is check your TASK Manager. If not then try something like this below.
I went to the directory where GL sidecar EXE file is located. You should be able to install the service. Mine is already install, hence the error.

PS C:\Program Files\Graylog\sidecar> ls

Directory: C:\Program Files\Graylog\sidecar

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       11/22/2021   8:27 PM                cache
d-----        8/23/2021  10:57 PM                generated
d-----       11/22/2021   8:27 PM                logs
-a----        2/15/2021   8:25 AM       54201312 filebeat.exe
-a----        3/11/2021   9:18 AM        7956480 graylog-sidecar.exe
-a----         3/4/2021   8:23 AM          73254 graylog.ico
-a----        3/11/2021   9:18 AM          30425 LICENSE
-a----        8/23/2021   6:52 PM             36 node-id
-a----       11/15/2021   8:28 PM           3413 sidecar.yml
-a----         3/4/2021   8:23 AM           3392 sidecar.yml.dist
-a----        8/23/2021   6:40 PM         176173 uninstall.exe
-a----        2/15/2021   8:27 AM       50942704 winlogbeat.exe
PS C:\Program Files\Graylog\sidecar> pwd
Path
----
C:\Program Files\Graylog\sidecar
PS C:\Program Files\Graylog\sidecar> .\graylog-sidecar.exe -service install
time="2021-11-26T20:13:06-06:00" level=fatal msg="Failed service action: Failed to install Graylog Sidecar: service graylog-sidecar already exists"
PS C:\Program Files\Graylog\sidecar>

If no problems occur then execute this. Remind you I’m within the sidecar directory.

.\graylog-sidecar.exe -service start

Have you tried to use the default Graylog Sidecar configuration first to see if everything works?
Once you have the service up and running and working correctly then reconfigure it.
That way you know if it doesn’t work it will be because of any new configuration you have made. Just an Idea.
For example:

server_url: "https://8.8.8.8:9000/api"
server_api_token: "647na8fg66oathdp4sa0869uv85gj57wee7we7wewe9"
node_id: "file:C:\\Program Files\\Graylog\\sidecar\\node-id"
node_name: "kool"
update_interval: 10
tls_skip_verify: true
send_status: true
log_path: "C:\\Program Files\\Graylog\\sidecar\\logs"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10

Not sure about this configuration with two back slashes but I could be wrong.

Mine looks like this.

list_log_files:
- C:\Program Files\Microsoft SQL Server\MSSQL13.VEEAMSQL2016\MSSQL\Log

Results

Hope that helps.

1 Like