Connection refused when collector is using port 9000

Graylog Central (peer support)
, , ,
1

Hi,

I’m just getting started with graylog and tried to follow the documentation for setting it up using ubuntu 16.04.2.

My graylog server seems to be running okay, but the sidecar collector on a linux box can’t make the connection when trying to use port 9000. If I change the collector config to use port 5044, it makes the connection (but has other issues related to certs), but I’m sure that’s not the way to go.

I want to use https and followed the documentation to do that, and am only running a single node, so I’m using apache2 as a proxy and reverse proxy. My only goal in using proxy and reverse proxy is to allow https. I’m not doing any load balancing at this point.

When I launch graylog-collector-sidecar on my linux test client I get:

INFO[0000] Using collector-id: longstring
INFO[0000] Fetching configurations tagged by: [linux]
INFO[0000] Starting signal distributor
INFO[0000] [filebeat] Starting (exec driver)
ERRO[0001] [filebeat] Backend finished unexpectedly, trying to restart 1/3.
INFO[0001] [filebeat] Stopping

after 3 attempts it gives up:

ERRO[0010] [filebeat] Unable to start collector after 3 tries, giving up!
time="2017-05-25T16:12:18-06:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put https://fqdn:9000/api/plugins/org.graylog.plugins.collector/collectors/longstring: dial tcp IP_of_fqdn:9000: getsockopt: connection refused"

server.conf:

is_master = true
node_id_file = /etc/graylog/server/node-id
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = https://127.0.0.1:9000/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
web_listen_uri = https://127.0.0.1:9000/
web_endpoint_uri = https://fqdn:9000/api/

I believe the proxy settings are to be specified in a sites-enabled conf (graylog-ssl.conf).

<IfModule mod_ssl.c>
	<VirtualHost *:443>
		ServerAdmin webmaster@localhost
		ServerName fqdn
		ProxyRequests Off
		DocumentRoot /var/www/html

		ErrorLog ${APACHE_LOG_DIR}/error.log
		CustomLog ${APACHE_LOG_DIR}/access.log combined

		SSLEngine on

		SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
		SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

	<Proxy *>
        	Order deny,allow
        	Allow from all
    	</Proxy>

    	<Location />
        	RequestHeader set X-Graylog-Server-URL "https://fqdn/api/"
        	ProxyPass http://127.0.0.1:9000/
        	ProxyPassReverse http://127.0.0.1:9000/
    	</Location>

		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
		</Directory>

	</VirtualHost>
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Thanks for any help you can provide.

M

2

Hej M.

you should check the configuration following the documentation - the server_urlof your sidecar configuration must be the rest_listen_uri of your Graylog server.

The step-by-step guide includes how you should configure the Sidecar.

3

Thanks for your help. I am not sure whether the rest_listen_uri of the graylog server should be the loopback address (and that the apache proxy/reverse proxy makes that addresss reachable by collectors), or the external address of the server, that is reachable directly from the collectors. When the rest_listen_uri is set to 127.0.0.1, the web interface provides a successful login, but the collector gets “connection refused”. When I change the rest_listen_uri to the external address graylog server, I can’t get a weblogin (but the connector does reach it).

The error on the web interface of the graylog server is:

Error message
Bad request
Original Request
GET https://IP-Address/api/system/sessions
Status code
undefined
Full error message
Error: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.

I have tried it with both the IP address, and the fqdn.

When the rest_listen_uri is set to the external address of the graylog server, and this is the address on the collector configuration, I get a different error (but at least it’s talking):

time=“2017-05-26T10:09:02-06:00” level=error msg=“[UpdateRegistration] Failed to report collector status to server: Put https:/FQDN:9000/api/plugins/org.graylog.plugins.collector/collectors/longstring: net/http: TLS handshake timeout”

At this point, I’m still not sure whether the graylog server is not set up correctly, or whether the apache proxy is misconfigured.

It seems like the collector on the client can’t use 127.0.0.1 for the address of the graylog server, b/c that would point it back to itself, and never reach the server.

Thanks again for your help,

Marc

4

I still can’t login to the webserver. However, It’s still not clear to me whether this error indicates that the proxying configurations are incorrect, or whether the rest_listen_uri is incorrect. Can someone provide me with more information? I have read the documentation, but I’m still unclear on this point.

Thanks,
M

5

your sidecar server_url must be your public reachable api - as your rest_listen_uri is localhost that must be the configured from the apache.

RequestHeader set X-Graylog-Server-URL "https://fqdn/api/"

That URL need to be used in your sidecar.

1 Like
6

Thanks for your reply. I’m still getting the “connection refused” error on my graylog_collector_sidecar. I’ve posted the collector_sidecar.yml, graylog-ssl.conf, and server.conf files below in the hope that you can spot an error that I have not been able to find.

Thanks again for your help.

My collector_sidecar.yml has the correct server_url. The whole config is:

server_url: https://FQDN:9000/api/
update_interval: 10
tls_skip_verify: true
send_status: true
list_log_files: /var/log
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
cache_path: /var/cache/graylog/collector-sidecar
log_path: /var/log/graylog/collector-sidecar
log_rotation_time: 86400
log_max_age: 604800
tags:
    - linux
backends:
    - name: nxlog
      enabled: false
      binary_path: /usr/bin/nxlog
      configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf
    - name: filebeat
      enabled: true
      binary_path: /usr/bin/filebeat
      configuration_path: /etc/graylog/collector-sidecar/generated/filebeat.yml

When I launch it it fails with “Backend finished unexpectedly” and then logs the following:

time="2017-05-30T14:30:28-06:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put https://FQDN:9000/api/plugins/org.graylog.plugins.collector/collectors/LongString: dial tcp IP_Address_of_Graylog_server:9000: getsockopt: connection refused"
time="2017-05-30T14:30:31-06:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get https://FQDN:9000/api/plugins/org.graylog.plugins.collector/LongString?tags=%5B%22linux%22%5D: dial tcp IP_Address_of_Graylog_server:9000: getsockopt: connection refused"

My graylog-ssl.conf is the only ssl conf I have on the server, and is:

<IfModule mod_ssl.c>
	<VirtualHost *:443>
		ServerAdmin webmaster@localhost
		ServerName FQDN
		ProxyRequests Off
		DocumentRoot /var/www/html

		ErrorLog ${APACHE_LOG_DIR}/error.log
		CustomLog ${APACHE_LOG_DIR}/access.log combined

		SSLEngine on

		SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
		SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

	<Proxy *>
        	Order deny,allow
        	Allow from all
    	</Proxy>

    	<Location />
        	RequestHeader set X-Graylog-Server-URL "https://FQDN/api/"
        	ProxyPass http://127.0.0.1:9000/
        	ProxyPassReverse http://127.0.0.1:9000/
    	</Location>

		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
		</Directory>

	</VirtualHost>
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

I’ve tried various options in my server.conf, but I can’t seem to get the Connection Refused error to go away. The client that is running the sidecar has access to the graylog server. It can ping and ssh.

Here’s server.conf:

is_master = true
node_id_file = /etc/graylog/server/node-id
plugin_dir = /usr/share/graylog-server/plugin

rest_listen_uri = https://localhost:9000/api/
trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128
web_listen_uri = https://127.0.0.1:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
7

It occurs to me now that I might not have all of the necessary proxy mods enabled in apache2. Which mods should be enabled?

8

the configuration of the sidecars server_url is not configured in your apache proxy …

your sidecar is not able to talk to https://FQDN:9000/api/ because you have configured https://FQDN/api/ in your apache…

9

Thank you. I didn’t understand the mismatch between the collector config and the apache config.

According to this http://docs.graylog.org/en/2.2/pages/configuration/web_interface.html, the apache config should contain https://FQDN/api/ and not https://FQDN:9000/api

According to this http://docs.graylog.org/en/2.2/pages/collector_sidecar.html though, the server_url is https://FQDN:9000/api/, and not https://FQDN/api/

Given that I’m using a proxy, I’m guessing that the collector server url is the one that needs to change. I tried it the other way around, and continued to get “connection refused.”

The linux client running the collector is getting further though. The error I’m getting now is:

Configuration change detected, rewriting configuration file."
time="2017-05-31T11:03:38-06:00" level=info msg="[filebeat] Stopping"
time="2017-05-31T11:03:40-06:00" level=info msg="[filebeat] Starting (exec driver)"
time="2017-05-31T11:03:41-06:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!"

I’ll continue to hunt around for the reason for this error, but would welcome any hints you may have. The linux machine is ubuntu mate, 16.04.2.

Thanks,

Marc

10

In reviewing this thread I see that you replied earlier, and that I had missed that. Sorry!

Changing the collector server_url to work with a proxied graylog server (so no :9000) was the trick.

My collector appears to have been unable to start b/c I hadn’t yet configured an input. Once I did that, the information began to flow.

Again - thanks.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.