Graylog Community

Sidecar Collector Connection to Graylog Cluster

Graylog Central (peer support)

drstaind May 19, 2017, 6:39am 1

Hi All,

Experiencing some difficulty in gaining a successful connection from the Sidecar Collector to a Graylog Cluster.
The cluster itself is working fine. No firewalld or iptables have been enabled.

Brief overview of the setup

10.40.1.230 - ClusterControl server + Graylog web UI + HAProxy
10.40.1.231 - Graylog server + MongoDB Replica Set + ElasticSearch
10.40.1.232 - Graylog server + MongoDB Replica Set + ElasticSearch
10.40.1.233 - Graylog server + MongoDB Replica Set + ElasticSearch

Graylog Version = 2.2.3
Sidecar Collector Version = 0.1.1-1
Graylog OS = centos-release-7-3.1611.el7.centos.x86_64
Sidecar Collector OS = WinServer2k16r2

Configuration of each graylog server (/etc/graylog/server/server.conf)

rest_listen_uri = http://0.0.0.0:12900/
#rest_transport_uri = http://0.0.0.0:12900/

Configuration of the graylog web server (/etc/graylog/web/web.conf)

graylog2-server.uris="http://10.40.1.231:12900/,http://10.40.1.232:12900/,http://10.40.1.233:12900/"

Web URL = http://10.40.1.230:9000/

I’ve previously had single instance Graylog nodes (non-cluster) working with the sidecar collector without issue, however my inability to connect to the api to establish a Collector has me scratching my head a little. After doing some research, most people were able to resolve by modifying their respective configurations between :9000 and :12900, with appending /api and without. I thought i’d do the same thing.

Attempting to send to the graylog server instance:

Attempt 1:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:12900/

Result:

time="2017-05-19T16:15:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:12+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:15:12+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:15:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:22+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:22+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:15:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found"

Attempt 2:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:12900/api

Result:

time="2017-05-19T16:17:02+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:17:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:17:12+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:17:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found"

Attempt 3:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:9000/

Result:

time="2017-05-19T16:19:04+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:04+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:15+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:16+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it."

Attempt 4:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:9000/api

Result:

time="2017-05-19T16:20:51+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:20:52+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:21:02+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:21:03+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it."

Attempting to connect to the webserver instance: (already considered this unlikely)

Attempt 1:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:12900/

Result:

time="2017-05-19T16:06:30+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:31+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:41+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:43+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:52+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:55+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:07:03+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:07:07+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it."

Attempt 2:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:12900/api

Result:

time="2017-05-19T16:08:25+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:26+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:36+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:39+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:47+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it."

Attempt 3:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:9000/

Result:

time="2017-05-19T16:10:23+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:23+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:10:23+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:10:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:33+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:33+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:10:34+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found"

Attempt 4:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:9000/api

Result:

time="2017-05-19T16:12:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:12+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:12:12+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:12:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:22+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:22+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:12:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found"

If anyone that has been able to successfully deploy a sidecar-collector to a graylog cluster would be kind enough to point out where i’m going wrong, i’d be very grateful.

jochen (Jochen) May 19, 2017, 9:19am 2

The old standalone Graylog web interface is not compatible with Graylog 2.0.0 or higher.

In order to solve your issues:

Make sure that you’re running Graylog 2.2.3.
Make sure that the Graylog Collector plugin is installed (check on the System / Nodes / Details pages in the Graylog web interface).
Make sure that the Graylog Collector Sidecars can communicate with the Graylog REST API (either directly or proxied by HAProxy).

drstaind May 22, 2017, 2:31am 3

Downloaded graylog-plugin-collector-2.3.0-alpha.2.jar to /usr/share/graylog-server/plugin.

Result after graylog-server restart:

May 22 12:27:30 graylog1 systemd: Starting Graylog server...
May 22 12:27:30 graylog1 graylog-server: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
May 22 12:27:30 graylog1 graylog-server: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
May 22 12:27:31 graylog1 graylog-server: Exception in thread "main" java.lang.NoSuchMethodError: org.graylog2.plugin.Version.from(IIILjava/lang/String;)Lorg/graylog2/plugin/Version;
May 22 12:27:31 graylog1 graylog-server: at org.graylog.plugins.collector.CollectorMetaData.getVersion(CollectorMetaData.java:55)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader$PluginComparator.compare(PluginLoader.java:112)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader$PluginComparator.compare(PluginLoader.java:103)
May 22 12:27:31 graylog1 graylog-server: at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
May 22 12:27:31 graylog1 graylog-server: at java.util.TimSort.sort(TimSort.java:220)
May 22 12:27:31 graylog1 graylog-server: at java.util.Arrays.sort(Arrays.java:1512)
May 22 12:27:31 graylog1 graylog-server: at com.google.common.collect.ImmutableSortedSet.construct(ImmutableSortedSet.java:428)
May 22 12:27:31 graylog1 graylog-server: at com.google.common.collect.ImmutableSortedSet$Builder.build(ImmutableSortedSet.java:562)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader.loadPlugins(PluginLoader.java:56)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.loadPlugins(CmdLineTool.java:264)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.installPluginConfigAndBindings(CmdLineTool.java:229)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:151)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.Main.main(Main.java:44)
May 22 12:27:31 graylog1 systemd: graylog-server.service: main process exited, code=exited, status=1/FAILURE
May 22 12:27:31 graylog1 systemd: Unit graylog-server.service entered failed state.
May 22 12:27:31 graylog1 systemd: graylog-server.service failed.

jan (Jan Doberstein) May 22, 2017, 6:02am 4

you have now the alpha version installed - that might have some issues with newer plugins. If you want to run a stable version of Graylog you should use Graylog 2.2.3.

Complete remove the Web Interface as this is part of the Graylog server. Take care of what @jochen had written.

drstaind May 23, 2017, 3:36am 5

The issue was indeed related to the old version of graylog_web installed.

Now running three current instances with the correct plugins for the collector. Working well.

Many thanks for your help.

system (system) Closed June 6, 2017, 3:36am 6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views	Activity
Sidecar Cannot Connect To Server Graylog Central (peer support)	12	1571	March 15, 2022
Graylog 3.0 Sidecar Graylog Central (peer support) sidecar	4	1448	February 28, 2019
Graylog Sidecar Failing with Winlogbeat Graylog Central (peer support) sidecar , winlogbeat	7	1706	October 17, 2018
Connecting Graylog Sidecar installed on SQL Server to Graylog Graylog Central (peer support)	9	1286	December 11, 2021
Graylog 3.2 sidecar collector/filebeat errors Graylog Central (peer support) sidecar	5	697	June 3, 2020