Sidecar Collector Connection to Graylog Cluster


#1

Hi All,

Experiencing some difficulty in gaining a successful connection from the Sidecar Collector to a Graylog Cluster.
The cluster itself is working fine. No firewalld or iptables have been enabled.

Brief overview of the setup

10.40.1.230 - ClusterControl server + Graylog web UI + HAProxy
10.40.1.231 - Graylog server + MongoDB Replica Set + ElasticSearch
10.40.1.232 - Graylog server + MongoDB Replica Set + ElasticSearch
10.40.1.233 - Graylog server + MongoDB Replica Set + ElasticSearch

Graylog Version = 2.2.3
Sidecar Collector Version = 0.1.1-1
Graylog OS = centos-release-7-3.1611.el7.centos.x86_64
Sidecar Collector OS = WinServer2k16r2

Configuration of each graylog server (/etc/graylog/server/server.conf)

rest_listen_uri = http://0.0.0.0:12900/
#rest_transport_uri = http://0.0.0.0:12900/

Configuration of the graylog web server (/etc/graylog/web/web.conf)

graylog2-server.uris="http://10.40.1.231:12900/,http://10.40.1.232:12900/,http://10.40.1.233:12900/"

Web URL = http://10.40.1.230:9000/

I’ve previously had single instance Graylog nodes (non-cluster) working with the sidecar collector without issue, however my inability to connect to the api to establish a Collector has me scratching my head a little. After doing some research, most people were able to resolve by modifying their respective configurations between :9000 and :12900, with appending /api and without. I thought i’d do the same thing.

Attempting to send to the graylog server instance:

Attempt 1:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:12900/

Result:

time="2017-05-19T16:15:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:12+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:15:12+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:15:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:22+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:15:22+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:15:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 

Attempt 2:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:12900/api

Result:

time="2017-05-19T16:17:02+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:17:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:17:12+10:00" level=error msg="Can't fetch configuration from Graylog API: GET http://10.40.1.231:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: 404 HTTP 404 Not Found" 
time="2017-05-19T16:17:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 

Attempt 3:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:9000/

Result:

time="2017-05-19T16:19:04+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:04+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:15+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:19:16+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 

Attempt 4:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.231:9000/api

Result:

time="2017-05-19T16:20:51+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:20:52+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:21:02+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:21:03+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.231:9000/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.231:9000: connectex: No connection could be made because the target machine actively refused it." 

Attempting to connect to the webserver instance: (already considered this unlikely)

Attempt 1:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:12900/

Result:

time="2017-05-19T16:06:30+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:31+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:41+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:43+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:52+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:06:55+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:07:03+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:07:07+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 

Attempt 2:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:12900/api

Result:

time="2017-05-19T16:08:25+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:26+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:36+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:39+10:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/collectors/925b5261-6142-44ca-b96c-a292d0503a04: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 
time="2017-05-19T16:08:47+10:00" level=error msg="[RequestConfiguration] Fetching configuration failed: Get http://10.40.1.230:12900/api/plugins/org.graylog.plugins.collector/925b5261-6142-44ca-b96c-a292d0503a04?tags=%5B%22windows%22%2C%22iis%22%5D: dial tcp 10.40.1.230:12900: connectex: No connection could be made because the target machine actively refused it." 

Attempt 3:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:9000/

Result:

time="2017-05-19T16:10:23+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:23+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:10:23+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:10:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:33+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:10:33+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:10:34+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 

Attempt 4:

Configuration of Sidecar-Collector (C:\Program Files\Graylog\collector-sidecar\collector_sidecar.yml)

server_url: http://10.40.1.230:9000/api

Result:

time="2017-05-19T16:12:12+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:12+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:12:12+10:00" level=error msg="[filebeat] Unable to start collector after 3 tries, giving up!" 
time="2017-05-19T16:12:13+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:22+10:00" level=error msg="[RequestConfiguration] Bad response status from Graylog server: 404 Not Found" 
time="2017-05-19T16:12:22+10:00" level=error msg="Can't fetch configuration from Graylog API: invalid character '<' looking for beginning of value" 
time="2017-05-19T16:12:24+10:00" level=error msg="[UpdateRegistration] Bad response from Graylog server: 404 Not Found" 

If anyone that has been able to successfully deploy a sidecar-collector to a graylog cluster would be kind enough to point out where i’m going wrong, i’d be very grateful.


(Jochen) #2

The old standalone Graylog web interface is not compatible with Graylog 2.0.0 or higher.

In order to solve your issues:

  • Make sure that you’re running Graylog 2.2.3.
  • Make sure that the Graylog Collector plugin is installed (check on the System / Nodes / Details pages in the Graylog web interface).
  • Make sure that the Graylog Collector Sidecars can communicate with the Graylog REST API (either directly or proxied by HAProxy).

#3

Downloaded graylog-plugin-collector-2.3.0-alpha.2.jar to /usr/share/graylog-server/plugin.

Result after graylog-server restart:

May 22 12:27:30 graylog1 systemd: Starting Graylog server...
May 22 12:27:30 graylog1 graylog-server: OpenJDK 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
May 22 12:27:30 graylog1 graylog-server: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
May 22 12:27:31 graylog1 graylog-server: Exception in thread "main" java.lang.NoSuchMethodError: org.graylog2.plugin.Version.from(IIILjava/lang/String;)Lorg/graylog2/plugin/Version;
May 22 12:27:31 graylog1 graylog-server: at org.graylog.plugins.collector.CollectorMetaData.getVersion(CollectorMetaData.java:55)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader$PluginComparator.compare(PluginLoader.java:112)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader$PluginComparator.compare(PluginLoader.java:103)
May 22 12:27:31 graylog1 graylog-server: at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
May 22 12:27:31 graylog1 graylog-server: at java.util.TimSort.sort(TimSort.java:220)
May 22 12:27:31 graylog1 graylog-server: at java.util.Arrays.sort(Arrays.java:1512)
May 22 12:27:31 graylog1 graylog-server: at com.google.common.collect.ImmutableSortedSet.construct(ImmutableSortedSet.java:428)
May 22 12:27:31 graylog1 graylog-server: at com.google.common.collect.ImmutableSortedSet$Builder.build(ImmutableSortedSet.java:562)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.shared.plugins.PluginLoader.loadPlugins(PluginLoader.java:56)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.loadPlugins(CmdLineTool.java:264)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.installPluginConfigAndBindings(CmdLineTool.java:229)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:151)
May 22 12:27:31 graylog1 graylog-server: at org.graylog2.bootstrap.Main.main(Main.java:44)
May 22 12:27:31 graylog1 systemd: graylog-server.service: main process exited, code=exited, status=1/FAILURE
May 22 12:27:31 graylog1 systemd: Unit graylog-server.service entered failed state.
May 22 12:27:31 graylog1 systemd: graylog-server.service failed.

(Jan Doberstein) #4

@drstaind

you have now the alpha version installed - that might have some issues with newer plugins. If you want to run a stable version of Graylog you should use Graylog 2.2.3.

Complete remove the Web Interface as this is part of the Graylog server. Take care of what @jochen had written.


#5

The issue was indeed related to the old version of graylog_web installed.

Now running three current instances with the correct plugins for the collector. Working well.

Many thanks for your help.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.