Graylog HTTPS issues

jamiebuxxx · January 2, 2018, 9:18pm

Hello,

We are trying to enable HTTPS on our Graylog server using self-signed certs, as mentioned on the link:

When testing on one Graylog node, we are getting the following error:

2018-01-02T21:13:26.900Z ERROR [ServiceManager] Service JerseyService [FAILED] has failed in the STARTING state.
java.lang.IllegalStateException: Couldn't initialize SSL context for HTTP server
	at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:390) ~[graylog.jar:?]
	at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:206) ~[graylog.jar:?]
	at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:139) ~[graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
2018-01-02T21:13:26.904Z ERROR [InputSetupService] Not starting any inputs because lifecycle is: Uninitialized�[LB:DEAD]
2018-01-02T21:13:26.919Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2018-01-02T21:13:26.919Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2018-01-02T21:13:26.920Z INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2018-01-02T21:13:26.921Z INFO  [JournalReader] Stopping.
2018-01-02T21:13:26.924Z WARN  [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2018-01-02T21:13:26.933Z INFO  [node] [graylog-324deda9-a2ae-4c0a-b5ba-ce4adf057b4b] stopping ...
2018-01-02T21:13:26.933Z INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2018-01-02T21:13:26.937Z INFO  [LogManager] Shutting down.
2018-01-02T21:13:26.961Z INFO  [node] [graylog-324deda9-a2ae-4c0a-b5ba-ce4adf057b4b] stopped
2018-01-02T21:13:26.961Z INFO  [node] [graylog-324deda9-a2ae-4c0a-b5ba-ce4adf057b4b] closing ...
2018-01-02T21:13:26.965Z INFO  [node] [graylog-324deda9-a2ae-4c0a-b5ba-ce4adf057b4b] closed
2018-01-02T21:13:26.967Z INFO  [LogManager] Shutdown complete.

Please let us know if we are missing something or if there are any other issues.

Thanks!

jamiebuxxx · January 2, 2018, 10:36pm

Ok, I was able to figure out the issue from the above error message. That is related to a config typo. However, we are trying to test, HTTPS on a single node, we are able to see the following:

2018-01-02T21:41:05.168Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
2018-01-02T21:41:05.169Z INFO  [HttpServer] [HttpServer] Started.
2018-01-02T21:41:05.169Z INFO  [JerseyService] Started REST API at <http://0.0.0.0:9000/api/>
2018-01-02T21:41:05.169Z INFO  [JerseyService] Started Web Interface at <http://0.0.0.0:9000/>
2018-01-02T21:41:05.171Z INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=2, OutputSetupService [RUNNING]=11, BufferSynchronizerService [RUNNING]=14, KafkaJournal [RUNNING]=16, JournalReader [RUNNING]=20, ConfigurationEtagService [RUNNING]=24, PeriodicalsService [RUNNING]=28, StreamCacheService [RUNNING]=902, IndexerSetupService [RUNNING]=4000, JerseyService [RUNNING]=8335}
2018-01-02T21:41:05.173Z INFO  [ServiceManagerListener] Services are healthy
2018-01-02T21:41:05.175Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized�[LB:DEAD] to Running�[LB:ALIVE]
2018-01-02T21:41:05.209Z INFO  [ServerBootstrap] Graylog server up and running.
2018-01-02T21:41:05.255Z WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawTCPInput{title=TCP, type=org.graylog2.inputs.raw.tcp.RawTCPInput, nodeId=null} should be 1048576 but is 212992.
2018-01-02T21:41:05.260Z INFO  [InputStateListener] Input [Raw/Plaintext TCP/5788023e27cb3055b4edc33e] is now STARTING
2018-01-02T21:41:05.330Z INFO  [InputStateListener] Input [Raw/Plaintext TCP/5788023e27cb3055b4edc33e] is now RUNNING
2018-01-02T21:41:09.023Z INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:3624240}] to ops-graylog-mongo-501:27017
2018-01-02T21:41:09.099Z INFO  [connection] Opened connection [connectionId{localValue:12, serverValue:3624241}] to ops-graylog-mongo-501:27017

When we try to access the web server, we get the following error about the API server is not available. We can confirm this with curl:
curl http://graylog_host_ip:9000/api
curl(52) CURLE_GOT_NOTHING

We have set http://graylog_host_ip:9000/api/ for the rest_transport_uri. Once we put everything back to http, its working fine.

Even though we are currently testing HTTPS on one Graylog instance, we have an 8 node Graylog server cluster we would like to implement HTTPS on. We just want to make since it working on the servers first, before we update the Nginx configs.

Speaking of Nginx configs, we have tried to just add SSL to our Nginx config to load balance the cluster. However, we get a 502 Gateway error everytime. Is this normal behavior and why SSL is needed on the Graylog server side?

jamiebuxxx · January 3, 2018, 12:49am

Ok, I was able to rest_transport_uri error resolved by the making sure to add https instead of http for the uri address. Making progress! However, we are setting SSL handshake errors.

2018-01-03T00:43:59.526Z ERROR [LdapNetworkConnection] Message failed : something wrong has occurred
2018-01-03T00:43:59.526Z ERROR [LdapUserAuthenticator] LDAP error
org.apache.directory.ldap.client.api.exception.InvalidConnectionException: SSL handshake failed.

We also set the enabled the Allow self-signed certificates checkbox. We will still continue to play around with this. But the second question, still stands, could we just create a SSL endpoint on our load balancer and have the Graylog proxy through that?

Thanks!

jan · January 3, 2018, 8:54am

hej @jamiebuxxx

for me is not clear what your goal is and what your current situation is. When you are a little more verbose on your actuall configuration someone might help you.

You might want to re-read this part in the documentation: http://docs.graylog.org/en/2.4/pages/configuration/https.html

jamiebuxxx · January 9, 2018, 7:47pm

Hey @jan,

Apologies for the delay and confusing post. I was working through the issue and was just posting my result.

So, to take a step back. is it possible to have to setup our nginx load balance with HTTPS using our company’s certs to proxy the Graylog cluster without HTTPS configured? When we tried this in the past, we would be 502 Gateway error.

I will wait for a reply to this question, before asking another related to issues we are experience setting HTTPS on our Graylog cluster.

Cheers,

jan · January 10, 2018, 9:16am

yes it is possible to have your Proxy for HTTPS termination and run the Cluster with HTTP communication between all nodes.

jamiebuxxx · January 10, 2018, 7:48pm

@jan I was finally about to get HTTPS termination to work by updating nginx config to match what was in the docs. We were not setting proxy_set_header X-Graylog-Server-URL in our nginx config. Since we were able to resolve this, please feel free to close this ticket.

Thanks!

jan · January 11, 2018, 6:56am

just to let you know - this is a community forum, not a ticket system or anything else. This is volunteer work, nothing you get from a company for free.

rvazquez · January 22, 2018, 9:08pm

@jamiebuxxx Where is that log file your looking at? The one in your first post?

jamiebuxxx · January 23, 2018, 9:25pm

@rvazquez Our logs are configured to go to /var/log/graylog-server/server.log.

jan · January 24, 2018, 9:10am

we have in our documentation covered where to find all files - in default situations

http://docs.graylog.org/en/latest/pages/configuration/file_location.html

system · February 7, 2018, 9:11am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cant get https up and running Graylog Central (peer support)	24	5106	March 7, 2019
JerseyService won't start after enabling https Graylog Central (peer support)	3	2681	November 2, 2017
Graylog crashes on startup if HTTPS is enabled Graylog Central (peer support)	20	1147	May 8, 2020
Graylog unable to start after the setup of SSL/TLS Graylog Central (peer support)	25	12898	November 21, 2017
Understand Https Issues Graylog Central (peer support)	2	336	April 18, 2018

Graylog HTTPS issues

Related topics