I am facing a strange problem. Well at least it’s strange to me.
I have a Graylog 2.2.3 server running inside of a Docker Container. With some tinkering I managed to get it work with https. There is a Gelf input with utilizes TLS for a secure connection and it works like a charm. It did not work in the beginning but this was due to the missing cert in the keystore.
But here is the problem. As soon as I activate TLS in the graylog-config file my inputs appear as not running, but they are. I can tell because a) the nxlog logfiles tell me that a secure connection was made for the client to the graylog server and b) because the is new data inside the search section of my graylog server sent over from said clients. The message count in the top right corner also shows 0 messages, which is also strange.
If I try to activate the “not running” input (just for the fun of it) I get an error message.
But again, my logs aren’t showing any related errors.
Am I doing something wrong here? Did I miss someting? I can provide further information or config files if needed.
Any help would be great, thanks in advance.
Cheers, Andy
2017-06-28 11:49:34,141 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <https://0.0.0.0:9000/api/>
2017-06-28 11:49:34,141 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <https://0.0.0.0:9000/>
Port 9300 and 9350 are Elasticsearch Ports.
Port 36403 isn’t mentioned in the log file.
Strange, I just used this command from the mentioned how-to: sudo openssl req -x509 -days 365 -nodes -newkey rsa:2048 -keyout pkcs5-plain.pem -out cert.pem
openssl is version OpenSSL 1.0.2g
It did ask for a CN, but it wasn’t mandatory. Should I try generating new ones and proceed as I did with the current ones?
So I created some new certs and tested with them. If I don’t change my config it is still the same as before.
So then I changed: web_listen_uri = https://0.0.0.0:9000/ to https://graylog.example.com:9000/ rest_transport_uri = https://0.0.0.0:9000/api/ to https://graylog.example.com:9000/api
I ended up with this error: 2017-06-30 14:39:40,450 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration com.github.joschi.jadconfig.ValidationException: Wildcard IP addresses cannot be used if the Graylog REST API and web interface listen on the same port.
So I changed rest_listen_uri = https://0.0.0.0:9000/api/ to https://graylog.example.com:9000/api
After that I got several errors including: 2017-06-30 14:42:30,929 ERROR: org.graylog2.bootstrap.ServerBootstrap - Graylog startup failed. Exiting. Exception was: java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]} at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:742) ~[graylog.jar:?] at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:555) ~[graylog.jar:?] at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:304) ~[graylog.jar:?] at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:147) [graylog.jar:?] at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?] at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
I must say that rest_transport_uri was uncommented before these tests. I changed everything back to 0.0.0.0:9000, now it runs as it did before but with some minor errors: 2017-06-30 14:53:25,614 WARN : org.graylog2.plugin.BaseConfiguration - "https://0.0.0.0:9000/api/" is not a valid setting for "rest_transport_uri". Using default [https://172.21.0.4:9000/api/].
2017-06-30 14:49:57,169 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://172.21.0.4:9000/api/system/metrics/multiple on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698> javax.net.ssl.SSLPeerUnverifiedException: Hostname 172.21.0.4 not verified:
The last one sounds plausible because the cn is graylog.example.com and not the internal container IP 172.21.0.4
# Listen on all network interfaces
rest_listen_uri = https://0.0.0.0:9000/api/
web_listen_uri = https://0.0.0.0:9000/
# Publish the correct URI of the Graylog REST API of this Graylog node
rest_transport_uri = https://graylog.example.com:9000/api/
I changed the config as suggested. It is still the same as before, Inputs appear as if they are not running and the Input counter shows 0 incoming messages, although there are messages incoming. But now I only get two WARN messages in the logs
2017-07-03 08:02:57,438 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog.example.com:9000/api/system/inputstates on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698>
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?`
`2017-07-03 08:02:57,150 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://graylog.example.com:9000/api/system/metrics/multiple on node <2e5b7b01-4bbb-4c31-9d58-609a26bcf698>
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
