Just created a self-signed x.509 certificate and encrypted pkcs 8 key for my single node server. I added the files to the server.conf, and uncommented the enable https, restarted the server and nothing. The web interface is no longer listening anywhere. There is a socket listening on 127.0.0.1:9200 and 9300.
Is there no https_bind_address option in the config? How do I tell the server to start listening on a dedicated https port? How come the http listen option has just disappeared? Is graylog unable to listen for both http and https?
###############
# HTTP settings
###############
#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
http_bind_address = 192.168.3.2:8080
#http_bind_address = [2001:db8::1]:9000
################
# HTTPS settings
################
#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true
# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /etc/graylog/server/graylog-x509.crt
# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /etc/graylog/server/graylog-pkcs8.key
# The password to unlock the private key used for securing the HTTP interface.
#http_tls_key_password = secret
$ ls -l /etc/graylog/server/graylog-x509.crt
-rw-r--r-- 1 root root 2057 Mar 22 13:52 /etc/graylog/server/graylog-x509.crt
$ ls -l /etc/graylog/server/graylog-pkcs8.key
-rw-r--r-- 1 root root 3434 Mar 22 13:54 /etc/graylog/server/graylog-pkcs8.key
Both files are pem format, with the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----
As for the creation of the certificates, I followed the tutorial below:
Have you looked at our existing documentation about how to use TLS with Graylog? That’s honestly the best place to start, if you’ve not done that already. Also the docs contain pretty much all you’ll need in terms of knowing what ports are listening.
I have read it and there is no mention about network ports after configuration of HTTPS.
I read through it and saw a bunch of stuff that didn’t seem relevant to a single node all in one server.
I took note of the certificates the server needs and generated them on my own.
Can I not just point the server.conf at a cert and key and have it just work?
Do note that the certs I’m using here are from Letsencrypt and IIRC, the CA files are already part part of Ubuntu. The challenge with TLS in Graylog is when you start to use self-signed certs that the OS doesn’t know about or that are not already trusted on the system.The instructions in the doc I linked cover that from a Java perspective, though adding them to your system’s CA store is a separate matter.
My problem is graylog web interface is not listening at all.
When I go uncomment the tls lines in server.conf, and restart the service, the web gui is no longer found in netstat.
Looking at the server logs now, it appears the server.conf is not the only place required to configure HTTPS unfortunately. I am not sure why because the HTTPs documentation page says for rest API the cert must be in the JVM trust store. I am not using rest API with multiple nodes, so I assumed this does not apply to me.
Despite the log below, systemd still says graylog is running and it still has loopback connections active in netstat. So I guess partial start failure??
2021-03-22T18:17:37.567-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], PeriodicalsService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], StreamCacheService [TERMINATED], EtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-22T18:17:37.568-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
Could be a silly question but what’s the URL that you expect users to hit with your HTTPS in place? You might need to configure that in your server.conf if it’s different from your bind address.
So as far as I can tell, when I uncomment the tls lines in server.conf and restart the graylog server, it takes a minute but systemd says the service is running. I check netstat and ps and java is in fact running.
Does the web interface start separately? How can I check the status of the web interface itself.
After I enable the TLS Lines in server.conf, there is no socket on the server listening on port 9000.
when I said my url was http://192.168.3.2:9000/ I meant that was the one I am using for HTTP right now.
I will double check my server.conf.
The certs I generated I made sure where in pem format, the cert in x.509 and the key as pkcs#8 just as the graylog documentation says.
Hmmmm…and mongo is confirmed as running? IIRC the API won’t work if Mongo’s down as all of the config and stuff needed for the API is stored there.
Uncomment the TLS lines in server.conf, and restart.
Web interface does not come up in netstat: $ sudo netstat -taplnu | grep \:9000
Netstat for java:
$ sudo netstat -taplnu | egrep “java|mongod”`
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 533/mongod
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 519/java
tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN 519/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47654 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52118 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52130 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47656 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52060 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52116 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47660 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52062 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:27017 127.0.0.1:52116 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52062 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47652 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47656 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:52118 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:52060 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52130 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52110 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47652 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:47654 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47660 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52110 ESTABLISHED 533/mongod
/var/log/graylog-server/server.log
2021-03-24T17:01:17.361-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], PeriodicalsService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], EtagService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-24T17:01:17.362-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:834) ~[?:?]
I don’t know what this log is telling me, but obviously whatever JerseyService is, is probably my problem?
@erasedhammer
Have you checked you permissions on your keystore (i.e. changeit, or secret etc…)
Can Graylog access the Java keystore?
Did you create your own keystore or are you using java default (i.e. cacerts)
I would follow Graylog documention first when creating your certs maybe you might find where you problem is.
I have ran through almost all of that post, but I’m hung up on the last thing, Add “-Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks” em /etc/sysconfig/graylog-server
I am on a debian 10 system, so I have no clue what this command is or what it is supposed to do.
I found the JVM settings file on debian, /etc/default/graylog-server
I’m not sure where to put the trust store location argument through?
Do I just put it at the bottom of the file? Do I have to have -XX: before it? Does it go in the quotes for GRAYLOG_SERVER_ARGS or GRAYLOG_SERVER_JAVA_OPTS?
That looks like it’s referring to the env var file on a Centos system. Specifically, the server opts in that file. In debian, this will be /etc/default/graylog-server and you’d add it to the GRAYLOG_SERVER_JAVA_OPTS line like so:
