HTTPS what port is it listening on?

[erasedhammer]

Just created a self-signed x.509 certificate and encrypted pkcs 8 key for my single node server. I added the files to the server.conf, and uncommented the enable https, restarted the server and nothing. The web interface is no longer listening anywhere. There is a socket listening on 127.0.0.1:9200 and 9300.

Is there no https_bind_address option in the config? How do I tell the server to start listening on a dedicated https port? How come the http listen option has just disappeared? Is graylog unable to listen for both http and https?

Please help.

[dickinsonzach]

Good afternoon, by default the Web UI will be on 9000 Whether that’s HTTP or HTTPS

If you’re having trouble getting HTTPS working on the Web UI, please post how you created your certs and config file settings.

Also take a look through past posts, I needed a lot of help getting it working

Thank you, Zach.

[erasedhammer]

###############
# HTTP settings
###############

#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
http_bind_address = 192.168.3.2:8080
#http_bind_address = [2001:db8::1]:9000


################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /etc/graylog/server/graylog-x509.crt

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /etc/graylog/server/graylog-pkcs8.key

# The password to unlock the private key used for securing the HTTP interface.
#http_tls_key_password = secret


$ ls -l /etc/graylog/server/graylog-x509.crt
-rw-r--r-- 1 root root 2057 Mar 22 13:52 /etc/graylog/server/graylog-x509.crt
$ ls -l /etc/graylog/server/graylog-pkcs8.key  
-rw-r--r-- 1 root root 3434 Mar 22 13:54 /etc/graylog/server/graylog-pkcs8.key

Both files are pem format, with the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

As for the creation of the certificates, I followed the tutorial below:

Create self-signed TLS certificates - PKCS8 key and x509 cert (works for...

[aaronsachs]

Have you looked at our existing documentation about how to use TLS with Graylog? That’s honestly the best place to start, if you’ve not done that already. Also the docs contain pretty much all you’ll need in terms of knowing what ports are listening.

[erasedhammer]

I have read it and there is no mention about network ports after configuration of HTTPS.

I read through it and saw a bunch of stuff that didn’t seem relevant to a single node all in one server.
I took note of the certificates the server needs and generated them on my own.

Can I not just point the server.conf at a cert and key and have it just work?

[aaronsachs]

For ports, Securing Graylog — Graylog 4.0.0 documentation covers what you’ll need. As for what’s needed in the config for TLS to work all the way around, you’ll be using:

# grep "^[^#;]" server.conf | grep -E "^http"
http_bind_address = 192.168.1.3:9000
http_enable_tls=true
http_tls_cert_file=/etc/graylog/ssl/fullchain.pem
http_tls_key_file=/etc/graylog/ssl/privkey.pem

Do note that the certs I’m using here are from Letsencrypt and IIRC, the CA files are already part part of Ubuntu. The challenge with TLS in Graylog is when you start to use self-signed certs that the OS doesn’t know about or that are not already trusted on the system.The instructions in the doc I linked cover that from a Java perspective, though adding them to your system’s CA store is a separate matter.

[erasedhammer]

My problem is graylog web interface is not listening at all.
When I go uncomment the tls lines in server.conf, and restart the service, the web gui is no longer found in netstat.

Looking at the server logs now, it appears the server.conf is not the only place required to configure HTTPS unfortunately. I am not sure why because the HTTPs documentation page says for rest API the cert must be in the JVM trust store. I am not using rest API with multiple nodes, so I assumed this does not apply to me.

Despite the log below, systemd still says graylog is running and it still has loopback connections active in netstat. So I guess partial start failure??

2021-03-22T18:17:37.567-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], PeriodicalsService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], StreamCacheService [TERMINATED], EtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-22T18:17:37.568-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]

[rmentor]

Could be a silly question but what’s the URL that you expect users to hit with your HTTPS in place? You might need to configure that in your server.conf if it’s different from your bind address.

[gsmith]

@erasedhammer
After read this over a couple times, I belive your certificates are incorrect.
Look again at the links @aaronsachs suggested.

@aaronsachs

Have you looked at our existing documentation about how to use TLS with Graylog? That’s honestly the best place to start, if you’ve not done that already. Also the docs contain pretty much all you’ll need in terms of knowing what ports are listening.

Question, does your Graylog server start up without HTTPS ( TLS) configured?

Maybe something in here might shed some light.

@rmentor
Sorry about tagging you earlier, my apologies.

[rmentor]

@gsmith
No worries. Honest mistake.

[erasedhammer]

url is just http://192.168.3.2:9000/
No dns, it is the same as in my server.conf

[erasedhammer]

So as far as I can tell, when I uncomment the tls lines in server.conf and restart the graylog server, it takes a minute but systemd says the service is running. I check netstat and ps and java is in fact running.

Does the web interface start separately? How can I check the status of the web interface itself.

[gsmith]

@erasedhammer

url is just http://192.168.3.2:9000/
No dns, it is the same as in my server.conf

Ok so if you

@erasedhammer

uncomment the tls lines in server.conf

Then your URL should be
https://192.168.3.2:9000/

NOT
http://192.168.3.2:9000/

your server.conf should look something like this:

http_bind_address = IP_ADDRESS:9000
http_publish_uri = https://IP_ADDRESS:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

And again double check your certs if you havent already. Those links @aaronsachs showed you will help.

[erasedhammer]

After I enable the TLS Lines in server.conf, there is no socket on the server listening on port 9000.
when I said my url was http://192.168.3.2:9000/ I meant that was the one I am using for HTTP right now.

I will double check my server.conf.

The certs I generated I made sure where in pem format, the cert in x.509 and the key as pkcs#8 just as the graylog documentation says.

[aaronsachs]

Hmmmm…and mongo is confirmed as running? IIRC the API won’t work if Mongo’s down as all of the config and stuff needed for the API is stored there.

[erasedhammer]

Without any tls settings, mongo is running.

When I uncomment all the tls lines in server.conf and restart, mongo still is running.
Its just the web interface isn’t listening.

I will create new certificates and see if that helps.

[erasedhammer]

Just redid my certificates with the following commands:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout selfsigned-nonpkcs8-key.pem -out graylog-x509.crt 
sudo openssl pkcs8 -topk8 -inform PEM -outform PEM -in selfsigned-nonpkcs8-key.pem -out graylog-pkcs8.pem

Uncomment the TLS lines in server.conf, and restart.
Web interface does not come up in netstat:
$ sudo netstat -taplnu | grep \:9000

Netstat for java:

$ sudo netstat -taplnu | egrep “java|mongod”`
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 533/mongod
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 519/java
tcp 0 0 127.0.0.1:9300 0.0.0.0:* LISTEN 519/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47654 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52118 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52130 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47656 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52060 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52116 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47660 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52062 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:27017 127.0.0.1:52116 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52062 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47652 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47656 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:52118 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:52060 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52130 ESTABLISHED 533/mongod
tcp 0 0 127.0.0.1:52110 127.0.0.1:27017 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:9200 127.0.0.1:47652 ESTABLISHED 519/java
tcp 0 0 127.0.0.1:47654 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:47660 127.0.0.1:9200 ESTABLISHED 1371/java
tcp 0 0 127.0.0.1:27017 127.0.0.1:52110 ESTABLISHED 533/mongod

/var/log/graylog-server/server.log

2021-03-24T17:01:17.361-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], PeriodicalsService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], EtagService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-24T17:01:17.362-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]

I don’t know what this log is telling me, but obviously whatever JerseyService is, is probably my problem?

[gsmith]

@erasedhammer
Have you checked you permissions on your keystore (i.e. changeit, or secret etc…)
Can Graylog access the Java keystore?
Did you create your own keystore or are you using java default (i.e. cacerts)
I would follow Graylog documention first when creating your certs maybe you might find where you problem is.

Maybe give this a try.

Graylog3 with https (easy tutorial)

My ambient test: -Oracle Linux 7.6 (VMware 15); -Network: NAT; -Graylog3, MongoDB and Elastic: running together; Documentation base for RPM distro. First of all, read Docs » Configuring Graylog » Using HTTPS 0- mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/ 1- openssl req -x509 -days 1095 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem 2- openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem 3- openssl pkcs8 -in pk…

[erasedhammer]

I have ran through almost all of that post, but I’m hung up on the last thing,
Add “-Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks” em /etc/sysconfig/graylog-server

I am on a debian 10 system, so I have no clue what this command is or what it is supposed to do.

I found the JVM settings file on debian, /etc/default/graylog-server
I’m not sure where to put the trust store location argument through?
Do I just put it at the bottom of the file? Do I have to have -XX: before it? Does it go in the quotes for GRAYLOG_SERVER_ARGS or GRAYLOG_SERVER_JAVA_OPTS?

[aaronsachs]

That looks like it’s referring to the env var file on a Centos system. Specifically, the server opts in that file. In debian, this will be /etc/default/graylog-server and you’d add it to the GRAYLOG_SERVER_JAVA_OPTS line like so:

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks"