HTTPS what port is it listening on?

erasedhammer · March 22, 2021, 6:21pm

Just created a self-signed x.509 certificate and encrypted pkcs 8 key for my single node server. I added the files to the server.conf, and uncommented the enable https, restarted the server and nothing. The web interface is no longer listening anywhere. There is a socket listening on 127.0.0.1:9200 and 9300.

Is there no https_bind_address option in the config? How do I tell the server to start listening on a dedicated https port? How come the http listen option has just disappeared? Is graylog unable to listen for both http and https?

Please help.

dickinsonzach · March 22, 2021, 6:35pm

Good afternoon, by default the Web UI will be on 9000 Whether that’s HTTP or HTTPS

If you’re having trouble getting HTTPS working on the Web UI, please post how you created your certs and config file settings.

Also take a look through past posts, I needed a lot of help getting it working

Thank you, Zach.

erasedhammer · March 22, 2021, 7:04pm

###############
# HTTP settings
###############

#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
http_bind_address = 192.168.3.2:8080
#http_bind_address = [2001:db8::1]:9000


################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /etc/graylog/server/graylog-x509.crt

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /etc/graylog/server/graylog-pkcs8.key

# The password to unlock the private key used for securing the HTTP interface.
#http_tls_key_password = secret


$ ls -l /etc/graylog/server/graylog-x509.crt
-rw-r--r-- 1 root root 2057 Mar 22 13:52 /etc/graylog/server/graylog-x509.crt
$ ls -l /etc/graylog/server/graylog-pkcs8.key  
-rw-r--r-- 1 root root 3434 Mar 22 13:54 /etc/graylog/server/graylog-pkcs8.key

Both files are pem format, with the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----

As for the creation of the certificates, I followed the tutorial below:

aaronsachs · March 22, 2021, 9:14pm

Have you looked at our existing documentation about how to use TLS with Graylog? That’s honestly the best place to start, if you’ve not done that already. Also the docs contain pretty much all you’ll need in terms of knowing what ports are listening.

erasedhammer · March 22, 2021, 9:44pm

I have read it and there is no mention about network ports after configuration of HTTPS.

I read through it and saw a bunch of stuff that didn’t seem relevant to a single node all in one server.
I took note of the certificates the server needs and generated them on my own.

Can I not just point the server.conf at a cert and key and have it just work?

aaronsachs · March 22, 2021, 9:58pm

For ports, Securing Graylog — Graylog 4.0.0 documentation covers what you’ll need. As for what’s needed in the config for TLS to work all the way around, you’ll be using:

# grep "^[^#;]" server.conf | grep -E "^http"
http_bind_address = 192.168.1.3:9000
http_enable_tls=true
http_tls_cert_file=/etc/graylog/ssl/fullchain.pem
http_tls_key_file=/etc/graylog/ssl/privkey.pem

Do note that the certs I’m using here are from Letsencrypt and IIRC, the CA files are already part part of Ubuntu. The challenge with TLS in Graylog is when you start to use self-signed certs that the OS doesn’t know about or that are not already trusted on the system.The instructions in the doc I linked cover that from a Java perspective, though adding them to your system’s CA store is a separate matter.

erasedhammer · March 22, 2021, 10:23pm

My problem is graylog web interface is not listening at all.
When I go uncomment the tls lines in server.conf, and restart the service, the web gui is no longer found in netstat.

Looking at the server logs now, it appears the server.conf is not the only place required to configure HTTPS unfortunately. I am not sure why because the HTTPs documentation page says for rest API the cert must be in the JVM trust store. I am not using rest API with multiple nodes, so I assumed this does not apply to me.

Despite the log below, systemd still says graylog is running and it still has loopback connections active in netstat. So I guess partial start failure??

2021-03-22T18:17:37.567-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], PeriodicalsService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], StreamCacheService [TERMINATED], EtagService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-22T18:17:37.568-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]

rmentor · March 22, 2021, 11:50pm

Could be a silly question but what’s the URL that you expect users to hit with your HTTPS in place? You might need to configure that in your server.conf if it’s different from your bind address.

gsmith · March 23, 2021, 2:26am

@erasedhammer
After read this over a couple times, I belive your certificates are incorrect.
Look again at the links @aaronsachs suggested.

Question, does your Graylog server start up without HTTPS ( TLS) configured?

Maybe something in here might shed some light.

@rmentor
Sorry about tagging you earlier, my apologies.

rmentor · March 23, 2021, 2:11pm

@gsmith
No worries. Honest mistake.

erasedhammer · March 23, 2021, 6:17pm

url is just http://192.168.3.2:9000/
No dns, it is the same as in my server.conf

erasedhammer · March 23, 2021, 6:17pm

So as far as I can tell, when I uncomment the tls lines in server.conf and restart the graylog server, it takes a minute but systemd says the service is running. I check netstat and ps and java is in fact running.

Does the web interface start separately? How can I check the status of the web interface itself.

gsmith · March 24, 2021, 2:13am

Ok so if you

Then your URL should be
https://192.168.3.2:9000/

NOT
http://192.168.3.2:9000/

your server.conf should look something like this:

http_bind_address = IP_ADDRESS:9000
http_publish_uri = https://IP_ADDRESS:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

And again double check your certs if you havent already. Those links @aaronsachs showed you will help.

erasedhammer · March 24, 2021, 4:54pm

After I enable the TLS Lines in server.conf, there is no socket on the server listening on port 9000.
when I said my url was http://192.168.3.2:9000/ I meant that was the one I am using for HTTP right now.

I will double check my server.conf.

The certs I generated I made sure where in pem format, the cert in x.509 and the key as pkcs#8 just as the graylog documentation says.

aaronsachs · March 24, 2021, 4:57pm

Hmmmm…and mongo is confirmed as running? IIRC the API won’t work if Mongo’s down as all of the config and stuff needed for the API is stored there.

erasedhammer · March 24, 2021, 5:07pm

Without any tls settings, mongo is running.

When I uncomment all the tls lines in server.conf and restart, mongo still is running.
Its just the web interface isn’t listening.

I will create new certificates and see if that helps.

erasedhammer · March 24, 2021, 9:10pm

Just redid my certificates with the following commands:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout selfsigned-nonpkcs8-key.pem -out graylog-x509.crt 
sudo openssl pkcs8 -topk8 -inform PEM -outform PEM -in selfsigned-nonpkcs8-key.pem -out graylog-pkcs8.pem

Uncomment the TLS lines in server.conf, and restart.
Web interface does not come up in netstat:
$ sudo netstat -taplnu | grep \:9000

Netstat for java:

$ sudo netstat -taplnu | egrep tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:9200 tcp 0 0 127.0.0.1:9300 tcp 0 0 127.0.0.1:9200 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:52130 tcp 0 0 127.0.0.1:47656 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:52116 tcp 0 0 127.0.0.1:9200 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:52062 tcp 0 0 127.0.0.1:47652 tcp 0 0 127.0.0.1:9200 tcp 0 0 127.0.0.1:52118 tcp 0 0 127.0.0.1:52060 tcp 0 0 127.0.0.1:27017 tcp 0 0 127.0.0.1:52110 tcp 0 0 127.0.0.1:9200 tcp 0 0 127.0.0.1:47654 tcp 0 0 127.0.0.1:47660 tcp 0 0 127.0.0.1:27017 “java|mongod”`
0.0.0.0:* LISTEN 533/mongod
0.0.0.0:* LISTEN 519/java
0.0.0.0:* LISTEN 519/java
127.0.0.1:47654 ESTABLISHED 519/java
127.0.0.1:52118 ESTABLISHED 533/mongod
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:9200 ESTABLISHED 1371/java
127.0.0.1:52060 ESTABLISHED 533/mongod
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:47660 ESTABLISHED 519/java
127.0.0.1:52062 ESTABLISHED 533/mongod
127.0.0.1:52116 ESTABLISHED 533/mongod
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:9200 ESTABLISHED 1371/java
127.0.0.1:47656 ESTABLISHED 519/java
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:52130 ESTABLISHED 533/mongod
127.0.0.1:27017 ESTABLISHED 1371/java
127.0.0.1:47652 ESTABLISHED 519/java
127.0.0.1:9200 ESTABLISHED 1371/java
127.0.0.1:9200 ESTABLISHED 1371/java
127.0.0.1:52110 ESTABLISHED 533/mongod

/var/log/graylog-server/server.log

2021-03-24T17:01:17.361-04:00 ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], UrlWhitelistService [TERMINATED], StreamCacheService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], PeriodicalsService [TERMINATED], GracefulShutdownService [TERMINATED], OutputSetupService [TERMINATED], EtagService [TERMINATED], ConfigurationEtagService [TERMINATED], LookupTableService [TERMINATED], BufferSynchronizerService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}
2021-03-24T17:01:17.362-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:161) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) ~[graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) ~[graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.security.NoSuchAlgorithmException: 1.2.840.113549.1.5.13 SecretKeyFactory not available
                at javax.crypto.SecretKeyFactory.<init>(SecretKeyFactory.java:122) ~[?:?]
                at javax.crypto.SecretKeyFactory.getInstance(SecretKeyFactory.java:168) ~[?:?]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:70) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:834) ~[?:?]

I don’t know what this log is telling me, but obviously whatever JerseyService is, is probably my problem?

gsmith · March 24, 2021, 10:24pm

@erasedhammer
Have you checked you permissions on your keystore (i.e. changeit, or secret etc…)
Can Graylog access the Java keystore?
Did you create your own keystore or are you using java default (i.e. cacerts)
I would follow Graylog documention first when creating your certs maybe you might find where you problem is.

Maybe give this a try.

erasedhammer · March 25, 2021, 7:33pm

I have ran through almost all of that post, but I’m hung up on the last thing,
Add “-Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks” em /etc/sysconfig/graylog-server

I am on a debian 10 system, so I have no clue what this command is or what it is supposed to do.

I found the JVM settings file on debian, /etc/default/graylog-server
I’m not sure where to put the trust store location argument through?
Do I just put it at the bottom of the file? Do I have to have -XX: before it? Does it go in the quotes for GRAYLOG_SERVER_ARGS or GRAYLOG_SERVER_JAVA_OPTS?

aaronsachs · March 25, 2021, 8:00pm

That looks like it’s referring to the env var file on a Centos system. Specifically, the server opts in that file. In debian, this will be /etc/default/graylog-server and you’d add it to the GRAYLOG_SERVER_JAVA_OPTS line like so:

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks"