Setting up HTTPS connection error - Unreadable or missing HTTP private key

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Getting error while setting up self-signed certificate for a single node graylog.

2. Describe your environment:

  • OS Information: CentOS

  • Package Version:4

  • Service logs, configurations, and environment variables:

2022-05-06T03:33:55.954Z ERROR [CmdLineTool] Invalid configuration
com.github.joschi.jadconfig.ValidationException: Unreadable or missing HTTP private key: /etc/graylog/certs/pkcs8-plain.pem
        at org.graylog2.configuration.HttpConfiguration.validateTlsConfig(HttpConfiguration.java:252) ~[graylog.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_302]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_302]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_302]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_302]
        at com.github.joschi.jadconfig.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:53) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.invokeValidatorMethods(JadConfig.java:221) ~[graylog.jar:?]
        at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:100) ~[graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:448) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:256) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
2022-05-06T03:34:07.170Z INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}

3. What steps have you already taken to try and solve the problem?
I am following below links to create and setup HTTPS.

https://docs.graylog.org/docs/https#creating-a-selfsigned-private-keycertificate
Steps that I followed:-
1)Create a file named openssl-graylog.cnf with the following content (customized to your needs):
2)Create PKCS#5 private key and X.509 certificate:
3)Convert PKCS#5 private key into a unencrypted PKCS#8 private key:

And then added below configuration in server.conf


################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
#http_tls_cert_file = /path/to/graylog.crt
http_tls_cert_file = /etc/graylog/certs/cert.pem

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
#http_tls_key_file = /path/to/graylog.key
http_tls_key_file = /etc/graylog/certs/pkcs8-plain.pem

# The password to unlock the private key used for securing the HTTP interface.
#http_tls_key_password = secret

I didnt touched JVM etc part of the documents., not sure if I have to do it.

openssl-graylog.cnf file 
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = MD
L = German
O = xyz
OU = TCS
CN = graylog.chy.snc.jup.enghns.net

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 10.41.144.233
DNS.1 = graylog.chy.snc.jup.enghns.net

4. How can the community help?

What is the step I am missing?
And why I am getting that error?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

Hello @spandey,

I just went through a cert replacement earlier this week, it can be tricky but the documentation does lay out everything you need. It looks like you’re on the right track.

  1. Did you add the certificate as trusted to cacerts.jks (or another store) per documentation?
  2. Are the folder/file permissions on that key file such that it can be read by the graylog-server service user? For example, on my server (Ubuntu) if I check the service the user it is running as is “graylog” (systemctl show -pUser,UID graylog-server returns ‘graylog’ as the user) so I must make sure that user can read the keyfile.

@ttsandrew thanks for the response.
Till now I have just followed below steps.
Steps that I followed:-
1)Create a file named openssl-graylog.cnf with the following content (customized to your needs):
2)Create PKCS#5 private key and X.509 certificate:
3)Convert PKCS#5 private key into a unencrypted PKCS#8 private key:
4) Added TLS in server.conf and then restarted the server.

I havent looked into steps for keystore etc

@ttsandrew I dont think below steps are applicable to me.

CONVERTING AN EXISTING JAVA KEYSTORE TO PRIVATE KEY/CERTIFICATE PAIR

ADDING A SELF-SIGNED CERTIFICATE TO THE JVM TRUST STORE

Graylog nodes inside a cluster need to communicate with each other using the Graylog REST API. When using HTTPS for the Graylog REST API, the X.509 certificate must be trusted by the JVM trust store (similar to the trusted CA bundle in an operating system), otherwise communication will fail.

As I have just one Graylog node so I don’t think I should add self-signed certificate to the JVM trust store right?
I dont think there is any other step which I have not covered.

If you are using HTTPS or inputs with TLS you must follow the steps to make the certificate trusted by the jvm, which includes adding it to the keystore. It does not matter if it is single node or services are scaled out. The steps also apply whether the certificate is third party issued or self signed.

DId you only create the .cnf? Or did you complete the certificate creation process? That is only a CSR.

edit: nevermind, i see you created the certificate

Got it!
@ttsandrew I am not able to understand where to add this parameters.

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore

I understand that through graylog these parameters can be updated.
But where should I add the ‘JAVA_OPTS’ variable?
Is there any sample example to get an idea about this?

I have added JAVA_OPTS for SSL in /etc/sysconfig/graylog-server

When I am restarting graylog ., I am still getting error.

Do I need to change anything in below configuration?

Does your trust store have a password? The default is changeit, so if you haven’t changed that password then you would pass the option -Djavax.net.ssl.trustStorePassword=changeit

Unless your certificate is issued to a CN 192.168.1.1 or you have a DNS alt name of 192.168.1.1 you will have to modify the http_publish_uri to match the CN of the certificate. Otherwise clients will receive a certificate warning.

FWIW, I always add the CN (in your case FQDN) as a DNS alt name as well, and the hostname as well. So it appears your hostname is graylog. So I would have a DNS alt name of graylog and another of graylog.chy.snc.jup.enghns.net.

Sorry, misread, thought http_publish_uri was uncommented. You need to uncomment that and set it appropriately, most likely the fqdn is the correct setting. https://graylog.chy.snc.jup.enghns.net:9000

@ttsandrew I have modified http_publish_uri.
Also I have added below configuration

But I am still getting below issue while restarting the graylog service.

Did you verify the graylog-server service user can read the file? Does it have read permissions to the directory and the file?

Well I can see it has the read access.
image

I have added the read,write,execute access for user ‘graylog’.
Now graylog service is running but I am still getting below error.

Is there a firewall running somewhere? Is the client rsolving that dens address correctly? This appears to be network troubleshooting.

earlier when tls wasnt enabled the graylog gui was opening up.
iptables is turned off.
Now after enabling TLS ., GUI isnt opening up.

Are there errors in the server log? If the graylog service and the mongod service can’t communicate it can manifest like this.

Hello @spandey

Adding on to what @ttsandrew Suggested.

Something I found in your logs doesn’t seam correct.

If you were following that documentation you should have two certificates called graylog-key.pem & graylog-certificate.pem

Then perhaps something like this in Graylog’s Configuration file as shown below.
NOTE: I put those certificate in Graylog directory because Graylog owns that directory (easy).

Example of what I should see in your Graylog configuration file. I did a mockup from what you posted.

http_bind_address = 10.41.144.233:9000
http_publish_uri = https://graylog.chy.snc.jup.enghns.net:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/graylog-server/graylog-certificate.pem
http_tls_key_file = /etc/graylog-server/graylog/graylog-key.pem
http_tls_key_password = secret OR changeit

If your using JAVA default keystore called “cacerts” like @ttsandrew stated the password for that keystore is changeit . If you are using it then make sure you configuration match’s that password.

Example: You may run into something like this, use the newest version.

[root@graylog graylog_user]# locate cacerts
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/lib/security/cacerts
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.322.b06-1.el7_9.x86_64/jre/lib/security/cacerts
[root@graylog graylog_user]#

I’m not 100% sure but I think this issue is a combination of configuration an the wrong certificate used.
If you make you own KEYSTORE then I would use javax.net.ssl.trustStore /w -Djavax.net.ssl.trustStorePassword

Here are some other post that may have similar issues.

This post below I had to fine tune it for my needs but it did enlighten me on what I needed to do.

Hope that helps

@gsmith @ttsandrew Below is the most recent logs.
As can be seen that graylog is starting well and I dont see any error related to UI no opening up.
Even though I can see error specfic to an inputs for ‘Beats’ but that shouldnt lead to GUI not opening up.

2022-05-09T01:23:55.996Z INFO  [ServerBootstrap] Graylog server 4.2.7+879e651 starting up
2022-05-09T01:23:55.997Z INFO  [ServerBootstrap] JRE: Red Hat, Inc. 1.8.0_302 on Linux 4.18.0-305.3.1.el8.x86_64
2022-05-09T01:23:55.997Z INFO  [ServerBootstrap] Deployment: rpm
2022-05-09T01:23:55.997Z INFO  [ServerBootstrap] OS: CentOS Linux 8 (centos)
2022-05-09T01:23:55.997Z INFO  [ServerBootstrap] Arch: amd64
2022-05-09T01:23:56.035Z INFO  [PeriodicalsService] Starting 29 periodicals ...
2022-05-09T01:23:56.036Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-05-09T01:23:56.058Z INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-05-09T01:23:56.106Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2022-05-09T01:23:56.106Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-05-09T01:23:56.107Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-05-09T01:23:56.130Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2022-05-09T01:23:56.130Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-05-09T01:23:56.142Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-05-09T01:23:56.150Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-05-09T01:23:56.169Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-05-09T01:23:56.183Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-05-09T01:23:56.188Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-05-09T01:23:56.209Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-05-09T01:23:56.210Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-05-09T01:23:56.212Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-05-09T01:23:56.213Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-05-09T01:23:56.213Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-05-09T01:23:56.225Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-05-09T01:23:56.229Z INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:75}] to localhost:27017
2022-05-09T01:23:56.232Z INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-05-09T01:23:56.243Z INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:74}] to localhost:27017
2022-05-09T01:23:56.249Z INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:78}] to localhost:27017
2022-05-09T01:23:56.252Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-05-09T01:23:56.253Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-05-09T01:23:56.253Z INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-05-09T01:23:56.254Z INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-05-09T01:23:56.255Z INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2022-05-09T01:23:56.261Z INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2022-05-09T01:23:56.262Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-05-09T01:23:56.266Z INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-05-09T01:23:56.267Z INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2022-05-09T01:23:56.267Z INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2022-05-09T01:23:56.268Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-05-09T01:23:56.279Z INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:77}] to localhost:27017
2022-05-09T01:23:56.285Z INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:76}] to localhost:27017
2022-05-09T01:23:56.290Z INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:79}] to localhost:27017
2022-05-09T01:23:56.434Z INFO  [JerseyService] Enabling CORS for HTTP endpoint
2022-05-09T01:23:58.549Z INFO  [NetworkListener] Started listener bound to [10.41.144.233:9000]
2022-05-09T01:23:58.551Z INFO  [HttpServer] [HttpServer] Started.
2022-05-09T01:23:58.551Z INFO  [JerseyService] Started REST API at <10.41.144.233:9000>
2022-05-09T01:23:58.551Z INFO  [ServiceManagerListener] Services are healthy
2022-05-09T01:23:58.553Z INFO  [ServerBootstrap] Services started, startup times in ms: {JobSchedulerService [RUNNING]=196, PrometheusExporter [RUNNING]=196, FailureHandlingService [RUNNING]=196, EtagService [RUNNING]=197, OutputSetupService [RUNNING]=197, BufferSynchronizerService [RUNNING]=198, GracefulShutdownService [RUNNING]=199, UrlWhitelistService [RUNNING]=199, InputSetupService [RUNNING]=200, LocalKafkaMessageQueueReader [RUNNING]=200, LocalKafkaMessageQueueWriter [RUNNING]=200, UserSessionTerminationService [RUNNING]=202, LocalKafkaJournal [RUNNING]=206, PeriodicalsService [RUNNING]=244, ConfigurationEtagService [RUNNING]=259, MongoDBProcessingStatusRecorderService [RUNNING]=266, LookupTableService [RUNNING]=318, StreamCacheService [RUNNING]=375, JerseyService [RUNNING]=2524}
2022-05-09T01:23:58.561Z INFO  [ServerBootstrap] Graylog server up and running.
2022-05-09T01:23:58.562Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-05-09T01:23:58.743Z INFO  [InputStateListener] Input [Syslog TCP/6259fa8f8bb8cd6e2a8b11cf] is now STARTING
2022-05-09T01:23:58.768Z INFO  [InputStateListener] Input [Syslog UDP/6259fa5a8bb8cd6e2a8b1192] is now STARTING
2022-05-09T01:23:58.770Z INFO  [InputStateListener] Input [Beats/62702df4d8de1f2d3397b107] is now STARTING
2022-05-09T01:23:58.822Z INFO  [InputStateListener] Input [Beats/625e482dd8de1f2d33844d52] is now STARTING
2022-05-09T01:23:58.905Z WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Beats, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0xcfd9c287, L:/0:0:0:0:0:0:0:0%0:5044]) should be >= 1048576 but is 425984.
2022-05-09T01:23:58.908Z WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=SYSLOG TCP, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=58aedeab-6855-4a75-9826-d00e8dd18ac0} (channel [id: 0x5ce3f4f1, L:/0:0:0:0:0:0:0:0%0:1514]) should be >= 1048576 but is 425984.
2022-05-09T01:23:58.910Z WARN  [AbstractTcpTransport] Failed to start channel for input Beats2Input{title=LocalHost - 127.0.0.1 - 5044 Port, type=org.graylog.plugins.beats.Beats2Input, nodeId=58aedeab-6855-4a75-9826-d00e8dd18ac0}
io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2022-05-09T01:23:58.914Z INFO  [InputStateListener] Input [Beats/625e482dd8de1f2d33844d52] is now RUNNING
2022-05-09T01:23:58.916Z INFO  [InputStateListener] Input [Syslog TCP/6259fa8f8bb8cd6e2a8b11cf] is now RUNNING
2022-05-09T01:23:58.923Z ERROR [InputLauncher] The [org.graylog.plugins.beats.Beats2Input] input with ID <62702df4d8de1f2d3397b107> misfired. Reason: bind(..) failed: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_302]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_302]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_302]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_302]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: org.graylog2.plugin.inputs.MisfireException: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
        at org.graylog2.plugin.inputs.transports.AbstractTcpTransport.launch(AbstractTcpTransport.java:224) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:155) ~[graylog.jar:?]
        ... 7 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2022-05-09T01:23:58.946Z INFO  [InputStateListener] Input [Beats/62702df4d8de1f2d3397b107] is now FAILED
2022-05-09T01:23:58.960Z INFO  [InputStateListener] Input [Syslog UDP/6259fa5a8bb8cd6e2a8b1192] is now RUNNING

I am still getting ‘this site cant be reached error’ on GUI.