Issues connecting to Elasticsearch after adding HTTPS to web interface

Graylog Central (peer support)
1

Been having some issues getting HTTPS working for the web interface. Using the following commands from the guide I do the following.

openssl req -x509 -days 730 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out graylog.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:test
cp pkcs8-encrypted.pem /etc/graylog/server/graylog-key.pem
cp graylog.pem /etc/graylog/server/graylog-certificate.pem

Here is the pertitent section of my config.

################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /etc/graylog/server/graylog-certificate.pem

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /etc/graylog/server/graylog-key.pem

# The password to unlock the private key used for securing the HTTP interface.
http_tls_key_password = test

And yet when I start the server I get the following error

2020-02-20T18:44:12.032Z INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.

2020-02-20T18:44:12.033Z INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.

2020-02-20T18:44:12.033Z INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.

2020-02-20T18:44:12.051Z ERROR [Cluster] Couldn’t read cluster health for indices [graylog_, gl-events_, gl-system-events_*] (Could not connect to https://vdagraylog.dontsquatme.com:9200)

2020-02-20T18:44:12.051Z INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.

2020-02-20T18:44:12.060Z INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.

2020-02-20T18:44:12.073Z INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.

2020-02-20T18:44:12.088Z INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].

2020-02-20T18:44:12.090Z INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every
[3600s].

2020-02-20T18:44:12.095Z INFO [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable.

2020-02-20T18:44:12.096Z INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].

2020-02-20T18:44:12.105Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].

2020-02-20T18:44:12.124Z INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].

2020-02-20T18:44:12.133Z INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].

2020-02-20T18:44:12.136Z INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].

2020-02-20T18:44:12.136Z INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].

2020-02-20T18:44:12.244Z INFO [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.

2020-02-20T18:44:12.460Z INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]

2020-02-20T18:44:12.461Z ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]

2020-02-20T18:44:12.475Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].

2020-02-20T18:44:12.480Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.

2020-02-20T18:44:12.480Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.

2020-02-20T18:44:12.481Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.

2020-02-20T18:44:12.482Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].

2020-02-20T18:44:12.482Z INFO [GracefulShutdownService] Running graceful shutdown for <1> shutdown hooks

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.

2020-02-20T18:44:12.483Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].

2020-02-20T18:44:12.488Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp].

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] complete, took <0ms>.

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].

2020-02-20T18:44:12.489Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.

2020-02-20T18:44:12.490Z INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].

2020-02-20T18:44:12.491Z INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.

2020-02-20T18:44:12.491Z INFO [GracefulShutdownService] Initiate shutdown for

2020-02-20T18:44:12.492Z INFO [GracefulShutdownService] Finished shutdown for , took 0 ms

2020-02-20T18:44:12.505Z INFO [LogManager] Shutting down.

2020-02-20T18:44:12.510Z WARN [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.

2020-02-20T18:44:12.511Z INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput

2020-02-20T18:44:12.521Z INFO [LookupDataAdapterRefreshService] Stopping 0 jobs

2020-02-20T18:44:12.540Z INFO [LogManager] Shutdown complete.

2020-02-20T18:44:12.568Z INFO [JournalReader] Stopping.

2020-02-20T18:44:42.502Z ERROR [ServerBootstrap] Unable to shutdown properly on time. {STOPPING=[JobSchedulerService [STOPPING]], TERMINATED=[InputSetupService [TERMINATED], PeriodicalsService [TERMINATED], GracefulShutdownService [TERMINATED], MongoDBProcessingStatusRecorderService [TERMINATED], StreamCacheService [TERMINATED], UrlWhitelistService [TERMINATED], BufferSynchronizerService [TERMINATED], LookupTableService [TERMINATED], OutputSetupService [TERMINATED], ConfigurationEtagService [TERMINATED], EtagService [TERMINATED], KafkaJournal [TERMINATED], JournalReader [TERMINATED]], FAILED=[JerseyService [FAILED]]}

2020-02-20T18:44:42.503Z ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:

java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
        at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:148) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
        Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
        Caused by: java.io.IOException: ObjectIdentifier() -- data isn't an object ID (tag = 48)
                at sun.security.util.ObjectIdentifier.<init>(ObjectIdentifier.java:257) ~[?:1.8.0_242]
                at sun.security.util.DerInputStream.getOID(DerInputStream.java:314) ~[?:1.8.0_242]
                at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_242]
                at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_242]
                at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_242]
                at sun.security.x509.AlgorithmId.<init>(AlgorithmId.java:114) ~[?:1.8.0_242]
                at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_242]
                at javax.crypto.EncryptedPrivateKeyInfo.<init>(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_242]
                at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:69) ~[graylog.jar:?]
                at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:347) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:172) ~[graylog.jar:?]
                at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:146) ~[graylog.jar:?]
                at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
                at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
                at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_242]

2020-02-20T18:44:42.516Z INFO [Server] SIGNAL received. Shutting down.

2020-02-20T18:44:42.548Z INFO [GracefulShutdown] Graceful shutdown initiated.

2020-02-20T18:44:42.549Z INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting
<3sec> for possible load balancers to recognize state change.

2020-02-20T18:44:46.556Z INFO [GracefulShutdown] Goodbye.

I’m not sure why I am getting this. I am sure I am being dumb though.

2

did you adjust http_publish_uri and `http_external_uri´ to include https ?

3

Yes here is that part of the config.

###############
# HTTP settings
###############

#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
http_bind_address = 192.168.200.27:9000
#http_bind_address = [2001:db8::1]:9000

#### HTTP publish URI
#
# The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all
# clients using the Graylog web interface.
#
# The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
#
# This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address,
# for example if the machine has multiple network interfaces or is behind a NAT gateway.
#
# If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
# This configuration setting *must not* contain a wildcard address!
#
# Default: http://$http_bind_address/
http_publish_uri = https://192.168.200.27:9000/

#### External Graylog URI
#
# The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
#
# The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
# and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
#
# When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#
# This setting can be overriden on a per-request basis with the "X-Graylog-Server-URL" HTTP request header.
#
# Default: $http_publish_uri
#http_external_uri =

I also made sure I imported my CA as a .der as described in.

https://github.com/graylog-labs/shadowCA/blob/ba4985529178efbf3f086d3934f45ae0273a8969/bin/create_ca_certificate.sh#L79-L81

If I remove the HTTPS settings it connects to mongodb and elastic and comes online.

4

is your elasticsearch https secure?

because in the first post you have the message:

(Could not connect to https://vdagraylog.dontsquatme.com:9200)

But did you imported your CA into the java keystore that Graylog can connect to itself and validate the certificates?

5

Here is how we configured elastic.

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil ca --ca-dn CN=dontsquatme.com

sudo /usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip 192.168.200.27 --dns dontsquatme.com

sudo mkdir -p /etc/elasticsearch/certs sudo cp /usr/share/elasticsearch/bin/elastic-certificates.p12 /etc/elasticsearch/certs chmod 444 /etc/elasticsearch/certs/*

/etc/elasticsearch/elasticsearch.yml.

xpack.security.enabled: true 
xpack.security.transport.ssl.enabled: true 
xpack.security.transport.ssl.verification_mode: none 
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12 
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12 
xpack.security.http.ssl.enabled: true 
xpack.security.http.ssl.verification_mode: none 
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

network.host: 192.168.200.27
sudo service elasticsearch restart

But did you imported your CA into the java keystore that Graylog can connect to itself and validate the certificates?

I don’t understand. I imported all CA certs I created into the store graylog uses.

sudo keytool -importcert -keystore /etc/graylog/server/cacerts.jks -storepass changeit -alias elastic-cluster -file /etc/graylog/server/elastic.crt
sudo keytool -importcert -keystore /etc/graylog/server/cacerts.jks -storepass changeit -alias graylogca -file /opt/opensslkeys/graylogca.crt
6

Figured it out.

Create Self signed private key/certificate

openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

Adding a self signed certificate to the JVM trust store
(on Ubuntu 18.04.2 LTS)

cp -a /etc/ssl/certs/java/cacerts /etc/ssl/certs/graylog/cacerts.jks
keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem

Then in my server.conf I have:

http_publish_uri = https://graylog.domain.com:9000/ 
http_enable_tls =true
http_tls_cert_file =/etc/ssl/certs/graylog/cert.pem
http_tls_key_file =/etc/ssl/certs/graylog/pkcs8-plain.pem
#http_tls_key_password =secret

So that works for me. I just wish I could figure out why the tls key password isn’t working. I have it chmod’d just like the certificate, I even changed the owner to graylog.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.