Graylog 3.0.2 https

cbgraham · August 14, 2019, 7:15pm

Brand new graylog user here. I have it setup following the easy to follow setup guide for ubuntu 18.04. It’s working great, and I have logbeats coming in and am loving it so far. I’ve really run into a hurdle that I thought would be simple, and maybe it is simple and I’m just missing something, but I can’t get https working. I’ve tried using the resources at docs.graylog.org/en/3.0/pages/configuration/ to no avail. What I’ve tried so far:
Using my existing wildcard certificate from a pfx following the instructions for the pfx on the graylog https page. I input the resulting pem files in my server.conf:

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret =  #
root_password_sha2 = #
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.16.125:9000
http_publish_uri = https://192.168.16.125:9000/
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/cert.pem
http_tls_key_file = /etc/ssl/certs/graylog/pkcs8-encrypted.pem
http_tls_key_password =secret
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = true
proxied_requests_thread_pool_size = 32

I also imported the resulting certificate .pem file into the cacerts.jks file I’m pointing to in /etc/default/graylog-server:

# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/cacerts.jks"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=""

However, graylog fails to start with Jersey errors:

Hmm, I can’t paste my entire log without running into a forum error of maxing out 2 links in a post

I also tried generating a self signed certificate following the guide, and then importing the certificate into the jks file, I got the same results. Anyone have any suggestions?

cawfehman · August 14, 2019, 8:04pm

what does the log file say?

gsmith · August 15, 2019, 4:01am

@cbgraham
I had this problem with Graylog 3.0+ to work with HTTPS. I found my answer here.

I did run into a couple of issues with this , but i was able to work it out.
Hope this helps

cbgraham · August 15, 2019, 11:38am

Blockquote
2019-08-14T11:41:49.298-07:00 INFO [GracefulShutdown] Goodbye.
2019-08-14T11:42:02.164-07:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 3.0.2 [org.graylog.aws.AWSPlugin]
2019-08-14T11:42:02.166-07:00 INFO [CmdLineTool] Loaded plugin: Collector 3.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2019-08-14T11:42:02.167-07:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2019-08-14T11:42:02.425-07:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/cacerts.jks -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2019-08-14T11:42:02.607-07:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2019-08-14T11:42:04.278-07:00 INFO [InputBufferImpl] Message journal is enabled.
2019-08-14T11:42:04.297-07:00 INFO [NodeId] Node ID: 37a982d5-0835-4906-870d-c1517f87d65a
2019-08-14T11:42:04.464-07:00 INFO [LogManager] Loading logs.
2019-08-14T11:42:04.513-07:00 INFO [LogManager] Logs loading complete.
2019-08-14T11:42:04.517-07:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2019-08-14T11:42:04.526-07:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2019-08-14T11:42:04.544-07:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2019-08-14T11:42:04.590-07:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2019-08-14T11:42:04.611-07:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:2319}] to localhost:27017
2019-08-14T11:42:04.615-07:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 10]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2793402}
2019-08-14T11:42:04.641-07:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:2320}] to localhost:27017
2019-08-14T11:42:04.945-07:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2019-08-14T11:42:04.945-07:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2019-08-14T11:42:05.001-07:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2019-08-14T11:42:05.002-07:00 INFO [JestClientFactory] Node Discovery disabled…
2019-08-14T11:42:05.002-07:00 INFO [JestClientFactory] Idle connection reaping disabled…
2019-08-14T11:42:05.112-07:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2019-08-14T11:42:05.456-07:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-08-14T11:42:05.463-07:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2019-08-14T11:42:05.513-07:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-08-14T11:42:05.516-07:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:2321}] to localhost:27017
2019-08-14T11:42:05.569-07:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-08-14T11:42:05.609-07:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-08-14T11:42:05.646-07:00 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2019-08-14T11:42:06.067-07:00 INFO [ServerBootstrap] Graylog server 3.0.2+1686930 starting up
2019-08-14T11:42:06.067-07:00 INFO [ServerBootstrap] JRE: Private Build 1.8.0_222 on Linux 4.15.0-55-generic
2019-08-14T11:42:06.067-07:00 INFO [ServerBootstrap] Deployment: deb
2019-08-14T11:42:06.068-07:00 INFO [ServerBootstrap] OS: Ubuntu 18.04.2 LTS (bionic)
2019-08-14T11:42:06.068-07:00 INFO [ServerBootstrap] Arch: amd64
2019-08-14T11:42:06.090-07:00 INFO [PeriodicalsService] Starting 27 periodicals …
2019-08-14T11:42:06.091-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2019-08-14T11:42:06.100-07:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2019-08-14T11:42:06.103-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2019-08-14T11:42:06.105-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2019-08-14T11:42:06.106-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2019-08-14T11:42:06.107-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2019-08-14T11:42:06.109-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2019-08-14T11:42:06.111-07:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2019-08-14T11:42:06.112-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2019-08-14T11:42:06.113-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2019-08-14T11:42:06.113-07:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:2322}] to localhost:27017
2019-08-14T11:42:06.119-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2019-08-14T11:42:06.125-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2019-08-14T11:42:06.127-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2019-08-14T11:42:06.131-07:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2019-08-14T11:42:06.135-07:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2019-08-14T11:42:06.137-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2019-08-14T11:42:06.137-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2019-08-14T11:42:06.143-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2019-08-14T11:42:06.146-07:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:2323}] to localhost:27017
2019-08-14T11:42:06.158-07:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2019-08-14T11:42:06.163-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2019-08-14T11:42:06.164-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2019-08-14T11:42:06.175-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2019-08-14T11:42:06.186-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2019-08-14T11:42:06.187-07:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2019-08-14T11:42:06.188-07:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2019-08-14T11:42:06.191-07:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2019-08-14T11:42:06.192-07:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2019-08-14T11:42:06.194-07:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2019-08-14T11:42:06.508-07:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2019-08-14T11:42:06.509-07:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2019-08-14T11:42:06.515-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2019-08-14T11:42:06.516-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2019-08-14T11:42:06.516-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2019-08-14T11:42:06.517-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2019-08-14T11:42:06.517-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2019-08-14T11:42:06.517-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2019-08-14T11:42:06.517-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2019-08-14T11:42:06.517-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2019-08-14T11:42:06.518-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical].
2019-08-14T11:42:06.519-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRangesCleanupPeriodical] complete, took <0ms>.
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical].
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] complete, took <0ms>.
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread].
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] complete, took <0ms>.
2019-08-14T11:42:06.520-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads].
2019-08-14T11:42:06.521-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2019-08-14T11:42:06.521-07:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2019-08-14T11:42:06.521-07:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2019-08-14T11:42:06.529-07:00 INFO [LogManager] Shutting down.
2019-08-14T11:42:06.534-07:00 INFO [LookupDataAdapterRefreshService] Stopping 0 jobs
2019-08-14T11:42:06.546-07:00 INFO [Buffers] Waiting until all buffers are empty.
2019-08-14T11:42:06.555-07:00 INFO [Buffers] All buffers are empty. Continuing.
2019-08-14T11:42:06.557-07:00 INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2019-08-14T11:42:06.558-07:00 INFO [LogManager] Shutdown complete.
2019-08-14T11:42:06.614-07:00 INFO [JournalReader] Stopping.
2019-08-14T11:42:06.615-07:00 INFO [ServiceManagerListener] Services are now stopped.
2019-08-14T11:42:06.615-07:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:741) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:314) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:148) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:210) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.io.IOException: ObjectIdentifier() – data isn’t an object ID (tag = 48)
at sun.security.util.ObjectIdentifier.(ObjectIdentifier.java:257) ~[?:1.8.0_222]
at sun.security.util.DerInputStream.getOID(DerInputStream.java:314) ~[?:1.8.0_222]
at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267) ~[sunjce_provider.jar:1.8.0_222]
at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293) ~[?:1.8.0_222]
at sun.security.x509.AlgorithmId.decodeParams(AlgorithmId.java:132) ~[?:1.8.0_222]
at sun.security.x509.AlgorithmId.(AlgorithmId.java:114) ~[?:1.8.0_222]
at sun.security.x509.AlgorithmId.parse(AlgorithmId.java:372) ~[?:1.8.0_222]
at javax.crypto.EncryptedPrivateKeyInfo.(EncryptedPrivateKeyInfo.java:95) ~[?:1.8.0_222]
at org.graylog2.shared.security.tls.PemKeyStore.generateKeySpec(PemKeyStore.java:69) ~[graylog.jar:?]
at org.graylog2.shared.security.tls.PemKeyStore.buildKeyStore(PemKeyStore.java:98) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.buildSslEngineConfigurator(JerseyService.java:342) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:168) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:142) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:119) ~[graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222]
2019-08-14T11:42:06.620-07:00 INFO [Server] SIGNAL received. Shutting down.
2019-08-14T11:42:06.626-07:00 INFO [GracefulShutdown] Graceful shutdown initiated.
2019-08-14T11:42:06.626-07:00 INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2019-08-14T11:42:10.628-07:00 INFO [GracefulShutdown] Goodbye.

Blockquote

cbgraham · August 15, 2019, 2:41pm

I actually followed your guide step by step before posting this and still had the same problem. I’m not sure what I’m missing.

gsmith · August 15, 2019, 10:00pm

@cbgraham
Yeah, I had some problems also, but I was able to get it to work.
My Environment is;

Virtual machine with CentOS 7, all packages is fully updated. Hardware: 6 Processors, 8GB Ram, and 1TB HDD.
graylog-server-3.0.0-12.noarch
elasticsearch-6.6.1-1.noarch
mongodb-org-4.0.6-1.el7.x86_64

This is how I solved my problem/s.
When I was executing the command for this, I noticed I had to adjust some of the line/s to fit my needs.

From;

keytool -import -trustcacerts -file graylog.crt -alias server -keystore graylog_keystore.jks -storepass secret

To;

keytool -import -trustcacerts -file graylog.crt -alias graylog-labs.net -keystore graylog_keystore.jks -storepass secret

Then I was able to execute this;
From;

keytool -list -v -keystore graylog_keystore.jks -alias graylog.domain.com

To;

keytool -list -v -keystore graylog_keystore.jks -alias graylog-labs.net

My graylog.conf was set as this;

[http]
http_bind_address = graylog-labs.net:9000
http_publish_uri = https://graylog-labs.net:9000/

[https]
http_enable_tls = true
http_tls_cert_file =/etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file =/etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Once I restart the Graylog service I received an error;
“PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”

I execute the following;

keytool -import -trustcacerts -file graylog-certificate.pem -alias graylog -labs.net -keystore graylog-key.jks
keytool -storepasswd -keystore graylog_keystore.jks (making sure it was set as secret)

I restart the Graylog Service again
I tail’d the graylog log file, looking for errors.
Took a few minutes but came up.
NOTE: My graylog server has DNS PTR

My Edit is as follow;
root# vi /etc/sysconfig/graylog-server

# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/graylog-key.jks"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=""

Hope this helps

cbgraham · August 15, 2019, 10:36pm

Here is what I have tried numerous times already. I really don’t want to admit the amount of time I’ve spent on this.

https://docs.graylog.org/en/3.0/pages/configuration/https.html#creating-a-self-signed-private-key-certificate
combined with
https://docs.graylog.org/en/3.0/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store
Which seems like the right combination to get the cert, and add it the java trust store, but maybe it’s not what you are supposed to do?
I used the resultant cert.pem and pkcs8-encrypted.pem as the certificates in my server.conf.
Result: didn’t work

I tried the steps in Graylog3 with https (easy tutorial)
But I had to modifiy a couple things at the end which seemed to overwrite some steps that were previously done. There is a confusing array of .jks files in the final steps with 3 different jks files. I imagine they were typos referencing actually just 1 jks file?

Greg, I noticed your helpful response you have 2 different jks files, did you end up with 2, or was that just a typo? Also, I thought the jks default password was changeit and if you changed it from that you have to add another java opt to set the password?

I might have to end up going with the nginx solution here:

if this doesn’t work, but I’m really trying to use the graylog built https method.

gsmith · August 16, 2019, 3:34am

@cbgraham
just an FYI, from the begining I reset my java keystore called “cacerts”. I made a backup before starting all of this. So i reset my cacerts file to default.
Then after the first line of Graylog3 with https (easy tutorial)

mkdir /etc/ssl/certs/graylog/ && cd /etc/ssl/certs/graylog/

I made the file called “openssl-graylog.cnf” as shown below.
Then I used this ssl-setup

I quote “create a file named openssl-graylog.cnf with the following content (customized to your needs):”

    [req]
    distinguished_name = req_distinguished_name
    x509_extensions = v3_req
    prompt = no

    # Details about the issuer of the certificate
    [req_distinguished_name]
    C = US
    ST = Some-State
    L = Some-City
    O = My Company
    OU = My Division
    CN = graylog.example.com

    [v3_req]
    keyUsage = keyEncipherment, dataEncipherment
    extendedKeyUsage = serverAuth
    subjectAltName = @alt_names

    # IP addresses and DNS names the certificate should include
    # Use IP.### for IP addresses and DNS.### for DNS names,
    # with "###" being a consecutive number.
    [alt_names]
    IP.1 = 203.0.113.42
    DNS.1 = graylog.example.com

Then I continued through the rest of the documentation from Graylog3 with https (easy tutorial)
As stated from above I made some configuration prior to finishing the steps in that documentation.
When you restart the Graylog service make sure you tail graylogs log file and look for WARN or ERROR signs.I also stated from above what error I received and how i fixed it. These instruction is not what you do, Its how you do it. You might need to start from the being, it seems that you have different things configured from different sources. I to had to roll everything back and get a fresh start. To be honest I have tried for three months to figure out this problem unitl I came accross Graylog3 with https (easy tutorial) I just used his direction and thats it, with some adjustments.
Hope this helps

gsmith · August 16, 2019, 3:43am

@cbgraham

vi /etc/graylog/server/server.conf

http_tls_key_password = secret
Set password in Java config no I did not, I reset the Java keystore password I made from the instruction the same as in the Graylog config file

tobiasreckhard · August 16, 2019, 1:15pm

Don’t give up, it can be done. I just went through the process yesterday (on a Debian 9 (jessie) system), but with a slightly different certificate setup. I had previously created a private CA and had it sign some certificates, and used that to sign my Graylog certificate, too. I used rather old-fashioned RSA with 4096 bytes, using graylog.$domain as CN of the subject and providing $http_bind_address (just the IP address, not the port) and the hostname in FQDN form as subjectAltNames. I did not encrypt the secret key, but instead gave it 0600 permissions and made the graylog user the owner. Then made a copy of the openjdk cacerts file in /etc/graylog/server and added the CA certificate (not the graylog certificate) to it using keytool. And then I modified GRAYLOG_SERVER_JAVA_OPTS in /etc/default/graylog-server, adding “-Djavax.net.ssl.trustStore=/etc/graylog/server/cacerts,jks”.

On the way there, I’d stumbled twice:

The first certificate I’d created hadn’t had the extension for server authentication
The key file wasn’t readable by graylog, I’d had it belonging to root:root initially with permissions set to 0600.

I’m not sure I completely understand the log file snippet you provide, but I suspect there is some problem accessing the certificate (PEM file), the secret key or perhaps the keystore. You say you used a wildcard certificate. That should probably work, but what did you put into the CA store? You need to put the CA certificate into it, not the wildcard certificate. And I would perhaps try with the key in unencrypted form.

Good luck.

cbgraham · August 19, 2019, 11:39pm

Update: Got it working… mostly.
I updated to 3.1 in the hopes that might help. (it actually made troubleshooting more difficult as graylog does not start and stop gracefully any longer with https configurations in server.conf that it doesn’t like)
I went through the instructions on creating self signed certificate and then importing to a fresh copy of cacerts.jks one more time from a blank slate while on 3.1 making sure every t was crossed and i dotted. Still didn’t work. I then took Tobias’ instructions of trying without an encrypted key and that worked. So here is the final rundown of what worked for me (from https://docs.graylog.org/en/3.1/pages/configuration/https.html#ssl-setup)

Create Self signed private key/certificate

openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem
openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

Adding a self signed certificate to the JVM trust store
(on Ubuntu 18.04.2 LTS)

cp -a /etc/ssl/certs/java/cacerts /etc/ssl/certs/graylog/cacerts.jks
keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem

Then in my server.conf I have:

http_publish_uri = https://graylog.domain.com:9000/ 
http_enable_tls =true
http_tls_cert_file =/etc/ssl/certs/graylog/cert.pem
http_tls_key_file =/etc/ssl/certs/graylog/pkcs8-plain.pem
#http_tls_key_password =secret

And in my /etc/default/graylog-server:

GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/ssl/certs/graylog/cacerts.jks"

So that works for me. I just wish I could figure out why the tls key password isn’t working. I have it chmod’d just like the certificate, I even changed the owner to graylog.

system · September 2, 2019, 11:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cant get HTTPS working on graylog Graylog Central (peer support)	12	4638	June 7, 2017
Secure Multi-node environment Graylog Central (peer support)	2	684	December 2, 2020
Configuration HTTPS Graylog Central (peer support)	17	4172	April 26, 2018
HTTPS setup in Graylog 3.0.0 Graylog Central (peer support)	3	1890	April 9, 2019
Graylog 3.2 + https - pkix error Graylog Central (peer support)	6	1345	February 25, 2020

Graylog 3.0.2 https

Related Topics