HTTPS does not work but it also spoils HTTP

I don’t even know where to start since I’m desperate.

First of all, it is my intention to enter, our signed certificate and everything in GrayLog for an HTTPS connection, a PKCS12 certificate which I have followed the passes and I have divided into X.509 and PKCS8 and apparently, graylog me detects without problems.

The problem appears every time that in the configuration file /etc/graylog/server/server.conf I make a modification regarding TLS nothing works for me, I will show examples so that you can understand me:

image

I am accessing with my IP address and port 9000, there is only configured in server.conf the following

image

Now I am going to proceed to enter the URL that I will use for the HTTPS connection

image

image

It is already searched and does not access at any time neither by accessing by HTTP: IP: 9000 nor by https even though it already has the certificates installed

I have a gigantic mess on my head and I would need to please someone with patience since I am brand new could help me, thank you very much in advance.

All the best

Hey there, I get your frustration and understand that TLS isn’t the easiest thing to implement. One thing I’ll note: we have a category template that is designed to gather information when you open a topic. It’s there to help folks in the community better help you–the more information you provide, generally the speedier resolution you’ll have.

That said, it’s not clear what you’ve tried and what you’ve not tried. See How to Post a Question in the Community that Gets Responses. So, walk us through, step by step, what you’ve done. Are you using self-signed certs? If so, did you follow the docs for enabling TLS on Graylog?

From what you’ve provided, it definitely doesn’t seem like you’ve followed the docs and have all the attributes configured in server.conf to enable TLS. See my config below:

Aaron's Graylog Config
root@logs00:~# cat /etc/graylog/server/server.conf
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = THISISASECRETDONTTELLANYONE
root_password_sha2 = SHHSHHSHHSHHAAAAAAA2
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_enable_cors = true
elasticsearch_hosts = http://elastisearch00.example.com:9200,http://elasticsearch01.example.com:9200,http://elasticsearch02.example.com:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://mongo00.example.com:27017,mongo01.example.com:27017,mongo02.example.com:27017/graylog?replicaSet=rs0
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
versionchecks = false
http_enable_tls=true
http_tls_cert_file=/etc/graylog/ssl/fullchain.pem
http_tls_key_file=/etc/graylog/ssl/privkey.pem
http_publish_uri=https://logs00.example.com:9000/
auditlog_log4j_enabled = true
auditlog_log4j_logger_name = gl-org.graylog.plugins.auditlog
auditlog_log4j_marker_name = AUDIT_LOG
elasticsearch_version = 7
prometheus_exporter_enabled = true
prometheus_exporter_bind_address = 0.0.0.0:9833

Of particular interest are the http_ attributes, which you can see here:

http_bind_address = 0.0.0.0:9000
http_enable_cors = true
http_enable_tls=true
http_tls_cert_file=/etc/graylog/ssl/fullchain.pem
http_tls_key_file=/etc/graylog/ssl/privkey.pem
http_publish_uri=https://logs00.example.com:9000/

I can’t really speak to using self-signed certs, as I use letsencrypt, which works great for my lab. So that said, using TLS on Graylog works.

The other part of what the community is missing is your logs. Given that this is a log management software, that’s often the very first thing that folks will ask for and there’s often useful information that will tell you why TLS isn’t working.

So, please provide your full config and your logs–they’ll both be necessary for folks in the community to continue to help you get your issue resolved.

2 Likes

Hello,
To add on @aaronsachs post. Out of curiosity in your URL, have you tried typing

https://you_domain_name.net:9000

The reason I’m asking is I see a number “1” in the url but in your file I see FQDN. Just a suggestion. Normally this happens when you dont redirect.

Good morning and first of all thank you very much for lending me your time and trying to help me.

my configuration is exactly the same as yours so it gives me to think that the certificate may not be working, my certificate is not self-signed
, It is a WildCar, but when I look at the GrayLog logs, I do not see any type of error on the part of the certificate, what I do appreciate is that all the errors are on the part of Elasticsearch, I do not know if this could be what I am causing the problem:

I am beginning to think that the problem is in the key storage, that the person in charge of doing this step may not have done it since I am verifying the steps one by one, and I have the Java version 1.8.292 installed but when performing the following command happens the following:

cp -a “$ {JAVA_HOME} / jre / lib / security / cacerts” graylog-certificate.pem

cp: cannot stat ‘/ jre / lib / security / cacerts’: No such file or directory

Likewise, when making a cat to
/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/security/cacerts

I can see that the certificate is already provided

but when I execute the following command, nothing appears
keytool -keystore /path/to/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1

Hi Good morning, I have been able to discover something else although I do not know what it can refer to or what measure I can take since it is a secure WildCard certificate of ours.

Error:
The supported TLS protocols could not be detected. Maintaining the default

What’s the broader context of the error message? Can you please post the logs showing this message?

Of course, I have done a Graylog restart to show it all the information it shows me in the logs

2021-08-20T11:37:03.765+02:00 ERROR [DefaultTLSProtocolProvider] Failed to detect supported TLS protocols. Keeping default <[TLSv1.2, TLSv1.3]>
java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.security.Provider$Service.newInstance(Provider.java:1617) ~[?:1.8.0_292]
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) ~[?:1.8.0_292]
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) ~[?:1.8.0_292]
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) ~[?:1.8.0_292]
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) ~[?:1.8.0_292]
at org.graylog2.shared.security.tls.DefaultTLSProtocolProvider.getDefaultSupportedTlsProtocols(DefaultTLSProtocolProvider.java:42) [graylog.jar:?]
at org.graylog2.Configuration.(Configuration.java:163) [graylog.jar:?]
at org.graylog2.commands.Server.(Server.java:105) [graylog.jar:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_292]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_292]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_292]
at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:39) [graylog.jar:?]
at com.github.rvesse.airline.DefaultCommandFactory.createInstance(DefaultCommandFactory.java:25) [graylog.jar:?]
at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:94) [graylog.jar:?]
at com.github.rvesse.airline.parser.ParseResult.getCommand(ParseResult.java:114) [graylog.jar:?]
at com.github.rvesse.airline.parser.command.CliParser.parse(CliParser.java:54) [graylog.jar:?]
at com.github.rvesse.airline.Cli.parse(Cli.java:127) [graylog.jar:?]
at com.github.rvesse.airline.Cli.parse(Cli.java:113) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Caused by: java.security.KeyStoreException: problem accessing trust store
at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:1.8.0_292]
at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:1041) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:1011) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:1186) ~[?:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292]
at java.security.Provider$Service.newInstance(Provider.java:1595) ~[?:1.8.0_292]
… 19 more
Caused by: java.io.IOException: Invalid keystore format
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:666) ~[?:1.8.0_292]
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57) ~[?:1.8.0_292]
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_292]
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71) ~[?:1.8.0_292]
at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_292]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365) ~[?:1.8.0_292]
at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313) ~[?:1.8.0_292]
at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55) ~[?:1.8.0_292]
at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:1.8.0_292]
at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:250) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:1041) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:1011) ~[?:1.8.0_292]
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:1186) ~[?:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_292]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_292]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_292]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_292]
at java.security.Provider$Service.newInstance(Provider.java:1595) ~[?:1.8.0_292]
… 19 more
2021-08-20T11:37:03.987+02:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.11 [org.graylog.aws.AWSPlugin]
2021-08-20T11:37:03.988+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 4.0.11 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-08-20T11:37:03.989+02:00 INFO [CmdLineTool] Loaded plugin: Integrations 4.0.11 [org.graylog.integrations.IntegrationsPlugin]
2021-08-20T11:37:03.990+02:00 INFO [CmdLineTool] Loaded plugin: Collector 4.0.11 [org.graylog.plugins.collector.CollectorPlugin]
2021-08-20T11:37:03.991+02:00 INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 4.0.11 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-08-20T11:37:03.992+02:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.11 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-08-20T11:37:03.992+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.11+e4e88a4 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-08-20T11:37:03.992+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.11+e4e88a4 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-08-20T11:37:04.235+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-08-20T11:37:04.476+02:00 INFO [Version] HV000001: Hibernate Validator null
2021-08-20T11:37:08.237+02:00 INFO [InputBufferImpl] Message journal is enabled.
2021-08-20T11:37:08.269+02:00 INFO [NodeId] Node ID: 8b9a28d0-af81-4fe9-8ea6-318bfbacb08f
2021-08-20T11:37:08.464+02:00 INFO [LogManager] Loading logs.
2021-08-20T11:37:08.487+02:00 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000129198.index, deleting and rebuilding index…
2021-08-20T11:37:08.614+02:00 INFO [LogManager] Logs loading complete.
2021-08-20T11:37:08.617+02:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-08-20T11:37:08.638+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2021-08-20T11:37:08.692+02:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-08-20T11:37:08.708+02:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:3}] to localhost:27017
2021-08-20T11:37:08.713+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 26]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2812690}
2021-08-20T11:37:08.725+02:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:4}] to localhost:27017
2021-08-20T11:37:08.925+02:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.

Also, I have been able to check the status of GrayLog, and it is active and working (or so it is supposed) but I have seen that it shows me the following information:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.assistedinject.FactoryProvider2$MethodHandleWrapper (file:/usr/share/graylog-server/graylog.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of com.google.inject.assistedinject.FactoryProvider2$MethodHandleWrapper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I have reinstalled java since from what I could see the error was there, after this I have managed to access graylog by HTTP again, I have reconfigured HTTPS and the certificates and it ended up throwing an error again, to keep its use although It was through HTTP, remove the HTTPS configuration and the certificates but nothing, now I get this error:

ERROR: Unable to probe any host for Elasticsearch version!

Please see the following link(s) to help you with this error:

Need further help?

Terminating. :frowning:

################################################################################

2021-08-20T13:17:20.812+02:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.11 [org.graylog.aws.AWSPlugin]
2021-08-20T13:17:20.815+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 4.0.11 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2021-08-20T13:17:20.816+02:00 INFO [CmdLineTool] Loaded plugin: Integrations 4.0.11 [org.graylog.integrations.IntegrationsPlugin]
2021-08-20T13:17:20.817+02:00 INFO [CmdLineTool] Loaded plugin: Collector 4.0.11 [org.graylog.plugins.collector.CollectorPlugin]
2021-08-20T13:17:20.818+02:00 INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 4.0.11 [org.graylog.plugins.enterprise.EnterprisePlugin]
2021-08-20T13:17:20.819+02:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.11 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-08-20T13:17:20.819+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.11+e4e88a4 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-08-20T13:17:20.819+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.11+e4e88a4 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-08-20T13:17:21.054+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-08-20T13:17:21.289+02:00 INFO [Version] HV000001: Hibernate Validator null

Hello

I feel your stuggle with certificates.Can you show us what/where the documentation you used to create your certificates?

How did you make your keystore? or are you using the Java default keystore ( cacerts)?
EDIT: Showning your configuration for https would be apperciated also.

Good afternoon and first of all thank you very much for lending me your help.

The storage key that I am using in this case is Java’s own, it is true that I am not very familiar with this so it is possible that I am not using it well.

The certificate that I am using is not one generated by me, it is a WildCard that we have contracted.

Now, I have reinstalled everything, both Graylog and MongoDB and Elasearch to start the clean configuration again, but when reconfiguring the certificate section the same thing happens again, but peculiarly with the following error:

java.net.ConnectException: Failed to connect to /127.0.0.1:9200
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:204) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.rootResponse(VersionProbe.java:120) ~[graylog.jar:?]
at org.graylog2.storage.versionprobe.VersionProbe.probe(VersionProbe.java:73) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_292]
at java.util.Collections$2.tryAdvance(Collections.java:4719) ~[?:1.8.0_292]
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_292]
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) ~[?:1.8.0_292]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) ~[?:1.8.0_292]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_292]
at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) ~[?:1.8.0_292]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_292]
at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531) ~[?:1.8.0_292]
at org.graylog2.storage.versionprobe.VersionProbe.probe(VersionProbe.java:54) ~[graylog.jar:?]
at org.graylog2.storage.providers.ElasticsearchVersionProvider.lambda$get$1(ElasticsearchVersionProvider.java:68) ~[graylog.jar:?]
at org.graylog2.storage.providers.AtomicCache.get(AtomicCache.java:36) [graylog.jar:?]
at org.graylog2.storage.providers.ElasticsearchVersionProvider.get(ElasticsearchVersionProvider.java:67) [graylog.jar:?]
at org.graylog2.storage.providers.ElasticsearchVersionProvider.get(ElasticsearchVersionProvider.java:35) [graylog.jar:?]
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85) [graylog.jar:?]
at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77) [graylog.jar:?]
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59) [graylog.jar:?]
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:198) [graylog.jar:?]
at com.google.inject.internal.RealMultibinder$RealMultibinderProvider.doProvision(RealMultibinder.java:151) [graylog.jar:?]
at com.google.inject.internal.InternalProviderInstanceBindingImpl$Factory.get(InternalProviderInstanceBindingImpl.java:113) [graylog.jar:?]
at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:52) [graylog.jar:?]
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [graylog.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:87) [graylog.jar:?]
at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:381) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:196) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_292]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_292]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_292]
at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_292]
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130) ~[graylog.jar:?]
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263) ~[graylog.jar:?]
… 125 more
2021-08-21T20:30:45.956+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.017+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.240+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.253+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.574+02:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.581+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2021-08-21T20:30:46.601+02:00 ERROR [CmdLineTool]

################################################################################

ERROR: Unable to probe any host for Elasticsearch version!

Please see the following link(s) to help you with this error:

Need further help?

Terminating. :frowning:

################################################################################

Hello

I assume this is just your Graylog log file?

You might want to check you graylog configuration /etc/graylog/server/server.conf. for ES connection/s.

To help you further, here is example of my lab GL server. As shown below, this is where all the HTTPS magic happens, and my configuration for Elasticsearch connection.

http_bind_address = 8.8.8.8:9000  ### This can be configured as (127.0.0.1, 0.0.0.0, or your_ipaddress)
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://8.8.8.8:9200

Here is my elasticsearch.yml file

cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 8.8.8.8
http.port: 9200

Make sure your graylog service has access the certificates, so the easiest way I know for you would be is to set them in your Graylog directory /etc/graylog/ and then check to make sure graylog has permissions to acces them.

chown graylog:graylog -R /etc/graylog/

Here is a list of all default file locations.

https://docs.graylog.org/en/4.1/pages/configuration/file_location.html

Your issue can be a couple different incorrect configurations along with your certificates.
First, showing your Graylog/Elasticsearch configuration files would be apperciated. Maybe we can resolve any issue coming from those files.
Second, you might want to look at this documentation if you have not already.

https://docs.graylog.org/en/4.1/pages/configuration/https.html

You may find something in this post that will help you.

Couple of steps I would take:

Checking your elasticsearch status, look for any errors/warning that may pertain to this issue. Showing your results here would be great.

systemctl status elasticsearch

You can check your ES health. If your using ipaddress you can replace “localhost” with that address.

curl -XGET http://localhost:9200/_cluster/health?pretty=true
EDIT: check your firewall if its enabled, to allow port 9200.

hope that helps

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.