SSL Issues ProxiedResource Unable to call Hostname not verified

[Tg9DCnHB3azWmpJ7b]

1. Describe your incident:

1. I installed graylog 4.2 with a domain http://graylog.example.com
2. I added GELF TCP input and started sending logs from kubernetes cluster via Fluent-bit.
3. No issues. Worth mentioning here that I have purchased wildcard certificate and I use it on other websites without issues. This is not the case of using self signed certificates.
4. I added crt and key files for the domain under: /etc/ssl and enabled below options in /etc/graylog/server/server.conf

http_bind_address = 0.0.0.0:9000
http_external_uri = https://graylog.example.com:9000/
http_enable_tls = true
http_tls_cert_file = /etc/ssl/cert.crt
http_tls_key_file = /etc/ssl/key.key

10. After restarting graylog service I can now access http://graylog.example.com using https.
11. At this point GELF TCP will not start with those lines in the log /var/log/graylog-server/server.log:

2021-11-29T15:51:19.906Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-11-29T15:51:19.906Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/inputstates on node <xxxx-xxxx-xxxx-xxxx-xxxx>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]
2021-11-29T15:51:20.407Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <xxxx-xxxx-xxxx-xxxx-xxxx>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]

https://graylog.example.com works perfectly fine.
12. When clicking on Start Input I get bellow message:

Input 'xxx' could not be started
Request to start input 'xxx' failed. Check your Graylog logs for more information.

13. Weirdly the xxx input is not running but I do see messages coming in.

2. Describe your environment:

• OS Information:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"

• Package Version:
  graylog-4.2-repository_latest.deb

• Service logs, configurations, and environment variables:
  Which?

3. What steps have you already taken to try and solve the problem?
Googled a lot.

4. How can the community help?

• What does this error mean?

• Any other options I need to specify in the config file?

• My goals:

1. Have graylog accessible with https. This is done.
2. Fix the above issue.
3. Have gelf tcp be only accessible with TLS certificate. Can I use the same crt and key as I used for https?

More Logs:

curl -i 'https://xxx.xxx.xxx.xxx:9000/api/?pretty=true'
curl: (60) SSL: no alternative certificate subject name matches target host name 'xxx.xxx.xxx.xxx'
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

2021-11-29T17:11:50.233Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
2021-11-29T17:11:50.235Z INFO  [HttpServer] [HttpServer] Started.
2021-11-29T17:11:50.236Z INFO  [JerseyService] Started REST API at <0.0.0.0:9000>
2021-11-29T17:11:50.237Z INFO  [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=28, DevelopmentDirectoryObserverService [RUNNING]=44, GracefulShutdownService [RUNNING]=45, JobSchedulerService [RUNNING]=45, PrometheusExporter [RUNNING]=45, OutputSetupService [RUNNING]=45, BufferSynchronizerService [RUNNING]=46, UrlWhitelistService [RUNNING]=47, LocalKafkaMessageQueueReader [RUNNING]=47, LocalKafkaMessageQueueWriter [RUNNING]=47, LocalKafkaJournal [RUNNING]=48, InputSetupService [RUNNING]=53, ProcessingConfigurationManager [RUNNING]=55, MongoDBProcessingStatusRecorderService [RUNNING]=59, EtagService [RUNNING]=97, ConfigurationEtagService [RUNNING]=97, UserSessionTerminationService [RUNNING]=106, StreamCacheService [RUNNING]=220, PeriodicalsService [RUNNING]=261, LookupTableService [RUNNING]=414, JerseyService [RUNNING]=5808}
2021-11-29T17:11:50.246Z INFO  [ServerBootstrap] Graylog server up and running.
2021-11-29T17:11:50.247Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-11-29T17:11:50.249Z INFO  [ServiceManagerListener] Services are healthy
2021-11-29T17:11:50.252Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-11-29T17:11:50.314Z INFO  [InputStateListener] Input [GELF TCP/xxxxxxxxxxxxxxx] is now STARTING
2021-11-29T17:11:50.321Z INFO  [InputStateListener] Input [GELF TCP/xxxxxxxxxxxxxxx] is now STARTING
2021-11-29T17:11:50.458Z INFO  [InputStateListener] Input [GELF TCP/xxxxxxxxxxxxxxx] is now RUNNING
2021-11-29T17:11:50.463Z INFO  [InputStateListener] Input [GELF TCP/xxxxxxxxxxxxxxx] is now RUNNING
2021-11-29T17:11:50.493Z WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=xxxxxx, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=12a89a38-ed6d-4424-b915-eeb69cd651d6} (channel [id: 0x36e8e642, L:/0:0:0:0:0:0:0:0%0:12201]) should be >= 1048576 but is 425984.
2021-11-29T17:11:50.493Z WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=xxxxxx, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=null} (channel [id: 0x83dfb851, L:/0:0:0:0:0:0:0:0%0:12202]) should be >= 1048576 but is 425984.
2021-11-29T17:11:53.319Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <12a89a38-ed6d-4424-b915-eeb69cd651d6>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]
2021-11-29T17:11:53.358Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <12a89a38-ed6d-4424-b915-eeb69cd651d6>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]
2021-11-29T17:11:54.839Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <12a89a38-ed6d-4424-b915-eeb69cd651d6>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]

[gsmith]

@Tg9DCnHB3azWmpJ7b

2021-11-29T15:51:19.906Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/inputstates on node <xxxx-xxxx-xxxx-xxxx-xxxx>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]

Hello && Welcome
The above error can be a couple things. Since your able to log on the Graylog Web UI. Here is a list to check for your GELF TCP/TLS input.

1. TLS cert file make sure Graylog has access to this cert
   /etc/graylog/graylog3-certificate.pem
2. TLS private key file make sure Graylog has access to this cert
   . /etc/graylog/graylog3-key.pem
3. If you’re using a TLS key password, make sure it’s set in the Graylog server.conf file also
   Secret

4. Make sure your Keystore in accessible to Graylog

5. Make sure your DNS server has a PTR (Reverse Lookup)for your Graylog server or what you used for the domain.

6. Make sure you have the following. NOTE: IP Address is preferred in the certs that point to your Graylog server etc…

subjectAltName = @alt_names
# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 203.0.113.42
DNS.1 = graylog.example.com

More can be found here
https://docs.graylog.org/v1/docs/https
Sometime Windows have a problem running wild card certs we’ve have experienced this in the past and ended up not using them. Sometimes we were luck and had no problems.

Hope that helps

EDIT: I probably should have shown my setup for better clarity

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem <-- Graylog owns this directory an  has access
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem <-- Graylog owns this directory an  has access
http_tls_key_password = secret

If your not using the default JAVA keystore then the following will apply to you.

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If you’ve been using another password to encrypt the JVM trust store than the default changeit, you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret

[Tg9DCnHB3azWmpJ7b]

Hi,

Thanks for replying.

I’m sure it’s something to do with subjectAltName but all the guides I checked deal with self signed certificates.
I already have a certificate purchased from an external provider. A wildcard one which I can’t change.
How do I configure Graylog in this case?

From this error message I can figure it out that some internal graylog service is trying to connect to https://xxx.xxx.xxx.xxx:9000 but the certificate is from graylog.example.com and not for the IP address is trying to connect to and is failing.

2021-11-30T10:15:27.411Z WARN  [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx>: Hostname xxx.xxx.xxx.xxx not verified:
    certificate: sha256/xxxxxxxxxxxxxxxxxxxxxxxxxx
    DN: CN=*.example.com
    subjectAltNames: [*.example.com, example.com]

Let’s ignore the web ui ssl for a second. Can I use the same wildcard certificate on gelf tcp input?

Can I just add the wildcard certificate to below options? What would be the TLS Client Auth Trusted Certs in this case?

TLS cert file: /etc/ssl/wildcard_graylog.example.com.crt
TLS private key file: /etc/ssl/wildcard_graylog.example.com.key
Enable TLS: Yes
TLS key password: none
TLS client authentication: required
TLS Client Auth Trusted Certs: ?

[gsmith]

Hello

First lets start with this.

@Tg9DCnHB3azWmpJ7b

I checked deal with self signed certificates.

You can find more here and it should give you a better insight.

https://docs.graylog.org/v1/docs/sec-adcs-certificates

@Tg9DCnHB3azWmpJ7b

from this error message I can figure it out that some internal graylog service is trying to connect to https://xxx.xxx.xxx.xxx:9000 but the certificate is from graylog.example.com and not for the IP address

That is where the subjectAltName comes in. Have you tried to configure your /etc/hosts file?

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.12  graylog.domain.com

Basic steps with Graylog are that it has to be in the right format and a the certificates that are used should be in your keystore while graylog has access to them. If the error shows

@Tg9DCnHB3azWmpJ7b

WARN [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system/metrics/multiple on node <xxxxxx-xxxxxx-xxxxxx-xxxxxx-xxxxxx>: Hostname xxx.xxx.xxx.xxx not verified:

The hostname is not verified clues me in that something is either wrong with the certificates OR Graylog Cant access them. If its the format of the certificate then the link above should guide you to fix that.
If Graylog is unable to verified the hostname, I gave a couple of checks shown above hoping it would help you troubleshoot your issue but I don’t know if you did it. Even thou your able to use those certificate in other devices they may not work on Graylog UNLESS you satisfy the requirements stated in the documents.

Here is a few examples of a community members having the same problem.

After SSL - Hostname not verified

I was able to configure SSL with self signed certs Mongo, Elastic, and Graylog on the same server All inputs, alerts, etc… are working BUT… All my Inputs show NOT RUNNING [image] In the log I see… 2021-03-05T09:27:28.737-05:00 WARN [ProxiedResource] Unable to call https://…/api/system/inputstates on node <000>: Hostname host not verified: certificate: I’m not sure how to verify it. I’ve tried several versions of the http_publish_uri in server.conf https://host:9000 https://host.doma…

WARN [ProxiedResource] Unable to call and unable to start inputs

Hi All I installed a graylog server 4 on ubuntu 18 with elasticsearch-oss and nginx. The server is configured with https and a self signed certificate. I’m having problem with the server inputs, I can create the input with the port 1514 but the service didn’t start, in the server.log file I have the following messages: WARN [UdpTransport] Failed to start channel for input SyslogUDPInput{title=WindowsEventsUDP, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} io.netty.channel.…

Java - certification issue

Hello, Our graylog running behind Loadbalancer on two nodes. Bought trusted certificate is linked to Loadbalancer’s hostname/IP Certificate was imported on Loalbalancer + both Graylog nodes. Https connection works well (thanks to web_endpoint_uri=https://Loadbalancer:9000/api), but due to some Java issues inputs doesn’t works in Graylog. I imported plain-text certificate to JVM trust store (cacerts file), and also set Graylog for use this: GRAYLOG_SERVER_JAVA_OPTS="-Djavax.net.ssl.trustSto…

Next

@Tg9DCnHB3azWmpJ7b

Can I just add the wildcard certificate to below options? What would be the TLS Client Auth Trusted Certs in this case?

If you’re using a CA that’s known (e.g., Letsencrypt, etc.), then you can add the following attributes to your server config. Remember format and permissions but even if you use those for your input Graylog has to be able to find them meaning they should be in the right format in your keystore.
These would be the setting shown below

http_enable_tls=true
http_tls_cert_file=/etc/graylog/ssl/fullchain.pem
http_tls_key_file=/etc/graylog/ssl/privkey.pem
http_publish_uri=https://logs00.example.com:9000/

And as long as your CA is part of the systems certificate store, it should work.

@Tg9DCnHB3azWmpJ7b

3. What steps have you already taken to try and solve the problem?
Googled a lo

Can I ask what have you tried to resolve this issue besides googling? Because this issue could be a couple things that may need to be reconfigured, so in other words trying and reposting what you have done would help us narrow it down.

hope that helps

[Tg9DCnHB3azWmpJ7b]

Hi,
Thank you for the reply. I will read through and update the ticket.