Setting for Graylog to not verify SSL/TLS for comms between Opensearch

[cbabs (Chris B)]

1. Describe your incident:

I have Opensearch setup in K8s/Docker. When the Opensearch containers start they generate SSL for comms. I had to use SSL if I wanted to use Opensearch Dashboard(kinda pointless in K8s, but whatever).

Because the certs are self-signed, GL wont establish a connection.

2. Describe your environment:

• OS Information:

K8s ver 1.17.3
Worker nodes: RHEL 3.10.0-1160.90.1.el7.x86_64

• Package Version:

image: graylog/graylog:5.1.1-1

• Service logs, configurations, and environment variables:

metadata:
name: gl-configmap-master
data:
GRAYLOG_ELASTICSEARCH_HOSTS: “ht tps /opensearch-cluster-master:9200”
GRAYLOT_ELASTICSEARCH_DISCOVERY_ENABLED: “true”
GRAYLOG_MONGODB_URI: “mongo db: / /mongo:27017/graylog”
GRAYLOG_REST_TRANSPORT_URI: “htt p://0.0.0.0:12900”
GRAYLOG_REST_LISTEN_URI: “http: / /0.0.0.0:12900”
GRAYLOG_WEB_LISTEN_URI: “http: // 0.0.0.0:9000”
GRAYLOG_HTTP_PUBLISH_URI: “ht tp: //secret:31900”
GRAYLOG_IS_MASTER: “true”
GRAYLOG_transport_email_use_auth: “false”
GRAYLOG_transport_email_use_tls: “true”
GRAYLOG_transport_email_hostname: "secret
GRAYLOG_transport_email_use_ssl: “false”
GRAYLOG_transport_email_subject_prefix: “[graylog]”
GRAYLOG_transport_email_from_email: "secret
GRAYLOG_transport_email_enabled: “true”
GRAYLOG_transport_email_port: “25”

GL logs:
2023-06-14 13:18:20,020 ERROR: org.graylog2.storage.versionprobe.VersionProbe - Unable to retrieve version from Elasticsearch node: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target. - unable to find valid certification path to requested target.
2023-06-14 13:18:20,021 INFO : org.graylog2.storage.versionprobe.VersionProbe - Elasticsearch is not available. Retry #32053

3. What steps have you already taken to try and solve the problem?

I tried to find a setting that disables SSL verification for Opensearch, no luck. I also tested successfully without SSL.

4. How can the community help?

Confirm if I can tell GL to not verify certs for comms between GL and Opensearch.

[drewmiranda-gl (Drew Miranda)]

I’m not entirely sure if this can be done, but if you want to test something out, you can try adding the public key for the cert configured for opensearch, to your graylog java key store (JKS).

Check out the " JAVA KEY STORE" section of How-To Guide: Securing Graylog with TLS

For what it is worth, you can use opensearch dashboards without TLS. I’m doing this in my my lab, BUT the only way to get it working is to instal it locally via deb/rpm/yum, and then delete opensearch-dashboards/plugins/securityDashboards

[Kulgar (Regis Millet)]

I had the exact same issue but my opensearch was setup using their official ansible playbook.

The playbook is configured - by default - to install opensearch with SSL connections. I found it a bit hard to update it so that opensearch would run without HTTPS.

So… fortunately, I found this question and your answer @drewmiranda-gl

Thanks to the guide you pointed to, I managed to launch graylog and make it work using opensearch self-signed certificates.

Here is what I did:

Note: As my graylog install is really recent, I don’t have openJDK installed, so I used the JDK bundled in the graylog installation (thanks to the guide for pointing that).

sudo cp /usr/share/graylog-server/jvm/lib/security/cacerts /etc/graylog/opensearch.jks
sudo chown graylog:graylog /etc/graylog/opensearch.jks

/usr/share/graylog-server/jvm/bin/keytool -importcert -keystore /etc/graylog/opensearch.jks -storepass changeit -alias opensearch_http -file /usr/share/opensearch/config/host_http.pem

Do the keytool -importcert for whichever certs you need. I think you need to import only the one used by the HTTPS interface of opensearch. Do try to import the other ones if that doesn’t work for you.

Then, I edited /etc/default/graylog-server and added this to the first line:

-Djavax.net.ssl.trustStore=/etc/graylog/opensearch.jks

# Then, the first line becomes:
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/opensearch.jks"

And finally, make sure you have the correct settings in /etc/graylog/server/server.conf for the opensearch connection:

elasticsearch_hosts = https://user:password@opensearch_ip:opensearch_port

And that worked! Graylog is now using the HTTPS interface of opensearch.

Next step would be: how to set graylog so that it uses the nginx that proxies opensearch and configured with a let’s encrypt certificate instead of using the opensearch direct port & self signed generated certificates…

[gsmith]

Hey @cbabs

These setting are for Graylog 2.5.0 not for the image: graylog/graylog:5.1.1-1, might want to check your ENV variables.

@cbabs

GRAYLOG_REST_TRANSPORT_URI: “htt p://0.0.0.0:12900”
GRAYLOG_REST_LISTEN_URI: “http: / /0.0.0.0:12900”
GRAYLOG_WEB_LISTEN_URI: “http: // 0.0.0.0:9000”