Graylog TLS to Opensearch instance

1. Describe your incident:

I can’t get a proper TLS connection between Graylog and Opensearch to run.
If I check with openssl s_client the connection seems to have no problems at all!

2. Describe your environment:

  • OS Information:

OS: SLES 15 SP5 (for both Opensearch and Graylog)

Package Version:
Graylog: graylog-server 5.0.3-1
Opensearch: opensearch 2.5.0-1

  • Service logs, configurations, and environment variables:

graylog logs:
2024-01-26T14:55:40.813+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #128
2024-01-26T14:55:45.817+01:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on… - \n not found: limit=0 content=….
2024-01-26T14:55:45.817+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #129
2024-01-26T14:55:50.820+01:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on… - \n not found: limit=0 content=….
2024-01-26T14:55:50.821+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #130

opensearch logs:
[2024-01-26T14:56:30,853][ERROR][o.o.s.a.s.InternalOpenSearchSink] [XXX] Unable to index audit log {“audit_node_id”:“irjEwfHSQXKOqPYchU8p7Q”,“audit_request_layer”:“REST”,“audit_request_exception_stackt
race”:"io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

Config Opensearch
action.auto_create_index: false
discovery.type: single-node /var/lib/opensearch
path.logs: /var/log/opensearch
http.port: 9200 false xxx.y.z.crt.pem #esnode.pem xxx.y.z.key.pem #esnode-key.pem my-ca.pem #root-ca.pem false true #esnode.pem xxx.key.pem # esnode-key.pem my-ca.pem #root-ca.pem true true

  • CN=kirk,OU=client,O=client,L=test, C=de internal_opensearch true true [“all_access”, “security_rest_api_access”] true [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

config graylog
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = Etc/GMT-1
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = xxx.y.z:9000
http_enable_tls = true
http_tls_cert_file = /etc/ssl/private/xxx.y.z.crt.pem
http_tls_key_file = /etc/ssl/private/xxx.y.z.key.pem
elasticsearch_hosts =
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_disable_version_check = true
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000

3. What steps have you already taken to try and solve the problem?

I’ve ensured everything works fine without SSL enabled and then just added a proper certificate and CA file to the opensearch.yml file and also added the CA file to the jks that Graylog uses. Additionally I added the a certificate + key for Graylog to the truststore, in case that was a problem.
Note that the Logs that are sent to graylog are encrypted properly and there are no problems regarding that. The only unencrypted communication happens between Graylog and Opensearch.
The Graylog webinterface itself also flawlessly works via HTTPS.

4. How can the community help?

The install recommendation being add “ false”
to the opensearch configuration does not actually mean that it’s not possible to use the plugin to enable TLS, right?

I kept overlooking that I never changed this setting:
elasticsearch_hosts = http://…:9200
now when I switched it to https:// I can confirm that it validates the host correctly by comparing using IP vs DNS name. But I still get an error that I struggle to decipher properly:

Unable to retrieve version from Elasticsearch node opensearch-server:9200: unknown error - an exception occurred while deserializing error response: com.fasterxml.jackson.core.JsonParseException: Unrecognized token ‘Unauthorized’: was expecting (JSON String, Number, Array, Object or token ‘null’, ‘true’ or ‘false’)

Do I have to adjust anything with the roles now that the security plugin is enabled or how do I deal with this apparent mismatch of json format?

seems to work now, not sure if i changed anything else than elasticsearch_host = https://username:password@opensearch_host:9200

Indeed to get Graylog working with OpenSearch in TLS the right format is https://user:password@host:9200

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.