1. Describe your incident:
I can’t get a proper TLS connection between Graylog and Opensearch to run.
If I check with openssl s_client the connection seems to have no problems at all!
2. Describe your environment:
- OS Information:
OS: SLES 15 SP5 (for both Opensearch and Graylog)
Package Version:
Graylog: graylog-server 5.0.3-1
Opensearch: opensearch 2.5.0-1
- Service logs, configurations, and environment variables:
graylog logs:
2024-01-26T14:55:40.813+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #128
2024-01-26T14:55:45.817+01:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://10.252.18.78:9200/… - \n not found: limit=0 content=….
2024-01-26T14:55:45.817+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #129
2024-01-26T14:55:50.820+01:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://10.252.18.78:9200/… - \n not found: limit=0 content=….
2024-01-26T14:55:50.821+01:00 INFO [VersionProbe] Elasticsearch is not available. Retry #130
opensearch logs:
[2024-01-26T14:56:30,853][ERROR][o.o.s.a.s.InternalOpenSearchSink] [XXX] Unable to index audit log {“audit_node_id”:“irjEwfHSQXKOqPYchU8p7Q”,“audit_request_layer”:“REST”,“audit_request_exception_stackt
race”:"io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
Config Opensearch
action.auto_create_index: false
discovery.type: single-node
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: xxx.xxx.xxx.xxx
http.port: 9200
plugins.security.disabled: false
plugins.security.ssl.transport.pemcert_filepath: xxx.y.z.crt.pem #esnode.pem
plugins.security.ssl.transport.pemkey_filepath: xxx.y.z.key.pem #esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: my-ca.pem #root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath:xxx.crt.pem #esnode.pem
plugins.security.ssl.http.pemkey_filepath: xxx.key.pem # esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: my-ca.pem #root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3
config graylog
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = Etc/GMT-1
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = xxx.y.z:9000
http_enable_tls = true
http_tls_cert_file = /etc/ssl/private/xxx.y.z.crt.pem
http_tls_key_file = /etc/ssl/private/xxx.y.z.key.pem
stream_aware_field_types=false
elasticsearch_hosts = xxx.xxx.xxx.xxx:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_disable_version_check = true
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
3. What steps have you already taken to try and solve the problem?
I’ve ensured everything works fine without SSL enabled and then just added a proper certificate and CA file to the opensearch.yml file and also added the CA file to the jks that Graylog uses. Additionally I added the a certificate + key for Graylog to the truststore, in case that was a problem.
Note that the Logs that are sent to graylog are encrypted properly and there are no problems regarding that. The only unencrypted communication happens between Graylog and Opensearch.
The Graylog webinterface itself also flawlessly works via HTTPS.
4. How can the community help?
The install recommendation being add “plugins.security.disabled: false”
to the opensearch configuration does not actually mean that it’s not possible to use the plugin to enable TLS, right?