Problem in Connection Between Graylog and Wazuh Indexer (Opensearch Fork)

Graylog Central (peer support)
1

Hello everyone,

I hope you’re all doing well. I am currently facing an issue in establishing a connection between Graylog and the Opensearch fork of the Wazuh Indexer.

I have a single node with the following components:

  • graylog-server : Version 4.3.8-1
  • mongodb-org-server : Version 4.4.17
  • wazuh-indexer : Version 4.3.8-1 (Opensearch fork 1.3)

The logs from Graylog indicate continuous failures when attempting to retrieve version information from the Elasticsearch node.

tail -f /var/log/graylog-server/server.log

ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: unexpected end of stream on http://192.168.40.131:9200/... - \n not >
INFO  [VersionProbe] Elasticsearch is not available. Retry #418

cat /etc/graylog/server/server.conf         | egrep -v "^\s*(#|$)"
elasticsearch_version = 7
is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = CwT9Mx1NmXc3dqPwhap3f83AvDRp561fJfB0EG6V8nBNI89LkHb9E68WrcRiTlLyiAKPcvR06AZNLLfv7vS9kknqD9IPTJM7
root_password_sha2 = c7ff6dcd94d7161eff5da0585684a8d16fb00090c0f38336d31950819e2f2003
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
elasticsearch_hosts = http://graylog:stage@192.168.40.131:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32


cat /etc/wazuh-indexer/opensearch.yml    | egrep -v "^\s*(#|$)"
network.host: "192.168.40.131"
node.name: "LAB-SOC"
cluster.initial_master_nodes:
- "LAB-SOC"
cluster.name: "lab-wazuh-cluster"
discovery.seed_hosts:
  - "192.168.40.131"
node.max_local_storage_nodes: "1"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer
bootstrap.memory_lock: true
plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/LAB-SOC.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/LAB-SOC-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/LAB-SOC.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/LAB-SOC-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
compatibility.override_main_response_version: true```

curl -XGET http://192.168.40.131:9200/_cluster/health?pretty=true
curl: (52) Empty reply from server

curl -XGET http://192.168.40.131:9200/_cluster/allocation/explain?pretty

curl: (52) Empty reply from server

2

Hey @50m1a

Think this might be your issue.

plugins.security.ssl.http.enabled: true
3

Thanks for your response. Disabling SSL in Graylog resolves the connection with Wazuh indexer, but causes issues with wazuh dashboard.

systemctl status wazuh-dahsboard
Active: active (running)
LAB-SOC opensearch-dashboards[516]: {"type":"log","@timestamp":"2024-02-19T11:07:42Z","tags":["error","opensearch","data"],"pid":516,"message":"[ConnectionError]: write EPROTO 139706927773504:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n"}

I suspect the root cause may be related to Graylog not recognizing the certificate when SSL is enabled, here is the graylog configuration

ls /etc/graylog/server/certs
cacerts  root-ca.pem


nano /etc/default/graylog-server
# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"

# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"

# Fix for log4j CVE-2021-44228
#GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true -Djavax.net.ssl.trustStore=/etc/graylog/server/certs/cacerts -Djavax.net.ssl.trustStor>

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
GRAYLOG_COMMAND_WRAPPER=""
4

Hey @50m1a

Yes you are correct. Someone a while back in this forum actually figured out how to connect Elasticseach and Graylog using HTTPS with Certificates. Not sure how they did it. If they used Cert’s from ES/OS to Graylog or the other way around.

Graylog to connect to Opensearch.
example

elasticsearch_hosts = https://node1:9200,https://user:password@node2:19200

Have you tried using your node.crt from Wazuh into graylog or Graylog Cert’s into Wazuh?