1. Describe your incident:
Since an upgrade from Elasticseach 7.10 to OpenSearch 1.3, the logs of OpenSearch are flooded by this kind of error.
I have this on a simple install on a single server, without any licence.
But i have this same error on a test cluster with a free enterprise licence.
[2022-11-30T16:11:13,822][ERROR][o.o.s.a.s.InternalOpenSearchSink] [dirv-monitoring-graylog-01] Unable to index audit log {"audit_cluster_name":"graylog","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"dirv-monitoring-graylog-01","audit_trace_task_id":"AUGaB-CCTdyXQd1KubJEJg:7191822","audit_transport_request_type":"GetMappingsRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_node_id":"AUGaB-CCTdyXQd1KubJEJg","audit_request_layer":"TRANSPORT","@timestamp":"2022-11-30T15:11:13.821+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/mappings/get","audit_node_host_address":"10.128.10.10","audit_request_effective_user":"admin","audit_trace_indices":["graylog_1"],"audit_trace_resolved_indices":["graylog_1"],"audit_node_host_name":"10.128.10.10"} due to
org.opensearch.index.IndexNotFoundException: no such index [security-auditlog-2022.11.30]
2. Describe your environment:
OS Information: Rocky Linux release 8.7 (Green Obsidian)
2022-11-30T16:11:13,822][ERROR][o.o.s.a.s.InternalOpenSearchSink] [dirv-monitoring-graylog-01]
Unable to index audit log
{"audit_cluster_name":"graylog",
"audit_transport_headers":{"_system_index_access_allowed":"false"},
"audit_node_name":"dirv-monitoring-graylog-01","audit_trace_task_id":"AUGaB-CCTdyXQd1KubJEJg:7191822",
"audit_transport_request_type":"GetMappingsRequest","audit_category":"INDEX_EVENT",
"audit_request_origin":"REST","audit_node_id":"AUGaB-CCTdyXQd1KubJEJg",
"audit_request_layer":"TRANSPORT","@timestamp":"2022-11-30T15:11:13.821+00:00",
"audit_format_version":4,"audit_request_remote_address":"127.0.0.1",
"audit_request_privilege":"indices:admin/mappings/get",
"audit_node_host_address":"10.128.10.10",
"audit_request_effective_user":"admin",
"audit_trace_indices":["graylog_1"],
"audit_trace_resolved_indices":["graylog_1"],
"audit_node_host_name":"10.128.10.10"} due to
org.opensearch.index.IndexNotFoundException: no such index [security-auditlog-2022.11.30]
By chance do you have Graylog License? If so I would ask support about it.
If not from what I get in the log is
no such index [security-auditlog-2022.11.30]
It seams you have something configured and Graylog can not find it.
Posting your OpenSearch configuration file would be appreciated.
Yes I have a enterprise licence for my main setup, but was looking before asking them because it’s a test server.
Here is my full OpenSearch configuration, with auth. enabled because we need it.
# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: false
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
- 'CN=admin,OU=Monitoring,O=Yop,L=Jap,ST=Knoauf,C=FR'
plugins.security.nodes_dn:
- 'CN=node,OU=Monitoring,O=Yop,L=Jap,ST=Knoauf,C=FR'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*",
".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints",
".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*",
".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
cluster.name: graylog
action.auto_create_index: false
#plugins.security.disabled: true
network.host: [_site_, _local_]
discovery.type: single-node
Security audit logs are an OpenSearch feature, so it shouldn’t matter if you have a Graylog license or not.
I notice this: action.auto_create_index: false
Perhaps the audit log index could simply not be created.