OpenSearch 1.3, no such index security-auditlog

1. Describe your incident:
Since an upgrade from Elasticseach 7.10 to OpenSearch 1.3, the logs of OpenSearch are flooded by this kind of error.
I have this on a simple install on a single server, without any licence.
But i have this same error on a test cluster with a free enterprise licence.

[2022-11-30T16:11:13,822][ERROR][o.o.s.a.s.InternalOpenSearchSink] [dirv-monitoring-graylog-01] Unable to index audit log {"audit_cluster_name":"graylog","audit_transport_headers":{"_system_index_access_allowed":"false"},"audit_node_name":"dirv-monitoring-graylog-01","audit_trace_task_id":"AUGaB-CCTdyXQd1KubJEJg:7191822","audit_transport_request_type":"GetMappingsRequest","audit_category":"INDEX_EVENT","audit_request_origin":"REST","audit_node_id":"AUGaB-CCTdyXQd1KubJEJg","audit_request_layer":"TRANSPORT","@timestamp":"2022-11-30T15:11:13.821+00:00","audit_format_version":4,"audit_request_remote_address":"127.0.0.1","audit_request_privilege":"indices:admin/mappings/get","audit_node_host_address":"10.128.10.10","audit_request_effective_user":"admin","audit_trace_indices":["graylog_1"],"audit_trace_resolved_indices":["graylog_1"],"audit_node_host_name":"10.128.10.10"} due to
org.opensearch.index.IndexNotFoundException: no such index [security-auditlog-2022.11.30]

2. Describe your environment:

  • OS Information: Rocky Linux release 8.7 (Green Obsidian)

  • Package Version:

$ yum list installed | grep -E ".*(opensearch|graylog|mongo).*"
graylog-4.3-repository.noarch            1-6                                              @@commandline
graylog-integrations-plugins.noarch      4.3.9-1                                          @graylog
graylog-server.noarch                    4.3.9-1                                          @graylog
mongodb-org.x86_64                       4.2.23-1.el8                                     @mongodb-org-4.2
mongodb-org-mongos.x86_64                4.2.23-1.el8                                     @mongodb-org-4.2
mongodb-org-server.x86_64                4.2.23-1.el8                                     @mongodb-org-4.2
mongodb-org-shell.x86_64                 4.2.23-1.el8                                     @mongodb-org-4.2
mongodb-org-tools.x86_64                 4.2.23-1.el8                                     @mongodb-org-4.2
opensearch.x86_64                        1.3.6-1                                          @opensearch-1.x
opensearch-dashboards.x86_64             1.3.6-1                                          @opensearch-dashboards-1.x

Did you know of to get rid of this ?

Hello @benoitp

To make the logs more readable

2022-11-30T16:11:13,822][ERROR][o.o.s.a.s.InternalOpenSearchSink] [dirv-monitoring-graylog-01]
 Unable to index audit log
 {"audit_cluster_name":"graylog",
 "audit_transport_headers":{"_system_index_access_allowed":"false"},
 "audit_node_name":"dirv-monitoring-graylog-01","audit_trace_task_id":"AUGaB-CCTdyXQd1KubJEJg:7191822",
 "audit_transport_request_type":"GetMappingsRequest","audit_category":"INDEX_EVENT",
 "audit_request_origin":"REST","audit_node_id":"AUGaB-CCTdyXQd1KubJEJg",
 "audit_request_layer":"TRANSPORT","@timestamp":"2022-11-30T15:11:13.821+00:00",
 "audit_format_version":4,"audit_request_remote_address":"127.0.0.1",
 "audit_request_privilege":"indices:admin/mappings/get",
 "audit_node_host_address":"10.128.10.10",
 "audit_request_effective_user":"admin",
 "audit_trace_indices":["graylog_1"],
 "audit_trace_resolved_indices":["graylog_1"],
 "audit_node_host_name":"10.128.10.10"} due to
org.opensearch.index.IndexNotFoundException: no such index [security-auditlog-2022.11.30]

By chance do you have Graylog License? If so I would ask support about it.

If not from what I get in the log is

no such index [security-auditlog-2022.11.30]

It seams you have something configured and Graylog can not find it.

Posting your OpenSearch configuration file would be appreciated.

Hey @benoitp
I did quick test on mine.
I believe those messages are coming from these settings in /etc/opensearch/opensearch.yml

plugins.security.disabled: false
plugins.security.system_indices.enabled: true

Need to be

plugins.security.disabled: true
plugins.security.system_indices.enabled: false

Restart Opensearch Service

Hello @gsmith

Yes I have a enterprise licence for my main setup, but was looking before asking them because it’s a test server.

Here is my full OpenSearch configuration, with auth. enabled because we need it.

# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: false
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=admin,OU=Monitoring,O=Yop,L=Jap,ST=Knoauf,C=FR'
plugins.security.nodes_dn:
  - 'CN=node,OU=Monitoring,O=Yop,L=Jap,ST=Knoauf,C=FR'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", 
".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", 
".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", 
".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]

cluster.name: graylog
action.auto_create_index: false
#plugins.security.disabled: true
network.host: [_site_, _local_]
discovery.type: single-node

Security audit logs are an OpenSearch feature, so it shouldn’t matter if you have a Graylog license or not.
I notice this: action.auto_create_index: false
Perhaps the audit log index could simply not be created.

1 Like
action.auto_create_index: false

Is required by Graylog : Installation Prerequisites

Edit : index rotation was blocked by a custom index template.

Update, I solved this !
It comes from an OpenSearch feature called «Audit logs»
The parameter who NEED to be commented (or not present) is this one :

plugins.security.audit.type: internal_opensearch

I think this parameter conflict with this one, required by Graylog :

action.auto_create_index: false

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.