I get these error messages in log output (in opensearch 1.2.1 installed via helm chart on k8s):
[2021-12-14T15:07:57,149][ERROR][o.o.s.a.s.InternalOpenSearchSink] [opensearch-cluster-master-0] Unable to index audit log {"audit_cluster_name":"opensearch-cluster","audit_transport_headers":{"_system_index_access_allow
opensearch org.opensearch.index.IndexNotFoundException: no such index [security-auditlog-2021.12.14]
and
[2021-12-14T15:17:05,556][DEPRECATION][o.o.d.c.m.IndexNameExpressionResolver] [opensearch-cluster-master-0] this request accesses system indices: [.opendistro_security], but in a future major version, direct access to system indices will be prevented by default
I just installed it and ran graylog against it… I am now running graylog-4.2.3… previously I’ve run opensearch-1.0.4 and graylog-4.1.3 - and it was working. I’ve had to extend amount of fields allowed to 4000 though - so it maybe related to that? (got complaints about logs not being accepted - but could not find alerts in graylog about this - and which logs it was about)
opensearch is a fork of elaticsearch 7.10 - so it should be fine ?
The audit log index likely doesn’t exist unless you are using an enterprise liscence (enterprise only feature), so this error being printed is likely just a side effect of using opensearch. May I ask if functionality is in any way impaired?
We have an issue with field limt (1000) being reached - and we’ve increased field limit (which we see now was a bad idea and we have to find a way to identify the culprit log entries causing this - but thats unfortunately not something we’ve found any good way of doing.
But why is this being done by graylog on an open source installation of graylog? (no enterprise) - it logs a LOT in opensearch
Other than that - everything seems to work - also with latest opensearch with log4j fix.
Not sure if you saw this post about having 1000+ fields. You may be able to use some of the Elasticsearch commands in there to hunt down data in your OpenSearch. The short is it was Auditbeat that was causing the issue in the links story.
Some people have made the mistake of including the Enterprise tools in their install not sure what you are running but if it is Debian based you can use this command to find out:
Graylog manages the indexes in Elasticsearch for you, allowing you to rotate and reindex from the Graylog GUI, set rotation methods…among other things . Since there is an integration between the two, my personal pref would be to change from OpenSearch to Elastic. It’s not wrong, you may be able to get it to work now… but there may be revisions that in Graylog/Elastic or in Opensearch that will break things in the future and you will have no control of that - which would put you in the position of starting from scratch or converting to Elasticsearch. I am not familiar with Helm Chart or OpenSearch so it would be difficult for me to help you there.
As you know OpenSearch is merged form Open Distro for Elasticsearch. It uses Elasticsearch 7.10.2 & Kibana 7.10.2 for indexing and dashboards. As @tmacgbay explained
As for OpenSearch this also can be done through Index Management. I Have found OpenSearch is a little more difficult to correct issues and the learning curve is greater then Graylog. Graylog supplies simplicity and also has room to add/create to it system. Just an FYI by enabling Prometheus in Graylog configuration file and attaching it to Grafana its great to monitor all my GL servers in one spot. Since OpenSearch ( AWS) is renaming its packages things will break as @tmacgbay stated.
