Unusual behaviour while using elasticsearch non super user

[jpengenheiro]

Recently I’ve been asked to look at the possibility of having a multi-tenant elasticsearch cluster serving multiple graylog clusters.

I’ve read the ES documentation on security and created a dedicated ES user ( graylogDevNative ) for a non production graylog cluster to connect with.

This user has the graylogDevAdmin role, which I have created with full permissions on indices prefixed with graylogdev_, configured via ES API.

I’ve observed the following behaviour while starting graylog for the first time with user graylogDevNative:

1. default index set graylogdev_default contains graylogdev_default_deflector as an index
2. searching does not work cannot find index graylog_default_0
3. I get the warning The deflector exists as an index and is not an alias
4. Creating a new index set results in the same behaviour as above.
5. Rotating the indices does not solve the problem
6. Previously creating aliases and indices manually does not solve the problem

However bootstraping a new graylog with clean mongodb and clean ES with elastic superuser I observe the following:

1. default index set graylogdev_default contains graylogdev_default_0
2. everything works as expected

Changing the connection user after bootstraping graylog with elastic superuser:

1. index rotation works normally
2. searching works
3. creating a new index set results in correct behaviour creating indices with a numeric suffix

I suspected it was an authorization issue until changing the user after the bootstrap completely changed my opinion.

Can anyone provide a clue as to why this is happening?

if needed I can provide a debug graylog bootstraping server.log with each user, as well as bootstraping with elastic and switching to graylogDevNative after the bootstrap.

I can also provide the elasticsearch permissions for graylogDevNative and demonstrate index creation and deletion via API.

A few relevant entries in log files:

bootstraping as graylogDevNative:

2019-07-26T09:44:40.461Z ERROR [IndexRotationThread] Couldn’t point deflector to a new index

bootstraping as elastic:

2019-07-26T10:31:38.969Z INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-07-26T10:31:38.976Z INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2019-07-26T10:31:38.987Z INFO [MongoIndexSet] Cycling from to <graylogdev_default_0>.
2019-07-26T10:31:38.988Z INFO [MongoIndexSet] Creating target index <graylogdev_default_0>.
2019-07-26T10:31:39.074Z INFO [Indices] Successfully created index template graylog-internal

switching to graylogDevNative and creating a new index set:>

2019-07-26T10:43:37.737Z WARN [IndexFieldTypePollerPeriodical] Active write index for index set “teste” (5d3ad954db12b8634692707f) doesn’t exist yet
2019-07-26T10:43:40.763Z INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-07-26T10:43:40.767Z INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2019-07-26T10:43:40.776Z INFO [MongoIndexSet] Cycling from to <graylogdev_teste_0>.
2019-07-26T10:43:40.777Z INFO [MongoIndexSet] Creating target index <graylogdev_teste_0>.
2019-07-26T10:43:40.816Z INFO [Indices] Successfully created index template graylogdev_teste-template
2019-07-26T10:43:40.998Z INFO [MongoIndexSet] Waiting for allocation of index <graylogdev_teste_0>.
2019-07-26T10:43:41.008Z INFO [MongoIndexSet] Index <graylogdev_teste_0> has been successfully allocated.
2019-07-26T10:43:41.009Z INFO [MongoIndexSet] Pointing index alias <graylogdev_teste_deflector> to new index <graylogdev_teste_0>.
2019-07-26T10:43:41.039Z INFO [MongoIndexSet] Successfully pointed index alias <graylogdev_teste_deflector> to index <graylogdev_teste_0>.

[jan (Jan Doberstein)]

The question is - what Elasticsearch Version did you use? It might be that Graylog needs to change some API Usage of Elasticsearch to be compatible with different users - but we are still not compatible with ES7 so it might be that with the added ability to work with ES7 this will work too.

It would be nice if you create a github feature request for that - means working with restricted roles in the new available Elasticsearch User interface.

Please do this here: https://github.com/Graylog2/graylog2-server/issues with as much information you already have and what you have done exactly that the developers are able to look at this quickly. Please include links to how-tos or documentation you had followed also.

[jpengenheiro]

First I would like to rule out any misconfiguration on my part.

The elasticsearch version is 6.8 running on LXD containers.

These are the privileges defined in the graylogDevAdmin role for the graylogDevNative user:

"graylogDevAdmin": {
    "applications": [],
    "cluster": [
        "all"
    ],
    "indices": [
        {
            "allow_restricted_indices": false,
            "names": [
                "graylogdev_*"
            ],
            "privileges": [
                "all"
            ]
        }
    ],
    "metadata": {
        "comment": "role test for user graylogDevNative",
        "version": 1
    },
    "run_as": [],
    "transient_metadata": {
        "enabled": true
    }
}

I expect not all the cluster level permissions are strictly necessary.

The bootstrap logs were too lengthy to include in this post, if they are necessary I’ll post them somewhere else.

If this is indeed unexpected behaviour I’ll do the feature request.

[jan (Jan Doberstein)]

I do not know that rights are needed on what level. But to make the developers aware of that creating a github issue with the above data would be the best solution.

[jpengenheiro]

I’ve confirmed that I can only observe this behaviour when bootstraping graylog with a user that has been restricted to that specific prefix.

Graylog fails to create the graylog-internal template, and creates a graylogdev_default_deflector index,

Switching to the unprivileged user after bootstraping results in seemingly correct operation, and new templates for new index sets are correctly created.

Even without full knowledge of graylog’s ES authorization requirements I would call this a bug.
I will present the case on github and create a feature request.