Recently I’ve been asked to look at the possibility of having a multi-tenant elasticsearch cluster serving multiple graylog clusters.
I’ve read the ES documentation on security and created a dedicated ES user (
graylogDevNative ) for a non production graylog cluster to connect with.
This user has the
graylogDevAdmin role, which I have created with full permissions on indices prefixed with
graylogdev_, configured via ES API.
I’ve observed the following behaviour while starting graylog for the first time with user
- default index set graylogdev_default contains graylogdev_default_deflector as an index
- searching does not work
cannot find index graylog_default_0
- I get the warning
The deflector exists as an index and is not an alias
- Creating a new index set results in the same behaviour as above.
- Rotating the indices does not solve the problem
- Previously creating aliases and indices manually does not solve the problem
However bootstraping a new graylog with clean mongodb and clean ES with
elastic superuser I observe the following:
- default index set graylogdev_default contains graylogdev_default_0
- everything works as expected
Changing the connection user after bootstraping graylog with
- index rotation works normally
- searching works
- creating a new index set results in correct behaviour creating indices with a numeric suffix
I suspected it was an authorization issue until changing the user after the bootstrap completely changed my opinion.
Can anyone provide a clue as to why this is happening?
if needed I can provide a debug graylog bootstraping
server.log with each user, as well as bootstraping with
elastic and switching to
graylogDevNative after the bootstrap.
I can also provide the elasticsearch permissions for
graylogDevNative and demonstrate index creation and deletion via API.
A few relevant entries in log files:
2019-07-26T09:44:40.461Z ERROR [IndexRotationThread] Couldn’t point deflector to a new index
2019-07-26T10:31:38.969Z INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-07-26T10:31:38.976Z INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2019-07-26T10:31:38.987Z INFO [MongoIndexSet] Cycling from to <graylogdev_default_0>.
2019-07-26T10:31:38.988Z INFO [MongoIndexSet] Creating target index <graylogdev_default_0>.
2019-07-26T10:31:39.074Z INFO [Indices] Successfully created index template graylog-internal
switching to graylogDevNative and creating a new index set:>
2019-07-26T10:43:37.737Z WARN [IndexFieldTypePollerPeriodical] Active write index for index set “teste” (5d3ad954db12b8634692707f) doesn’t exist yet
2019-07-26T10:43:40.763Z INFO [MongoIndexSet] Did not find a deflector alias. Setting one up now.
2019-07-26T10:43:40.767Z INFO [MongoIndexSet] There is no index target to point to. Creating one now.
2019-07-26T10:43:40.776Z INFO [MongoIndexSet] Cycling from to <graylogdev_teste_0>.
2019-07-26T10:43:40.777Z INFO [MongoIndexSet] Creating target index <graylogdev_teste_0>.
2019-07-26T10:43:40.816Z INFO [Indices] Successfully created index template graylogdev_teste-template
2019-07-26T10:43:40.998Z INFO [MongoIndexSet] Waiting for allocation of index <graylogdev_teste_0>.
2019-07-26T10:43:41.008Z INFO [MongoIndexSet] Index <graylogdev_teste_0> has been successfully allocated.
2019-07-26T10:43:41.009Z INFO [MongoIndexSet] Pointing index alias <graylogdev_teste_deflector> to new index <graylogdev_teste_0>.
2019-07-26T10:43:41.039Z INFO [MongoIndexSet] Successfully pointed index alias <graylogdev_teste_deflector> to index <graylogdev_teste_0>.