Give indices:admin/template/put permission to user datanode

[mcury]

1. Describe your incident:
I want to give indices:admin/template/put permission to user datanode, or any other user, I just need to be able to apply a template as per below:

curl -X PUT --key mykey.key --cert mycert.crt --ca myca.crt "myhost:9200/_template/graylog_template" -H 'Content-Type: application/json' -d '{
  "index_patterns": ["graylog_*"],
  "settings": {
    "index.translog.flush_threshold_size": "2gb",
    "index.merge.scheduler.max_thread_count": 1,
    "index.translog.durability": "async",
    "index.translog.sync_interval": "10s",
    "index.refresh_interval": "30s",
    "index.merge.policy.max_merge_at_once": 5
  }
}'

2. Describe your environment:

• OS Information: Raspberry PI 5 8GB running Raspbian OS 64 bits (Debian 12)

• Package Version: graylog-server:arm64 (6.2), graylog-datanode:arm64 (6.2)

• Error when trying to
  {“error”:{“root_cause”:[{“type”:“security_exception”,“reason”:“no permissions for [indices:admin/template/put] and User [name=datanode, backend_roles=, requestedTenant=null]”}],“type”:“security_exception”,“reason”:“no permissions for [indices:admin/template/put] and User [name=datanode, backend_roles=, requestedTenant=null]”},“status”:403}

3. What steps have you already taken to try and solve the problem?
I’ve been trying to include the permission indices:admin/template/put in roles.yml.
Tried to create a test user and assign this role to it, but I’m probably not doing it correctly.
Tried to change the user admin password using hash.sh and changing in internal_users.yml, also no joy.

4. How can the community help?
If someone can point me in the right direction, that would be great.

Thanks !

[Tdvorak (Tomas Dvorak)]

Hi @mcury,
Every change you do to configuration files will be regenerated and overridden during the next datanode startup. Additionally, there is no basic auth enabled for the opensearch in the datanode.

The easiest approach would be to let graylog server generate you a bundle of client certificates and configure roles: Manage Certificates with Data Node.

With the user, role and certs, you should be able to trigger the request.

Best regards,
Tomas

[matthias_gl (Matthias)]

Hi @mcury ,
looks like you are already using the generated certificate. These are by default client certificates. In order to perform admin tasks, you will need to “elevate” one of these certificates to an admin certificate. These are certificates for which the dn is specified in opensearch.yml in the plugins.security.authcz.admin_dn.
Since opensearch.yml is auto-generated on startup in datanode, you will need to set this in a overrides file: Data Node Configuration Overrides

If you need additional help, feel free to reach out.

Matthias

[mcury]

Thanks both of you for answering.
Note: Only created admin certificate, not a node one.

Following instructions from: https://docs.opensearch.org/docs/2.17/security/configuration/generate-certificates/#generate-an-admin-certificate

Actions taken so far:

Generate a private key

openssl genrsa -out root-ca-key.pem 2048

Generate a root certificate

openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem -days 730

Generate an admin certificate

1. openssl genrsa -out admin-key-temp.pem 2048
2. openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
3. openssl req -new -key admin-key.pem -out admin.csr
4. openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730

Left everything to default but the Common Name.

Then, ran the script Sample script to convert PEM certificates to keystore and truststore files
Got the following output: (I just ignored what is related to nodes certificate).

sudo ./script.sh
cat: node1.pem: No such file or directory
cat: node1-key.pem: No such file or directory
Enter password for node1-cert.p12
Could not read private key from -in file from combined-node1.pem
Enter password for keystore.jks
Importing keystore node1-cert.p12 to keystore.jks...
Enter destination keystore password:
Re-enter new password:
keytool error: java.io.FileNotFoundException: node1-cert.p12 (No such file or directory)
Enter password for admin-cert.p12
Enter Export Password:
Verifying - Enter Export Password:
4040CD97FF7F0000:error:05800074:x509 certificate routines:X509_check_private_key:key values mismatch:../crypto/x509/x509_cmp.c:408:
Enter password for keystore.jks
Importing keystore admin-cert.p12 to keystore.jks...
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias admin successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
keytool error: java.io.FileNotFoundException: root-ca.cer (No such file or directory)

As you can see above, it successfully imported admin-cert.p12 to keystore.jks, and I think it is ok to ignore node’s certificate errors since I won’t be using them.

Converted the format to RFC2253:
openssl x509 -subject -nameopt RFC2253 -noout -in node.pem and updated /etc/graylog/datanode/overrideconfig.conf

plugins.security.authcz.admin_dn:
  - 'CN=graylog.home.arpa,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU'

Restarted graylog-datanode.service, the service is starting but I can see that Graylog is saying in the GUI that is not able to communicate with “ElasticSearch”.

And when I try to run the curl -X put command mentioned in the opening of this thread, I get:
curl: (7) Failed to connect to graylog.home.arpa port 9200 after 0 ms: Couldn't connect to server

Checking cat /var/log/graylog-datanode/opensearch/datanode-cluster.log
I believe that the problem is related to security plugin not loaded:

java.lang.IllegalArgumentException: unknown setting [-] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Sorry for the long post, I tried to give as much details as possible.
Thanks.

[Tdvorak (Tomas Dvorak)]

The whole procedure seems unnecessarily complicated. You are already using a data node. Data node is secured with a CA (probably self-signed, if you followed the installation/preflight wizard). All the certificates will be generated for you on one click, if you will follow our documentation. All the cert chains will be correct, the truststore will be configured, everything correctly propagated to the underlying opensearch.

The only manual step needed will be the plugins.security.authcz.admin_dn configuration in the overrides.

Or is there any special requirement why are you trying to generate and configure your own set of certificates?

[mcury]

@Tdvorak

plugins.security.authcz.admin_dn

Well, that was much much much easier than I initially thought.
Really thanks Tomas Dvorak.

openssl x509 -in cert_mycert.crt -noout -subject
subject=CN = datanode

cat /etc/graylog/datanode/overrideconfig.conf
plugins.security.authcz.admin_dn = CN = datanode

cat /etc/graylog/datanode/datanode.conf | grep overri
opensearch_configuration_overrides_file = /etc/graylog/datanode/overrideconfig.conf

Then, restarted graylog-datanode and the CURL PUT command worked.
I’ll be doing some tests now.

 curl -X PUT --key mykey.crt --cert mycert.crt --cacert myca.crt https://rpi5.home.arpa:9200/_template/graylog_template -H 'Content-Type: application/json' -d '{
  "index_patterns": ["ipfix_*"],
  "settings": {
    "index.translog.flush_threshold_size": "2gb",
    "index.merge.scheduler.max_thread_count": 1,
    "index.translog.durability": "async",
    "index.translog.sync_interval": "10s",
    "index.refresh_interval": "30s",
    "index.merge.policy.max_merge_at_once": 5
  }
}'
{"acknowledged":true}

[Tdvorak (Tomas Dvorak)]

Nice to hear that it worked for you! Thanks for letting us know and leaving some hints for any future reference!