1. Describe your incident:
I am unable to use a snapshot repo with Google Cloud Storage due to an authorization error and I’m trying to determine if there is more that needs to be set in Graylog or if it is a permissions issue with GCS.
2. Describe your environment:
-
OS Information: Ubuntu
-
Package Version: 6.2.2
-
Service logs, configurations, and environment variables:
The service account in Google has Storage Admin permissions, as well as general Owner permissions.
I set up a repo with the Graylog Data Node using the following API call:
sudo curl -XPUT "https://localhost:9200/_snapshot/gcloud-repo" --key /mnt/disks/graylog-data/certs/opensearchapi.key --cert /mnt/disks/graylog-data/certs/opensearchapi.crt --cacert /mnt/disks/graylog-data/certs/opensearchapica.crt -H 'Content-Type: application/json' -d' { "type": "gcs", "settings": { "bucket": "graylog-index-snapshots", "base_path": "/mnt/disks/graylog-data/gcloud-snapshots", "client": "default" } }'
I also tried setting the default user credentials using the following command:
sudo /usr/share/graylog-datanode/dist/opensearch-2.15.0-linux-x64/bin/opensearch-keystore add-file gcs.client.default.credentials_file /home/user/gcloudservice.json
then reloaded the secure settings:
curl -XPOST "https://localhost:9200/_nodes/reload_secure_settings" --key /mnt/disks/graylog-data/certs/opensearchapi.key --cert /mnt/disks/graylog-data/certs/opensearchapi.crt --cacert /mnt/disks/graylog-data/certs/opensearchapica.crt -H 'Content-Type: application/json'
When I try to make a backup to that repo, it doesn’t throw any errors, but the snapshot is never actually created:
sudo curl -XPUT "https://localhost:9200/_snapshot/gcloud-repo/graylog_9" --key /mnt/disks/graylog-data/certs/opensearchapi.key --cert /mnt/disks/graylog-data/certs/opensearchapi.crt --cacert /mnt/disks/graylog-data/certs/opensearchapica.crt -H 'Content-Type: application/json' -d' { "indices": "graylog_9", "ignore_unavailable": "true", "partial": true }'
output:
{"accepted":true}
sudo curl -XGET "https://localhost:9200/_snapshot/gcloud-repo/graylog_9" --key /mnt/disks/graylog-data/certs/opensearchapi.key --cert /mnt/disks/graylog-data/certs/opensearchapi.crt --cacert /mnt/disks/graylog-data/certs/opensearchapica.crt -H 'Content-Type: application/json'
output:
{"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[gcloud-repo:graylog_9] is missing"}],"type":"snapshot_missing_exception","reason":"[gcloud-repo:graylog_9] is missing"},"status":404}
And when I try to verify the repo, I get this:
sudo curl -XPOST "https://localhost:9200/_snapshot/gcloud-repo/_verify?timeout=0s&cluster_manager_timeout=50s" --key /mnt/disks/graylog-data/certs/opensearchapi.key --cert /mnt/disks/graylog-data/certs/opensearchapi.crt --cacert /mnt/disks/graylog-data/certs/opensearchapica.crt -H 'Content-Type: application/json'
output:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[gcloud-repo] path [][mnt][disks][graylog-data][gcloud-snapshots] is not accessible on cluster-manager node"}],"type":"repository_verification_exception","reason":"[gcloud-repo] path [][mnt][disks][graylog-data][gcloud-snapshots] is not accessible on cluster-manager node","caused_by":{"type":"storage_exception","reason":"403 Forbidden\nPOST https://storage.googleapis.com/upload/storage/v1/b/graylog-index-snapshots/o?ifGenerationMatch=0&projection=full&uploadType=multipart\n{\n \"error\": {\n \"code\": 403,\n \"message\": \"Provided scope(s) are not authorized\",\n \"errors\": [\n {\n \"message\": \"Provided scope(s) are not authorized\",\n \"domain\": \"global\",\n \"reason\": \"forbidden\"\n }\n ]\n }\n}\n","caused_by":{"type":"google_json_response_exception","reason":"403 Forbidden\nPOST https://storage.googleapis.com/upload/storage/v1/b/graylog-index-snapshots/o?ifGenerationMatch=0&projection=full&uploadType=multipart\n{\n \"error\": {\n \"code\": 403,\n \"message\": \"Provided scope(s) are not authorized\",\n \"errors\": [\n {\n \"message\": \"Provided scope(s) are not authorized\",\n \"domain\": \"global\",\n \"reason\": \"forbidden\"\n }\n ]\n }\n}\n"}}},"status":500}
3. What steps have you already taken to try and solve the problem?
I’ve modified and expanded the permissions of the Google service account, and I’ve also Google’d as well as searched this forum and reddit to find any reference of a gcs repo being used with Graylog. I’m honestly not even sure if this is possible with the current configuration options in Graylog. I’m not clear on how using the opensearch plugins actually changes Data Node/Graylog, or if there’s another way to try and test the credentials that I’m using.
4. How can the community help?
Any advice or tips would help at this point. I haven’t really seen any other posts about this, but if I’ve missed something then even just posting a link to that and calling me dense would be pretty awesome.