Prometheus returns 401 Unauthorized on k8s

Hi, I’m running Graylog 3.0.2 on k8s, installed using stable Graylog helm chart.
I want to scrape metrics from the Graylog pods using Prometheus on that same k8s cluster, so I’ve created a secret in k8s with graylog credentials and servicemonitor that selects graylog-web service
app.kubernetes.io/component: web
app.kubernetes.io/instance: graylog
and uses basicAuth option for authorization.
I’m able to see graylog pods among Prometheus targets, but all Endpoints marked as Down due to the error " server returned HTTP status 401 Unauthorized". I’ve doublechecked credentials in secret and was able to get metrics from one pod, using port-forwarding and curl using those credentials.

do you have the Graylog metric exporter installed? That is an additional plugin.

Yes, plugin is installed,. and I was able to get metrics from that plugin’s metrics endpoint after port-forwarding from one of the graylog pods using
curl --user user:pass http://localhost:9000/api/plugins/org.graylog.plugins.metrics.prometheus/metrics
I have such services in graylog’s namespace created by helm-chart:

  • graylog-master
  • graylog-tcp
  • graylog-web
    I’ve configured servicemonitor to select graylog-web or should I use graylog-master for this?
    UPD: graylog-master also doesn’t work.

I do not know the kubernetes helm chart.

Found this issue for grayog-plugin-metrics-reporter plugin