Graylog Monitoring - Additional Metrics (prometheus / grafana)

Graylog Central (peer support)
1

1. Describe your incident:

Dear graylog community, my dashboard in graylog showsthe latest logs with gl2_rempoe_ip, application name, timestamp,source (the host that send the log), loglevel etc… after filtering through pipelines and rules. I’d like to have this data exported in a form to use it with prometheus for alertig (e.g. if the number of logs of a host is increasing / reaching a general limit or by loglevel).

An additional metric could look like this - or similar - just to get an idea:

# HELP gl_host_logs counter for logs by reporting host
# TYPE gl_host_logs gauge
gl_host_logs{ip="192.168.0.5", host="hostname", loglevel="3", facility="kernel" } 42

2. Describe your environment:

  • OS Information: Debian 12, deployment via docker
  • Package Version: Graylog 6.0.5, MongoDB 6.0.14, Opensearch 2.12.0

3. What steps have you already taken to try and solve the problem?

  • Already enabled the built-in prometheus exporter in graylog.conf like
    • prometheus_exporter_enabled = true and
    • prometheus_exporter_bind_address = 0.0.0.0:9833
  • Enabled the ports in docker-compose.yml like 
  - job_name: 'graylog_exporter'
    metrics_path: /api/prometheus/metrics
    scrape_interval: 1m
    scrape_timeout: 1m
    file_sd_configs:
      - files:
        - /opt/monitoring/graylog_exporter/graylog_exporter_targets.yml
  • Where the graylog_exporter_targets.yml looks like
 - targets:
    - 'server.domain.de:9833'
  labels:
    category: graylog
  • But the provided logs do not fit my needs

Note: For now I used the Grafana Dashboard from asachs01 and modified it a little so it shows my values - seems to be still working, but only covering a small set of metrics provided by the built-in exporter.

4. How can the community help?

  • Is there a way in graylog to just make additional metrics / more information available in the build-in prometheus exporter? (I believe not)
  • If not, maybe there is a way to access the graylog api and write a script to extract the metrics myself - one could just setup a small webserver on another port to povide additional metrics - has anyone done that already / is willing to share some scripts?

Best regards,
justeverything

2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.