Adding include filter to datdog metric sends all metrics and seems to make no difference

kishorenc · August 21, 2018, 12:25pm

Hi,

We’re currently using the Datadog metrics reporter. It’s working fine but we wanted to filter the number of metrics it sends to Graylog. I tried doing that by adding the following line to Graylog config:

metrics_datadog_include_metrics = org.graylog2.shared.journal.KafkaJournal.uncommittedMessages

However, when I added this line and restarted the Graylog server, the filtering did not kick in. All Graylog metrics were sent to Datadog, while I expected only uncommittedMessages to be sent. I tried this one one more host with the same result.

I’ve only 2 other metrics related configuration:

metrics_datadog_enabled = true
metrics_datadog_transport = UDP

I am using the follwing JAR which does seem to contain the code required for metric filtering: https://github.com/graylog-labs/graylog-plugin-metrics-reporter/releases/download/2.4.0-beta.3/metrics-reporter-datadog-2.4.0-beta.3.jar

Any input on what could be going wrong is greatly appreciated.

Regards,

Kishore.

jan · August 21, 2018, 12:46pm

the metric you are looking for is org.graylog2.journal.entries-uncommitted on each node to get the nodes journal.

kishorenc · August 21, 2018, 2:22pm

@jan I see the org.graylog2.shared.journal.KafkaJournal.uncommittedMessages metric being reported to Datadog, and it’s defined here in code as well:

github.com

Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/shared/journal/KafkaJournal.java#L182


this.throttleThresholdPercentage = intRange(throttleThresholdPercentage, 0, 100);
this.serverStatus = serverStatus;
this.maxSegmentSize = segmentSize.toBytes();
// Max message size should not be bigger than max segment size.
this.maxMessageSize = Ints.saturatedCast(maxSegmentSize);


this.writtenMessages = metricRegistry.meter(name(this.getClass(), "writtenMessages"));
this.readMessages = metricRegistry.meter(name(this.getClass(), "readMessages"));
this.writeDiscardedMessages = metricRegistry.meter(name(this.getClass(), "writeDiscardedMessages"));


registerUncommittedGauge(metricRegistry, name(this.getClass(), "uncommittedMessages"));


// the registerHdrTimer helper doesn't throw on existing metrics
this.writeTime = registerHdrTimer(metricRegistry, name(this.getClass(), "writeTime"));
this.readTime = registerHdrTimer(metricRegistry, name(this.getClass(), "readTime"));


final Map<String, Object> config = ImmutableMap.<String, Object>builder()
        // segmentSize: The soft maximum for the size of a segment file in the log
        .put(LogConfig.SegmentBytesProp(), Ints.saturatedCast(segmentSize.toBytes()))
        // segmentMs: The soft maximum on the amount of time before a new log segment is rolled
        .put(LogConfig.SegmentMsProp(), segmentAge.getMillis())

In any case, I just tried using the configuration you suggested, but again, all other metrics are being reported as well. Basically, the filtering does not seem to be working at all.

jan · August 21, 2018, 3:39pm

This might be a bug - does it happen too if you select two values and not just one? Or if you select org.graylog2.shared.*

Would be nice to get a detailed bug report over at: https://github.com/graylog-labs/graylog-plugin-metrics-reporter/issues

kishorenc · August 21, 2018, 4:22pm

Thanks, I’ve posted an issue since I observe the same problem with a filter involving a wildcard too:

https://github.com/graylog-labs/graylog-plugin-metrics-reporter/issues/30

system · September 4, 2018, 4:22pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.