Consider using labels for zonestats in Prometheus metrics

[ruuda]

In NSD 4.12.0, when you enable Prometheus metrics with metrics-enable: yes, and zonestats for say example.com and example.org with zonestats: "example_com" and "example_org" respectively, NSD will expose metrics in this format:

nsd_queries_by_type_total{type="A"} 198
nsd_queries_by_type_total{type="NS"} 198
...
nsd_zonestats_example_com_queries_by_type_total{type="A"} 99
nsd_zonestats_example_com_queries_by_type_total{type="NS"} 99
...
nsd_zonestats_example_org_queries_by_type_total{type="A"} 99
nsd_zonestats_example_org_queries_by_type_total{type="NS"} 99
...

The following format would be more useful:

nsd_zonestats_queries_by_type_total{type="A",zone="example.com"} 99
nsd_zonestats_queries_by_type_total{type="NS",zone="example.com"} 99
...
nsd_zonestats_queries_by_type_total{type="A",zone="example.org"} 99
nsd_zonestats_queries_by_type_total{type="NS",zone="example.org"} 99
...

In other words, put the zone in a label rather than in the metric name.

Advantages

• You can use any zone name, it doesn’t need to be a valid Prometheus identifier. In particular, it may contain a dot.
• It unlocks Prometheus features that work over metrics with labels but not over separate metrics:
  • For example, we can select a subset of zones by regex, e.g. nsd_zonestats_queries_by_type_total{zone =~ ".+\.nl"}.
  • Functions like sum, avg, etc. become available.
• It makes it easier to graph metrics.
  • For example, you can query rate(nsd_zonestats_queries_by_type_total{type="A"}[5m]) in the Prometheus webui, and it will graph the rates for all zones in a single graph.
  • This also applies to Grafana, you can use the zone label as series name, and then with a single query Grafana will all zones in a single graph. It is possible to achieve the same result with differently named metrics, but it requires manually duplicating the query n_zones times, and it requires manually updating the dashboard when zones are added or removed.
• Similar to graphing all zones at once, it is possible to create alerts for all zones at once, and use the zone label in the alert description.
• It doesn’t pollute Prometheus with O(n_zones) additional metrics. This matters for example in the Prometheus webui, or in graphing tools like Grafana, which autocomplete metric names, which is a valuable for exploring available metrics.

Impact

• For Prometheus storage it doesn’t matter, internally Prometheus stores every unique combination of labels as a separate metric so the number of timeseries doesn’t change.
• Queries change from nsd_zonestats_example_com_queries_by_type_total to nsd_zonestats_queries_by_type_total{zone="example.com"} which is slightly longer, but on the positive side, the Prometheus webui has autocomplete for metrics, labels, and label values.
• This is a breaking change for people who rely on the current zonestats format.

@mozzieongit, I see you implemented Prometheus metrics originally in #429 (thanks for this feature!), what do you think?

[mozzieongit]

That sounds like a good idea.

I'm trying to think back to why I implemented it this way... The only reason I can think of was that I probably mirrored what nsd-control stats printed.

So yeah, that would be a great improvement.

[mozzieongit]

Oh, I remember why I implemented it this way:

A metric must be summable to get the actual total amount across all labels. However, nsd_queries_by_type_total{type="A"} 99 and nsd_zonestats_example_org_queries_by_type_total{type="A"} 99 both count the same queries, so turning it into nsd_queries_by_type_total{type="A"} 99 and nsd_zonestats_queries_by_type_total{type="A",zone="example.com"} 99 would result in the sum of nsd_queries_by_type_total 198.

So the change would need to make sure to subtract the counters of a particular zone from the normal counter (nsd_queries_by_type_total). But not all zones need to be configured for zonestats, therefore only some would be subtracted. So the change would need to track which zones have zonestats available and subtract those from the normal total to be able to expose the zonestats counter as a label of nsd_queries_by_type_total.

Additional info about how NSD stores the stats internally. There is a "normal" counter counting all the queries, and then, if there is zonestats configured for one or more zones, there is an additional counter that counts the packets for the particular zonestats item. Also, multiple zones could have the same zonestats ID. So example.org and example.com could both be counted with the same zonestats: examplezones counter.

[ruuda]

I put the number 99 in my example above just to illustrate the format, my example doesn’t sum to the right totals, apologies for the confusion. Edit: Updated the example to have consistent totals.

What I am proposing is to change how metrics are output, basically from nsd_zonestats_%s_queries_by_type_total{type="%s"} into nsd_zonestats_queries_by_type_total{type="%s",zone="%s"}. Nothing about the numbers themselves needs to change.

[ruuda]

If you like, I can try to make a pull request?

[mozzieongit]

What I am proposing is to change how metrics are output, basically from nsd_zonestats_%s_queries_by_type_total{type="%s"} into nsd_zonestats_queries_by_type_total{type="%s",zone="%s"}.

Oh wow... I overlooked that the metric still contained nsd_zonestats_ in the front 🤦

I was thinking about merging the zonestats into the "normal" counter. But yeah, your approach is relatively easy to implement.

If you like, I can try to make a pull request?

Certainly. Go for it.

The relevant code locations in case you haven't looked at it already:

The zonestats total is created here, and the other metrics re-use the common print_stat_block that is both used for "normal" metrics and zonestats. You'll have to change how the zonestats name is embedded in the strings in the print_stat_block function.