resource-state-metrics

Summary

resource-state-metrics is a Kubernetes controller that builds on Kube-State-Metrics' Custom Resource State's ideology and generates metrics for custom resources based on the configuration specified in its managed resource, ResourceMetricsMonitor.

Development

Start developing by following these steps:

• Set up dependencies with make setup.
• Deploy the controller with make local.
• Test out your changes with make apply_testdata.
  • Telemetry metrics, by default, are exposed at :9998/metrics.
  • Resource metrics, by default, are exposed at :9999/metrics.
• Start an interactive pprof session with make pprof.

For more details, take a look at the Makefile targets.

Notes

• Garbage in, garbage out: Invalid configurations will generate invalid metrics. The exception to this being that certain checks that ensure metric structure are still present (for e.g., value should be a float64).
• Library support: The module is never intended to be used as a library, and as such, does not export any functions or types, with pkg/ being an exception (for managed types and such).
• Metrics stability: There are no metrics stability guarantees, as the metrics are user-generated.
• No middle-ware: The configuration is unmarshalled into a set of stores that the codebase directly operates on. There is no middle-ware that processes the configuration before it is used, in order to avoid unnecessary complexity. However, the expression(s) within the value and labelValues may need to be evaluated before being used, and as such, are exceptions.
• Metric configurations only scale horizontally, i.e., one metric configuration cannot end up generating multiple metrics. Please define collectors for complex cases. Multiple-metrics may be generated from a query resolution that targets a composite data structure. This also allows for recursively generating metrics for nested data structures as well.
• Non-turning-complete languages cannot express all possible metrics. For such cases, consider using a collector (/external). Such metrics are exposed through the /external endpoint of the "main" instance and defined in ./external.
• The managed resource, ResourceMetricsMonitor is namespace-scoped, but, to keep in accordance with KubeStateMetrics' CustomResourceState, which allows for collecting metrics from cluster-wide resources, it is possible to omit the field and label selectors to achieve that result.
• client_python's OpenMetrics implementation is the single source of truth upon which the various metric type implementations here are based on and tested against.

TODO

Planned (in the following order)

GA

• Meta-metrics for metric generation failures.
• Utilize fake client-set for all e2e tests.
• Add boilerplate headers automation.
• s/stores/generators Reverted: "stores" is clearer terminology
• rexagod/resource-state-metrics#1
• rexagod/resource-state-metrics#6
• Print controller logs in the CI.
• s/dependabot/renovate: kubernetes/org#6167
• Respect and keep up will all relevant metric types that are supported in Prometheus' OpenMetrics implementation.
• Add mixins.
• Cardinality estimation, and control
  • Talk to Prom server to get an idea of relevant label-sets' cardinality?
  • Use an offline-preferred approach with heuristics and internal context?
  • This will need to be reflected in the resource status (and tested outside of golden rules).
• Support testing status sub-resource in e2e tests (.out?).
• Typed spec instead of the YAML blob currently used in the ResourceMetricsMonitor CRD.
• Starlark resolver (for more demanding use-cases)

Post-GA

• Register the repository on the K8s release machinery, also integrate the bot.
• Add golden rules covering all CRS constructs.
• Dynamic admission control for ResourceMetricsMonitor CRD.

Done

• CEL expressions for metric generation (or *unstructured.Unstructured, if that suffices).
• Conformance test(s) for Kube-State-Metrics' Custom Resource State API.
• Benchmark(s) for Kube-State-Metrics' Custom Resource State API.
• E2E tests covering the controller's basic functionality.
• s/CRSM/CRDMetrics.
• Draft out a KEP.
• s/CRDMetrics/ResourceStateMetrics.
• Make ResourceMetricsMonitor namespaced-scope. This allows for:
  • per-namespace configuration (separate configurations between teams), and,
  • garbage collection (without finalizers), since currently the namespace-scoped deployment manages its cluster-scoped resources ResourceMetricsMonitors are user-managed, and should persist.

License