Metrics: Instrumentation design

Metric labels are validated and bounded before ingestion using per-metric cardinality budgets, preserving high-cardinality debug paths without letting unbounded tags explode storage or query cost.

Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.

Decision

High-cardinality labels are controlled at ingestion time with per-metric budgets rather than after storage costs spike. Each metric family declares allowed labels, maximum distinct series, and drop or aggregate behavior when budgets are exceeded. Debuggability is preserved through intentional escape hatches—scoped debug sessions or dedicated low-retention diagnostic metrics—not by allowing unbounded user ids or request ids on hot paths. Label validation rejects unknown or malformed tags before they enter the time-series backend.

Scope

• Label allowlists and cardinality counters per metric name.
• Ingestion-side enforcement, drop rules, and operator-visible drop counters.
• Debug override channels with TTL and approval for temporary high-cardinality capture.
• Dashboards and alerts on budget utilization and dropped series.
• Developer guidance for safe label design on new instrumentation.

Outside this block

• Log pipeline cardinality (see structured logging block).
• Trace tail sampling policies.
• Billing negotiations with vendors.

Contract

• Every hot-path metric declares a cardinality budget before production ingestion is enabled.
• Labels are validated pre-ingestion; unknown labels on budgeted metrics are dropped or rejected per profile.
• Exceeding budget increments a visible cardinality_drop_total counter and never silently discards the entire metric stream without audit.
• Debug overrides require ticket id, expires_at, and auto-expire; they cannot permanently raise hot-path budgets.
• User ids, email addresses, and raw URLs are forbidden as label values on budgeted metrics.
• Aggregated overflow label route_template=cardinality_overflow preserves SLO dashboards when detail is dropped; the same overflow label name and value are used in all fixtures and enforcement paths.
• Budget configuration changes are versioned and take effect without restarting application processes where possible.

Implementation guidance

• Implement enforcement in agents or receivers, not only in documentation.
• Expose per-metric series counts to service owners weekly.
• Pair dropped-label events with structured log correlation ids only in debug mode.

Failure handling and safeguards

• If the cardinality controller is unavailable, default to drop-unknown-labels rather than accept all tags.
• Emergency budget increases require ticket linkage, expires_at, and auto-expire; expired overrides revert without redeploy.
• Do not strip labels required for legally mandated audit metrics without compliance review.

Verification and operations

• Track budget utilization, dropped series rate, and debug override usage.
• Load-test ingestion with synthetic label explosions to verify enforcement latency.
• Execute acceptance scenarios in the target repository; packaged scenarios are not executed evidence.

Adoption assumptions

• Thresholds in example.yaml and metric-budget.yaml are profile choices, not universal defaults.
• Teams map metric registries and ownership to their observability stack.

References

• prometheus-label-practices