NOTE
Designing Metrics Monitoring
Metric naming/types, labels, cardinality control, RED/USE signals, SLOs, aggregation, and alerting.
This is a historical learning note and may contain outdated or incomplete understanding.
Metrics compress system behavior into numeric time series. Use counters for cumulative events, gauges for current state, and histogram-style distributions for latency/size rather than averaging away tails.
Monitor user-facing RED signals (rate, errors, duration) and resource USE signals (utilization, saturation, errors). Build alerts from actionable SLO/risk symptoms rather than every metric threshold.
Control label cardinality: request IDs, user IDs, raw URLs, or unbounded values can make the metrics backend itself fail.