NOTE
Designing an Observability System
Metrics, logs, traces, events, SLOs, correlation, storage, alerting, dashboards, and cost/cardinality control.
This is a historical learning note and may contain outdated or incomplete understanding.
An observability platform ingests telemetry from services/infrastructure, stores/query-indexes it, and turns it into alerts, dashboards, traces, and incident evidence.
Start from questions/SLOs rather than collecting everything. Standardize service/resource identity and trace/request correlation so metrics, logs, and traces can be joined conceptually.
High-cardinality telemetry and long retention are expensive. Apply sampling, aggregation, retention tiers, quotas, and privacy/access controls while preserving enough evidence for debugging.