Metric Definition Drift
Metric definition drift is when a metric retains the same name but its meaning changes over time. This often happens through quiet changes to business logic, filters, join paths, or default aggregations, leading to inconsistent decisions even when dashboards look stable.
Common failure signals
- Same report shows different totals after "minor" model updates
- Metric aligns for one segment but diverges for others
- Week-over-week comparisons break due to definition changes
- Different teams compute "the same metric" differently
Often confused with
- Data changes (the data can be correct while the metric definition changed)
- Visualization issues (drift is upstream in logic/semantics)
- KPI strategy changes (drift can happen unintentionally)
Where it shows up in Analytical Reliability
- Semantic Reliability: measures and relationships shift meaning
- Change Reliability: PRs or model edits modify logic without renaming the metric
- Data Movement Reliability: upstream mapping changes alter what the metric counts