Monitoring & Alerting Code Audit

Common Monitoring Audit Findings

Alert fatigue is the number one finding. Teams have hundreds of alerts, most of which fire regularly without requiring action. Engineers mute notifications. When a real incident occurs, the page is ignored because it looks like every other false alarm. Monitoring exists but doesn’t function - it generates noise instead of signal.

Dashboard sprawl is the second finding. Fifty Grafana dashboards created by different engineers at different times. No hierarchy. No consistent naming. No agreement on what metrics matter. Engineers create new dashboards for each investigation and never delete them. During an incident, nobody knows which dashboard to open.

Missing business metrics is the third. Plenty of infrastructure monitoring - CPU, memory, disk - but no visibility into what users experience. The database has 10% CPU but queries are taking 5 seconds because of lock contention. Infrastructure metrics say “everything is fine” while users say “the app is slow.”

Our Monitoring Audit Approach

We start with the alerts. Every alert is classified: actionable (someone needs to do something), informational (useful context but doesn’t require action), or noise (neither useful nor actionable). Noise alerts are candidates for deletion. Informational alerts move to dashboards. Actionable alerts get runbooks.

Dashboard inventory identifies coverage and gaps. We map dashboards to services and check for redundancy. We assess whether dashboards answer the questions engineers actually ask during incidents. Missing dashboards for critical user journeys are flagged. Orphaned dashboards for decommissioned services are archived.

Metrics analysis quantifies cost and coverage. High-cardinality metrics that consume disproportionate resources are identified. Missing metrics for critical operations are flagged. We recommend SLOs based on user expectations and SLIs that measure them. The monitoring system becomes intentional instead of accidental.

What Changes After the Audit

Alert count drops dramatically - often by 70-80%. Every remaining alert requires human action and has a runbook. On-call shifts become manageable because pages are meaningful. Mean time to detection improves because real alerts aren’t buried in noise.

Dashboards become navigable. A top-level dashboard shows system health at a glance. Service dashboards show the RED metrics (Rate, Errors, Duration) for each service. Investigation dashboards support debugging with appropriate drill-down. Engineers know exactly where to look during incidents.

Monitoring costs come under control. High-cardinality label combinations that inflate Prometheus storage or drive up Datadog bills are restructured or dropped. Metric retention policies align with actual usage: high-resolution data for the last 48 hours, downsampled data for the last 30 days, and aggregated data for long-term trend analysis. We typically see a 30-50% reduction in monitoring platform costs after the audit, achieved not by reducing visibility but by eliminating redundant series, consolidating duplicate dashboards, and tuning scrape intervals to match the actual cadence at which data is consumed.