Core Metrics for Production Health

Why This Happens

Production systems exhibit varying states. These states range from optimal performance to complete failure. Metrics provide objective data about system behavior. Without quantifiable metrics, assessing system health becomes subjective. Subjective assessments lead to misdiagnosis or delayed incident response. Instrumentation captures data points from system components. Aggregation and visualization of these data points create a health overview. This overview allows engineers to identify deviations from normal operating patterns. Early detection of deviations prevents larger incidents.

Investigation Process

1. Define Critical User Journeys. Identify essential user interactions. Map the technical components supporting these journeys. For an e-commerce platform, checkout flow is critical. For an API, core request types are critical.
2. Establish Service Level Indicators (SLIs). Determine what to measure for each component. Common SLIs include latency, availability, and error rate. Latency measures response time. Availability measures uptime. Error rate counts failed requests. Each SLI must be measurable and quantifiable.
3. Set Service Level Objectives (SLOs). Define target values for each SLI. For example, a high availability target for a core service. Or, a latency target that most requests must meet. SLOs guide monitoring and alerting thresholds.
4. Instrument All System Layers. Deploy agents or code for data collection. Collect metrics from infrastructure (CPU, memory, disk I/O, network). Collect metrics from applications (request counts, error types, response times). Collect metrics from business logic (transaction success rates). Ensure consistent timestamping across all data sources.
5. Correlate Metrics to Business Impact. Link technical metrics to user experience. A spike in database latency might degrade login performance. Reduced login performance directly impacts user access. Understand the cascading effects of component failures. Instrumenting Products to Detect Product-Market Fit Signals provides further context.
6. Establish Baselines and Alerting. Collect historical metric data to define normal behavior. Configure alerts for deviations from established SLOs. Use thresholds for immediate notification. Set up anomaly detection for subtle shifts in patterns.
7. Implement Distributed Tracing. Trace requests across multiple services. This identifies latency bottlenecks within complex architectures. It visualizes the flow of execution and data. This is crucial for microservices environments. See also Strategies for Reducing Production Incident Diagnosis Time.

Practical Example

A user reported slow page loads on the product catalog. Initial checks showed overall service availability was normal. Latency metrics for the web frontend were normal. Database CPU utilization was low. However, application logs showed an increase in HTTP 500 responses from the image service. This image service was a third-party dependency. The image service HTTP 500s did not register as a critical system error. Our monitoring aggregated them only as application-layer errors. They did not trigger the broader HTTP 5xx alerts for our internal services. Further inspection of the image service’s dedicated metrics revealed high read latency. This was specific to image retrieval operations. The service became unresponsive after a long wait, and the frontend was timing out sooner. The root cause was identified as a degraded inode performance on the third-party image host's storage array. The slow image loading caused reported slow page loads. The HTTP 500s were actually timeouts. We identified the impact by correlating separate metrics such as frontend load times, internal application errors, and third-party service latency. We implemented a circuit breaker for the image service. We also introduced a fallback to a placeholder image after a short timeout.

Preventing Recurrence

Preventing recurrence requires systemic improvements. First, standardize observability across all services. Every new service must deploy with defined SLIs and SLOs. This includes third-party integrations. Second, implement comprehensive error categorization. Distinguish between internal service errors and external dependency errors. Route external errors to specific alert channels. Third, refine alert thresholds based on historical performance data. Use percentile-based alerts for latency. Focus on p90, p95, and p99 metrics instead of averages. Fourth, regularly review incident post-mortems. Identify gaps in monitoring and tooling. Implement changes to address these gaps. For example, add automated synthetic transactions to dependency checks. These transactions ensure critical paths are functional end-to-end. Finally, proactively test monitoring and alerting systems. Simulate failures to ensure alerts fire correctly. Review alert escalation paths regularly.

We define SLIs, SLOs and alerting as part of product engineering, before a service reaches production.

What Teams Usually Do Instead

Many teams rely solely on infrastructure metrics. They monitor CPU, memory, and disk usage. These metrics show resource health but not service health. A database with low CPU usage can still have high query latency. High resource utilization can sometimes indicate efficiency, not problems. Another common anti-pattern is alert fatigue. Too many alerts desensitize engineers. This leads to missed critical incidents. Alerts should be actionable and tied to SLO breaches. Teams often implement broad HTTP 5xx alerts. These alerts lack granularity for specific service issues. They mask problems in individual components within a distributed system. Over-reliance on vague logs is another issue. Logs are critical for debugging but not for real-time health assessment. They provide retrospective data. They offer limited immediate insights into evolving system state. Why Incident Severity Labels Lose Operational Utility Over Time discusses alert prioritization challenges.

Key Takeaways

• Monitor latency, availability, and error rates.
• Define Service Level Indicators (SLIs) and Objectives (SLOs).
• Instrument all system layers, including dependencies.
• Correlate technical metrics with concrete business impact.
• Implement distributed tracing for complex architectures.

For more on how we work, visit Better Software.

Frequently Asked Questions

What Metrics Show Production Health?

We start with availability, request latency, error frequency and resource consumption, measured for each critical user journey. Infrastructure metrics such as CPU and memory show resource health, while application and business-logic metrics show service health. We want all three layers instrumented with consistent timestamps.

What Are SLIs and SLOs?

Service level indicators are measurable signals such as latency, availability and error rate. Service level objectives are target values for those signals. We use SLOs to guide monitoring and alerting thresholds, so every alert ties back to a breach that matters to users.

Why Are Infrastructure Metrics Not Enough?

A database with low CPU usage can still have high query latency, and high utilization can sometimes signal efficiency rather than trouble. In the example we use, only dedicated metrics for a third-party service exposed the real problem behind the slow page loads.

How Do We Link Technical Metrics to Business Impact?

We trace how a technical symptom affects a user journey. A spike in database latency might slow logins, which blocks user access. Correlating frontend load times, internal application errors and third-party latency let us identify the impact in the example.

How Should We Set Up Alerts?

We build baselines from historical data, alert on deviations from SLOs, and use percentile thresholds such as p95 and p99 rather than averages. We separate internal errors from external dependency errors, keep alerts actionable to avoid fatigue, and test that alerts fire correctly.