Investigating Intermittent Production Issues

A production issue that cannot be reproduced locally or in staging environments often stems from specific environmental conditions, data permutations, or concurrency scenarios inherent to production. Investigation shifts from direct reproduction to inferring behavior from available telemetry.

Why This Happens

Production environments present unique characteristics. These include scale, real user data, network latency, resource contention, and external service interactions. Replicating this complexity in development or staging is cost-prohibitive. Context switching often occurs as engineers struggle to identify differing environmental variables. Specific user actions or data states can trigger edge cases. Concurrency issues may only manifest under high load. Third-party integrations can introduce non-deterministic failures. These factors combine to make direct reproduction impractical or impossible.

Investigation Process

1. Collect Comprehensive Telemetry: Gather all available logs, metrics, and traces. Prioritize logs from affected services and their dependencies. Look for anomalies in system resource utilization.
2. Define the Blast Radius: Identify affected users, features, and services. Determine the time window of the incident. This narrows the scope of log analysis.
3. Timeline Reconstruction: Order events chronologically across all relevant systems. Focus on events immediately preceding the observed failure. Look for common patterns or unique events.
4. Examine Environmental Factors: Check for recent deployments, configuration changes, upstream service outages, or scheduled jobs. Correlate these with the incident timeline.
5. Isolate Data-Specific Patterns: Analyze database queries and data payloads. Look for specific data values or formats that might trigger errors. Redact sensitive information where necessary.
6. Review Code Paths: Identify code sections executed during the incident time. Focus on areas interacting with external systems, handling concurrency, or manipulating critical data. This helps narrow down potential failure points.
7. Hypothesize and Monitor: Formulate hypotheses about the root cause. Implement targeted logging or metrics for the suspected component. Wait for the issue to recur, capturing more detailed information. This iterative process is crucial for reducing human investigation overhead.
8. Simplify the Problem Space: If the issue is still elusive, consider disabling non-critical features. Temporarily roll back recent changes. This can isolate the problematic component by elimination.

Practical Example

A payment processing service reported intermittent failures for a small percentage of transactions. The error message indicated a timeout from a third-party payment gateway. These failures were not reproducible in staging or local environments. Production logs showed successful calls to the gateway 99% of the time. Investigation involved correlating internal payment service logs with gateway callback logs. Engineers noticed a specific pattern: failures occurred only when the payment amount exceeded a certain threshold (e.g., $500). Further log analysis revealed that transactions over $500 were routed through a secondary, older gateway integration pathway. This pathway had a stricter timeout configuration on the internal service side. The primary gateway integration was more resilient. The issue was not gateway uptime, but an internal configuration difference based on transaction value. The team updated the timeout for the secondary gateway path to match the primary. This resolved the intermittent failures.

Preventing Recurrence

Robust monitoring and logging are essential. Instrument critical code paths with detailed logging, including input parameters and external service responses. Implement distributed tracing to visualize request flow across services. Establish alerts for anomalous metric patterns (e.g., increased error rates, elevated latency). Conduct thorough post-incident reviews to document findings and implement preventative measures. Regularly review and align staging environment configurations with production. Proactively identify production issues with high prevention ROI.

What Teams Usually Do Instead

Many teams attempt to reproduce the issue endlessly in non-production environments. This wastes significant engineering time without yielding results. Another common pitfall is immediately blaming external services without internal system analysis. This delays identification of internal configuration or code issues. Teams might also implement broad, untargeted logging, overwhelming log aggregation systems. This makes incident data extraction difficult. Ignoring intermittent issues due to low frequency also increases future operational risk. Prioritizing product features under perceived urgency while ignoring production stability can have cascading effects.

Key Takeaways

• Direct reproduction is often not possible.
• Focus on comprehensive telemetry and log analysis.
• Reconstruct event timelines across all services.
• Hypothesize root causes and implement targeted monitoring.
• Prevent recurrence through better instrumentation and reviews.

Related: how some teams handle this operationally