Chasing Issues You Cannot Reproduce

In our experience, a production issue that cannot be reproduced locally or in staging environments often stems from specific environmental conditions, data permutations, or concurrency scenarios inherent to production. Investigation shifts from direct reproduction to inferring behavior from available telemetry.

Why Some Production Issues Cannot Be Reproduced

Production environments present unique characteristics. These include scale, real user data, network latency, resource contention, and external service interactions. Replicating this complexity in development or staging is cost-prohibitive. Context switching often occurs as engineers struggle to identify differing environmental variables. Specific user actions or data states can trigger edge cases. Concurrency issues may only manifest under high load. Third-party integrations can introduce non-deterministic failures. These factors combine to make direct reproduction impractical or impossible.

How to Investigate Without Reproducing the Issue

1. Collect Comprehensive Telemetry. Gather all available logs, metrics, and traces. Prioritize logs from affected services and their dependencies. Look for anomalies in system resource utilization.
2. Define the Blast Radius. Identify affected users, features, and services. Determine the time window of the incident. This narrows the scope of log analysis.
3. Timeline Reconstruction. Order events chronologically across all relevant systems. Focus on events immediately preceding the observed failure. Look for common patterns or unique events.
4. Examine Environmental Factors. Check for recent deployments, configuration changes, upstream service outages, or scheduled jobs. Correlate these with the incident timeline.
5. Isolate Data-Specific Patterns. Analyze database queries and data payloads. Look for specific data values or formats that might trigger errors. Redact sensitive information where necessary.
6. Review Code Paths. Identify code sections executed during the incident time. Focus on areas interacting with external systems, handling concurrency, or manipulating critical data. This helps narrow down potential failure points.
7. Hypothesize and Monitor. Formulate hypotheses about the root cause. Implement targeted logging or metrics for the suspected component. Wait for the issue to recur, capturing more detailed information. This iterative process is crucial for reducing human investigation overhead.
8. Simplify the Problem Space. If the issue is still elusive, consider disabling non-critical features. Temporarily roll back recent changes. This can isolate the problematic component by elimination.

A Practical Example With a Payment Gateway

Here is an illustrative example we use to show the pattern. A payment processing service reported intermittent failures for a small percentage of transactions. The error message indicated a timeout from a third-party payment gateway. These failures were not reproducible in staging or local environments. Production logs showed successful calls to the gateway 99% of the time. Investigation involved correlating internal payment service logs with gateway callback logs. Engineers noticed a specific pattern. Failures occurred only when the payment amount exceeded a certain threshold (e.g., $500). Further log analysis revealed that transactions over $500 were routed through a secondary, older gateway integration pathway. This pathway had a stricter timeout configuration on the internal service side. The primary gateway integration was more resilient. The issue was not gateway uptime, but an internal configuration difference based on transaction value. The team updated the timeout for the secondary gateway path to match the primary. This resolved the intermittent failures.

Preventing Recurrence With Better Instrumentation

Robust monitoring and logging are essential. Instrument critical code paths with detailed logging, including input parameters and external service responses. Implement distributed tracing to visualize request flow across services. Establish alerts for anomalous metric patterns (e.g., increased error rates, elevated latency). Conduct thorough post-incident reviews to document findings and implement preventative measures. Regularly review and align staging environment configurations with production. Proactively identify production issues with high prevention ROI.

What Teams Usually Do Instead

Many teams attempt to reproduce the issue endlessly in non-production environments. This wastes significant engineering time without yielding results. Another common pitfall is immediately blaming external services without internal system analysis. This delays identification of internal configuration or code issues. Teams might also implement broad, untargeted logging, overwhelming log aggregation systems. This makes incident data extraction difficult. Ignoring intermittent issues due to low frequency also increases future operational risk. Prioritizing product features under perceived urgency while ignoring production stability can have cascading effects. We treat production stability as part of product engineering, not as a side task.

Key Takeaways for Intermittent Issues

• Direct reproduction is often not possible.
• Focus on comprehensive telemetry and log analysis.
• Reconstruct event timelines across all services.
• Hypothesize root causes and implement targeted monitoring.
• Prevent recurrence through better instrumentation and reviews.

If an intermittent production issue keeps defeating your team, talk to Better Software about a structured investigation.

Frequently Asked Questions

Why Can I Not Reproduce a Production Issue?

In our experience production has unique characteristics, including scale, real user data, network latency, resource contention and external service interactions. Replicating that complexity in development or staging is cost-prohibitive, so direct reproduction is often impractical.

How Do I Investigate an Issue I Cannot Reproduce?

We shift from reproduction to inference. That means collecting comprehensive telemetry, defining the blast radius, reconstructing a timeline across systems, checking environmental factors, isolating data-specific patterns, reviewing code paths, and then hypothesizing and adding targeted monitoring until the issue recurs.

What Should I Look for in the Timeline?

We order events chronologically across all relevant systems and focus on what happened immediately before the failure. We correlate recent deployments, configuration changes, upstream outages and scheduled jobs with that timeline, and look for common patterns or unique events.

What If the Issue Still Cannot Be Found?

We simplify the problem space by disabling non-critical features or temporarily rolling back recent changes, which can isolate the problematic component by elimination. We also keep targeted logging in place and wait for the issue to recur with more detail.

What Mistakes Waste Time on Intermittent Issues?

We see teams try to reproduce the issue endlessly outside production, blame external services before analyzing their own systems, add broad untargeted logging that overwhelms aggregation, or ignore low-frequency issues. Each wastes time or raises future operational risk.