Cutting Unplanned Engineering Work

Unplanned engineering work stems from system instability, incidents, or emergent technical debt requiring immediate attention. It displaces planned development, impacting project timelines and team morale. Reducing its occurrence involves improving system predictability, observability, and structured incident response processes.

Why This Happens

Unplanned work arises when systems behave outside expected parameters. Insufficient monitoring often masks subtle degradation, leading to sudden, critical failures. Technical debt accumulation also contributes; poorly designed or maintained components eventually break. Lack of clear ownership or operational runbooks means incident response is ad hoc. Architecturally expensive components often accrue significant technical debt. Organizational pressures to ship features quickly can deprioritize reliability, maintenance, and resiliency measures.

Investigation Process

1. Quantify Unplanned Work. Track time spent on incidents, bug fixes, and urgent maintenance outside planned sprints. Categorize it by system area and trigger.
2. Identify Triggers. Determine what initiates unplanned work. Is it production incidents, user bug reports, security vulnerabilities, or operational alerts?
3. Root Cause Analysis (RCA). For each major unplanned event, conduct a blameless RCA. Focus on system failures, process gaps, and missing safeguards.
4. Dependency Mapping. Understand upstream and downstream dependencies for critical services. Failures often cascade from unexpected sources.
5. Observability Gaps. Review existing monitoring and alerting for missed signals. Were there indicators that could have predicted the event?
6. Historical Trend Analysis. Look for recurring patterns across different incidents. Are similar types of errors happening in different services?
7. Team Capacity Assessment. Evaluate if teams have dedicated time for reliability work, operational improvements, and technical debt reduction.

Practical Example

A payment processing service experienced three critical outages in two months. Each outage required immediate engineering intervention, pulling developers off planned feature work. Investigation revealed all outages stemmed from a specific database query timing out under peak load. The query was part of a legacy reporting module, poorly optimized, and executed directly on the primary transactions database. Alerting existed for database CPU but not for specific query latency or error rates. Engineers manually killed the query during outages, then restarted the service. The solution involved rewriting the problematic query, optimizing its execution plan, and deploying it to a read replica. New alerts for query latency were added. This proactive work prevented further unplanned interventions.

Preventing Recurrence

1. Prioritize Reliability Work. Allocate dedicated engineering capacity for error budget management, platform stability, and technical debt. Treat reliability as a feature.
2. Enhance Observability. Implement comprehensive logging, metrics, and tracing for all critical services. Establish actionable alerts with clear runbooks. Focus on service-level objectives (SLOs).
3. Blameless Post-Mortems. Conduct post-mortems for all critical incidents. Focus on systemic failures and improvement opportunities, not individual blame. Ensure action items are tracked and completed.
4. Automate Routine Tasks. Automate deployments, rollbacks, and common operational procedures. Reduce manual intervention points.
5. Strict Change Management. Implement review processes, staging environments, and gradual rollouts for all production changes. Backout plans must be clear.
6. Chaos Engineering. Proactively inject failures into systems to uncover hidden weaknesses before they manifest as outages.
7. Technical Debt Allocation. Regularly allocate time in sprints for addressing identified technical debt. Do not let it accumulate indefinitely.

What Teams Usually Do Instead

Many teams react solely to incidents without conducting thorough root cause analysis. They implement quick fixes, believing they have solved the problem. This often addresses symptoms, not underlying issues. Another common pattern is deferring all reliability work in favor of new features, creating a perpetual cycle of unplanned interruptions. Some teams create extensive documentation and runbooks that become outdated quickly, leading to confusion during active incidents. They might invest in more monitoring tools without defining what metrics are critical or how alerts should trigger. This generates noise without actionable intelligence. Leadership often fails to adequately budget for operational improvements, prioritizing visible feature delivery over invisible stability.

Key Takeaways

• Quantify unplanned work to understand its cost.
• Conduct blameless root cause reviews consistently.
• Invest in comprehensive observability and actionable alerts.
• Dedicate resources to reliability and technical debt.
• Automate operational tasks to reduce manual errors.

See also how we help teams build products, separating planned from reactive work and measuring time lost to investigations.

Frequently Asked Questions

What Causes Unplanned Engineering Work?

We usually see systems behaving outside expected parameters, monitoring that masks subtle degradation, accumulated technical debt, no clear ownership or runbooks, and pressure to ship features that deprioritizes reliability. Each of these turns small weaknesses into sudden failures that pull people off planned work.

How Do You Measure Unplanned Work?

We track time spent on incidents, bug fixes and urgent maintenance outside planned sprints, then categorize it by system area and trigger. Triggers include production incidents, user bug reports, security vulnerabilities and operational alerts, and historical trends show whether similar errors repeat across services.

Why Run Blameless Post-Mortems?

We run them for all critical incidents because they focus on systemic failures and improvement opportunities rather than individual blame. We also track action items until they are completed, otherwise a post-mortem becomes a record of the problem instead of a way to prevent the next one.

What Do Teams Usually Do Instead?

We often see teams react to incidents with quick fixes and no root cause analysis, defer all reliability work in favor of features, keep runbooks that quickly go stale, and add monitoring tools without defining which metrics matter. That addresses symptoms and generates noise without actionable intelligence.

How Much Capacity Should Go to Reliability?

We allocate dedicated engineering capacity for error budget management, platform stability and technical debt, and treat reliability as a feature. Time for technical debt is scheduled in regular sprints so it does not accumulate indefinitely, and leadership has to budget for it even though stability is less visible than features.