Build AI Features That Degrade Gracefully

We design every AI feature on the assumption that the model will sometimes be slow, unavailable or wrong. Graceful degradation is the practice of building a system so that when one part fails, the rest keeps doing its core job at a reduced level instead of failing outright. For an AI feature, that means the product still works when the model or the provider behind it does not.

The problem is that many teams wire a model call straight into a user flow. When the provider rate-limits, a prompt returns nonsense or inference slows under load, the whole screen fails with it.

This post covers why AI features fail more often than ordinary code, how graceful degradation differs from fault tolerance, a step-by-step way to design for it, the fallback patterns we use and the mistakes that turn a model outage into a product outage.

Why AI Features Fail More Often Than Ordinary Code

AI features fail more often than ordinary code because their output is probabilistic and their dependencies sit outside your control. A function that adds two numbers either works or throws an error, while a model can return a confident answer that is simply wrong, which makes graceful degradation a design requirement rather than an extra.

We group the causes into a handful of failure modes that every AI feature should plan for:

• Provider outages, rate limits and quota errors from a hosted model API.
• Latency spikes during peak load, when inference takes far longer than the screen can wait.
• Low-confidence or irrelevant outputs that are technically a response but not a useful one.
• Data drift, where real inputs slowly move away from the data the model was built and tested on.
• Malformed output, such as a response that breaks the expected format and crashes the code that parses it.
• Cost limits, where a budget cap or a spending alert forces the feature to slow down or stop.

Google's People + AI Guidebook makes a related point about how users see these failures. It notes that when dealing with a probabilistic, dynamic system, a user could perceive a failure even when the system is working as intended, so the design has to cover perceived errors as well as technical ones.

Graceful Degradation Compared With Fault Tolerance and Hard Failure

Graceful degradation keeps a reduced version of a feature running, fault tolerance hides a failure completely, and a hard failure stops the user. We use the comparison to decide how much engineering each AI feature deserves, because full fault tolerance is expensive and rarely needed for an optional capability.

AWS frames the goal well in its reliability guidance, which says application components should continue to perform their core function even if dependencies become unavailable, turning hard dependencies into soft ones. For most AI features we aim for exactly that, a soft dependency on the model with a defined reduced mode.

How to Design AI Features for Graceful Degradation

Design AI features for graceful degradation by working through seven steps before the feature ships, from mapping dependencies to testing the fallback path in production-like conditions. We follow the same order on every AI build, because each step depends on decisions made in the one before.

1. Map Every Dependency the AI Feature Touches

List the model provider, any vector store or retrieval service, the data sources that feed the prompt and the code that parses the output. Each one is a separate way for the feature to fail.

For each dependency, note who controls it and how you would know it is unhealthy. A hosted model you pay for and a database you run have very different failure patterns and very different fixes.

2. Write Down the Failure Modes Users Would Notice

Turn the dependency map into concrete scenarios, such as no response, a slow response, a malformed response and a confident wrong answer. Describe what the user would see in each case today if nothing were done.

This list becomes the specification for the fallback work. A failure mode that nobody has written down is one the product will handle badly when it finally happens.

3. Set Thresholds That Trigger the Reduced Mode

Decide the latency, error rate and confidence levels at which the feature should stop waiting for the model and switch to its fallback. Google's SRE book asks the same question of any service, listing metrics such as CPU usage, latency and queue length that can decide when load shedding or graceful degradation should kick in.

Base the thresholds on what the screen can tolerate, not on what the model usually achieves. A suggestion that arrives after the user has moved on is a failure even if the request technically succeeded.

4. Design a Specific Fallback for Each Failure Mode

Every failure mode needs a planned alternative, such as a cached answer, a rules-based result, a smaller model or a manual path. A generic error message is not a fallback, because it gives the user nothing to do next.

Choose the fallback by asking what the user was trying to accomplish. If the AI drafts a reply, the fallback is an empty editor with templates. If the AI ranks results, the fallback is a sensible default sort.

5. Wrap Model Calls in Timeouts and Circuit Breakers

Every model call needs a timeout, a limited retry policy and a circuit breaker. Martin Fowler describes the pattern plainly, in which a breaker monitors calls and, once failures reach a threshold, the circuit breaker trips and further calls return immediately without reaching the failing service.

The breaker stops a struggling provider from tying up your servers with waiting requests, and it sends users straight to the fallback instead of making each one wait for a timeout.

6. Tell Users When the Feature Is Running in Reduced Mode

Say plainly when results are simpler than usual or when a suggestion is unavailable, and offer the manual route. Users forgive a reduced feature far more readily than a silent wrong answer.

The People + AI Guidebook puts it simply, "when an AI system fails, often the easiest path forward is to let the user take over." We design that handover as a normal part of the screen, not as an error state.

7. Test the Fallback Path as Often as the Main Path

Fallbacks that never run tend to break quietly. Simulate provider outages, slow responses and malformed output in staging, and include those cases in automated tests.

Testing probabilistic features needs its own methods, which we cover in our guide on how to test AI features when outputs are non-deterministic. Track in production how often each fallback fires, because a rising rate is an early warning that the model or the provider is drifting.

Fallback Patterns That Keep an AI Feature Useful

The fallback patterns that keep an AI feature useful all share one property, which is that they serve the user's goal without the model. We pick from a short list and often combine two of them for one feature:

• A cached or precomputed result, such as yesterday's recommendations, served when a live call fails.
• A rules-based result that covers the common cases, which is often good enough and easier to explain.
• A smaller or cheaper model, or a second provider, used when the primary model is slow or unavailable.
• A manual path with helpful defaults, so the user can finish the task without any AI help.
• Hiding the feature entirely for a while, which is acceptable when it is purely assistive.

Rules-based fallbacks deserve special attention, because for many decisions they are not a downgrade at all. Our post on when a rules engine is better than a machine learning model explains where explicit logic beats a model outright.

Choosing and building these paths is part of the product engineering work we do on AI features, because the fallback decides what users experience on the model's worst day. A feature with graceful degradation built in can launch sooner, since an occasional model failure no longer blocks the workflow.

Mistakes That Turn an AI Outage Into a Product Outage

The mistakes that turn an AI outage into a product outage are almost always gaps in graceful degradation design rather than model problems. We see the same patterns repeat across teams adding their first AI features:

• Calling the model synchronously in the middle of a critical flow, such as checkout or sign-up, with no timeout.
• Catching errors but showing a generic message, so the user has no way to finish the task.
• Ignoring confidence, so a low-confidence prediction is shown with the same weight as a strong one.
• Relying on a single provider with no cached, rules-based or manual alternative.
• Never testing the fallback path, so it fails the first time it is needed.
• Leaving fallback events unmonitored, so drift and provider trouble go unnoticed until users complain.

Cost is the other quiet failure mode. Retries and duplicate calls during an incident can multiply spend at exactly the wrong moment, which is one of the reasons we track the full set of hidden costs of running AI in production from the first release.

Treat Graceful Degradation as Part of the AI Feature

Graceful degradation is not a reliability project to schedule after launch. Map the dependencies, name the failure modes, set thresholds, design a fallback for each case, wrap calls in breakers, tell users when results are reduced and test the fallback as often as the main path. We would ship an AI feature only once its worst-case behavior is as deliberate as its best case.

If you are planning an AI feature and want a second view on how it should fail, the team at Better Software can work through it with you.

Frequently Asked Questions

What Is Graceful Degradation in AI System Design?

Graceful degradation in AI system design means the product keeps working at a reduced level when a model is slow, unavailable or unreliable. We plan it by giving each AI feature thresholds, a fallback path and a clear message to users, so the core workflow never depends on the model alone.

What Does Graceful Degradation Mean?

Graceful degradation means a system loses some capability under failure instead of stopping completely. We apply it so that users can still finish their main task, perhaps with simpler results or a manual step, while the failing component recovers or is fixed behind the scenes.

How to Do Graceful Degradation?

We start by listing every dependency and the ways each can fail, then set thresholds that trigger a reduced mode. Each failure gets a specific fallback, model calls get timeouts and circuit breakers, and the fallback paths are tested and monitored as carefully as the main path.

What Is Fallback in AI?

A fallback in AI is the alternative behavior a product uses when the model cannot give a timely or trustworthy answer. We typically use a cached result, a rules-based answer, a smaller model or a manual path, chosen according to what the user was trying to do.

How Often Is AI Wrong?

There is no single error rate, because it depends on the model, the task and the data it sees. We measure it on each product's own inputs before launch and keep measuring in production, then design fallbacks and confidence thresholds around the error rate we actually observe.