When Usage Data and User Feedback Disagree

Every founder we work with eventually hears users ask loudly for a feature that the numbers say almost nobody uses. Usage data is the record of what people actually do in a product, such as which screens they open, which actions they complete and where they drop off, while user feedback is what they tell you in interviews, surveys and support tickets.

When the two disagree, neither is simply wrong. Each is evidence of a different kind, and the disagreement itself is often the most useful finding. This post covers why stated wants and observed behavior diverge, how the two kinds of evidence compare, a step-by-step way to resolve the conflict, a worked example and the mistakes that lead teams to build the wrong thing.

Why Users Ask for What the Usage Data Never Shows

Users describe what they think they should do, what they would like to do or what a past frustration felt like, not a neutral log of their behavior. Jakob Nielsen's long-standing advice is to pay attention to what users do, not what they say, because self-reported claims and predictions about future behavior are unreliable.

We see a few repeat causes. Users propose a solution instead of describing the problem. A vocal minority files most of the requests. Users may not know that a feature already exists, or their mental model of the product differs from how it works.

Usage data has blind spots too. It shows what happened inside the product, but not what users did outside it, what they wanted and could not find or why they left. That is why we treat a conflict as a question to investigate rather than a vote to count.

Stated Preferences vs Observed Behavior in Product Decisions

The two kinds of evidence answer different questions, and good product decisions need both. Nielsen Norman Group frames it as attitudinal research that captures user opinions and feelings in the form of self-reported data and behavioral research that observes user actions.

Read that way, a conflict usually means users have a real problem and the requested feature is not the solution they will use. We find that framing far more productive than arguing about which source wins.

How to Resolve a Conflict Between Usage Data and Feedback

Resolve the conflict by confirming the usage data is right, segmenting it, finding the job behind the request and testing a small change before building anything large. We follow seven steps.

1. State the Conflict Precisely

Write the disagreement in one sentence. For example, users ask for a full project view, but usage data shows they live in the task list.

A precise statement stops the discussion from drifting into general opinions about what users want.

2. Check the Data Before Trusting It

Confirm the events are tracked correctly and mean what you think. Missing instrumentation, renamed events or a feature that is hard to find can all make real demand look like low usage.

We have seen teams drop a feature for low usage when the real problem was that the entry point was buried.

3. Segment Users Before Concluding

Split the usage data by plan, role, tenure and account size. The requests may come from a segment that matters a great deal even if it is small, or from a vocal group that does not represent the base.

Instrumenting for this from the start is part of what we cover in how to instrument your product to detect product-market fit.

4. Find the Job Behind the Request

Go back to the people who asked and find out what they were trying to get done. The Christensen Institute describes jobs to be done as the progress people are trying to make as they strive toward a goal in particular circumstances.

Ask about the last time they hit the problem and what they did instead. Their workaround is often the best design input you will get.

5. Watch Users Do the Task

Observe a few users completing the task in the product. Watching closes the gap between what the data shows and what users say, because you see both the behavior and the reasons in one session.

Small rounds are enough. The goal is to understand the pattern, not to produce statistics.

6. Test a Small Change First

Turn the finding into a hypothesis and test the smallest change that addresses it, such as moving an entry point, simplifying a screen or improving an export. Measure the effect on usage before committing to the large feature.

The same discipline applies before a product exists, which we cover in how to validate an MVP before you build.

7. Close the Loop With the People Who Asked

Tell the users and stakeholders who made the request what you found and what you chose to build. Show the usage data and the job you uncovered.

Users who see their problem solved rarely mind that the solution differs from their suggestion. Users who hear nothing assume they were ignored.

Signs the Users Are Right and the Usage Data Is Wrong

Usage data is not always the better witness. We give more weight to what users say in a few specific situations, because the data cannot see the problem.

• The requested capability does not exist yet, so there is no usage to measure.
• Users describe a workaround outside the product, such as spreadsheets, email or a competitor tool.
• The feature exists but users cannot find it, which shows up as low usage and repeated requests.
• Requests come from a segment that drives most of the revenue, even if it is a small share of users.

In each case, low usage reflects a gap in what the data can capture, not a lack of need. We would still test before building, but the starting assumption flips, and the test is about how to serve the need rather than whether it exists.

A Worked Example From a Marketing Analytics Product

Here is a hypothetical example, not a client story. A marketing platform receives many requests for advanced analytics, including custom report builders and detailed segmentation. Usage data for the existing analytics section tells a different story.

Few users open reports more than occasionally, and the most used view is a simple campaign summary with a handful of metrics. Drill-downs are rarely clicked, while data exports are common. Interviews explain the gap, because teams do their real analysis in spreadsheets and the in-app tools do not fit how they work.

Instead of a custom report builder, the team improves exports and adds a simpler summary view with the metrics people actually check. Usage of the summary rises and exports become easier. We like this example because the users were right about the problem and wrong about the solution, and the usage data showed which was which.

Usage Data Mistakes That Lead to the Wrong Build

The expensive mistakes come from treating either source as the whole truth. These are the ones we see most often.

• Building the loudest request without checking usage data, which produces features users asked for and rarely open.
• Reading low usage as low need, when the feature may be hard to find or poorly designed.
• Skipping segmentation, so a small but valuable group is ignored or a vocal group is overweighted.
• Treating requests as specifications instead of clues to an underlying job.
• Launching a large feature without a small test, which turns a hypothesis into a long project.
• Not explaining decisions, which teaches users that feedback disappears.

The first mistake is the most common, and it is how products fill up with features first users never touch. Pairing usage data with user conversations from the first release is part of how we approach product engineering with early-stage teams.

Let Usage Data and Users Answer Different Questions

When usage data contradicts what users say they want, trust behavior for what people will use and conversations for why. Check the data, segment it, find the job behind the request, test a small change and explain the decision.

We have found that teams who treat the conflict as a research question build fewer unused features and earn more trust from the users who asked. If you want help setting that up in your product, Better Software can work through it with you.

Frequently Asked Questions

What Is the Difference Between Stated and Revealed Preferences?

Stated preferences are what people say they want or would choose, while revealed preferences are what their actual choices show. In products, we treat interviews and surveys as stated preferences and usage data as revealed preferences, and we expect the two to differ.

What Is the Main Difference Between Behavioral and Attitudinal Research?

Behavioral research observes what people do, through usage data, analytics or watching tasks. Attitudinal research captures what people think and feel, through interviews and surveys. We use behavioral research to learn what happens and attitudinal research to learn why it happens.

When to Use Qualitative vs Quantitative Research?

We use quantitative research, such as usage data, to measure how many people do something and how often. We use qualitative research, such as interviews and observation, to understand reasons and context. Most product questions need quantitative data to find the pattern and qualitative work to explain it.

Can Qualitative and Quantitative Research Be Combined?

Yes, and we think they work best together. Usage data shows where users struggle or drop off, and interviews or observation explain why. Combining them keeps teams from building a feature on opinions alone or dropping one because of a misleading number.

What Is Revealed Preference?

Revealed preference is the idea that the real priorities of people show in what they choose, not in what they say. We apply it to products by trusting usage data for what users value in practice, while still asking users directly to understand their reasons.