Data Alone Doesn't Make Delightful Experiences

Knowing that only 40% of the participants are able to complete a task doesn't say why users had trouble with that task or how to make it easier.

Originally published on February 14, 2020

Spend some time on Twitter and LinkedIn, and you'll start to notice a lot of designers and product folks talk about being data-driven.

But if you spend some time talking to them, you'll realize that all they actually do is track and collect tons of user data, and then use it to A/B test their way to increase conversion.

From a strict marketing point of view, conversion-driven iteration is fine. But then, what I've noticed in many of our clients is that, very soon, variables like churn rate and the always elusive engagement start to go astray.

Disclaimer: All links to products are Amazon affiliate links.

The problem with the conversion-driven approach

A high-quality digital product should do more than increase sign-ups. It should be informative, easy to learn, and pleasant to use.

An easy to use product not only makes people want to use it more, but it also requires less customer support, which means you can invest more in marketing or engineering. Heck, you might even be able to afford that retreat you've been promising your team but haven't had the budget to plan.

Closeup take of a laptop displaying a dashboard with different metrics. Photo by Stephen Dawson on Unsplash. From Ed Orozco Design Strategy & UX Consulting.
Photo by Stephen Dawson on Unsplash

What hard, quantitative data doesn't tell you

Quantitative data tells you what's going on, but it doesn't tell you why. You might notice the user spending a lot of time in the checkout form and dropping off. But it doesn't tell you that they want to see a conversion in their local currency before paying.

It tells you they spend a long time choosing a movie to watch, but it doesn't tell you they want to watch a Marvel movie.

Knowing that only 40% of the participants are able to complete a task doesn't say why users had trouble with that task or how to make it easier. – NN Group
Qualitative data complements your understanding of user behavior.

Why You Need Qualitative Research

Qualitative research reveals what hard numbers obscure. And you don't even need that much of it.

Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford. – NN Group

When you understand why some parts of your product are difficult for your users, you can address them in creative ways, and craft an experience that makes them want to talk about it with their friends and family (it's the whole principle behind NPS measurements).


Qualitative data can point your design team in the right direction regarding specific pain points in the experience of your product. Is the difference between reading the music sheet for Beethoven's 9th Symphony and hearing it and seeing it a live orchestra perform it live.

Further reading

Subscribe to Ed Orozco

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.