Jakob Nielsen's Alertbox, January 22, 2008:
Usability ROI Declining, But Still Strong
Summary:
The average business metrics improvement after a usability redesign is now 83%. This is substantially less than 6 years ago, but ROI remains high because usability is still cheap relative to gains.
Six years ago, we conducted a survey of design projects and found that, after redesigning for usability, the average improvement in key performance indicators (KPI) was 135%. We've recently completed a new survey, and this time the average improvement was 83%. The return on investment (ROI) for usability is now smaller, since the cost has remained approximately constant, as the benefits have decreased.
In both cases, we arrived at the improvement estimates after eliminating outliers — that is, design projects that achieved a factor of 10 or more in the desired metrics after their redesign. About 12% of projects get such huge improvements, but to be conservative, I focus on the majority of projects with smaller (but still substantial) gains.
Our full research report offers detailed redesign case studies, before/after screenshots, and an in-depth discussion of specific measurements.
Typical business metrics include:
- Conversion rates, such as sales or lead generation
- Traffic numbers, such as page view statistics
- User performance, such as the time needed to perform key tasks
- Target feature usage, such as the number of users who click a link to crucial information
Declining ROI
Six years is long enough that we might expect to see a difference in improvement metrics across the two surveys — as indeed we did. The following chart compares the results distribution from both editions of our Usability ROI report. The different case studies are arranged across the horizontal axis (sorted by increasing metric improvements). The vertical axis shows each case study's recorded improvement.
We use a logarithmic scale because doing so is necessary to depict the huge improvements in some of the case studies. (We plot cases with no improvement as having had a 1% improvement, because we can't show zero on a logarithmic scale.)
The red curve (current data) is mostly below the blue curve (6 years ago), except for a short stretch in the upper range. This means that the newer case studies generally had smaller usability improvements than the old ones, but that both surveys included a few case studies with equally huge improvements.
The probability that the difference between the two datasets is due to random fluctuations — rather than an actual difference — is p=8%. In most studies, p should be less than 5% before we accept a significant difference between two datasets. But 8% is close enough to 5% that I conclude that the difference is marginally significant.
In other words, the expected improvement from redesigning for usability is mostly likely smaller now than it was six years ago.
(This study didn't focus on intranets, but our recent results on expected improvements in intranet metrics after redesigning for usability show the same trend: we've already cashed in the easy, huge advances, and so new projects will realize less spectacular gains. In the past, a mid-sized company would get $14.8 M/year in productivity savings from improving its intranet usability from poor to good; now it "only" gets $5.4 M/year. Of course, since the likely cost is about a million, doing so is still worthwhile.)
Why Usability Is Improving Less
The expected improvement from usability is smaller than it used to be for two reasons:- We have now harvested most of the low-hanging fruit from the truly horrible websites that dominated the lost decade of Web usability (approximately 1993–2003). In those early years, Web design was abominable — think splash screens, search that didn't find anything, bloated graphics everywhere. The only good thing about these early designs was that they were so bad that it was easy for usability people to be heroes: even the smallest study would inevitably reveal several immense opportunities for improvement.
- Usability budgets have not increased substantially, even as the Web has gotten better. As the full report discusses in detail, during the last decade, the share of project resources allocated to usability has held steady at around 10% in those enlightened companies that include usability in their design lifecycle. Yes, many more companies do usability now than ever before. However, individual projects don't see much more funding, even though they're now challenged with identifying a higher level of design improvements.
Can we double again and take expected conversion rates to 4%? Most likely, yes. Some sites already achieve this, just as some well-designed e-commerce sites were getting 2% conversion rates and better back in 2000.
Once we reach 4%, can we double again? Probably. It should be feasible to reach the state where a well-designed site has an 8% expected conversion rate, and the very best sites get a bit more. Going from 2 to 4 to 8 percent might take another decade for each doubling.
It's doubtful that average conversion rates will go much beyond 10%, for the simple reason that Internet users like to compare multiple sites before they buy. Also, many users simply research possible purchases or have a general interest in items without being anywhere near the point where they're actually shopping to buy.
The Loyalty Decade
The formula for website success is:B = V × C × L
Where
- B = amount of business done by the site
- V = unique visitors coming to the site
- C = conversion rate (the percentage of visitors who become customers); note that the concept of conversion applies not only to ecommerce sites, but to any site where there is something you want users to do
- L = loyalty rate (the degree to which customers return to conduct repeat business)
In a multiplication, if you want to increase the outcome by a certain percentage, you can increase any of the factors by that percentage. It doesn't matter which factor is increased — the result will be the same.
Thus, to double a site's business, you can double the number of unique visitors. However, this would be very expensive, requiring that you more than double the advertising budget (assuming you're already advertising under the most-promising keywords, and thus need to buy traffic from less promising or more expensive sources).
Alternatively, you can double the conversion rate and achieve the same business improvement. It's still fairly cheap to double conversion rates, though it's not as cheap as it was, say, in 2000. According to our survey, spending 10% of your development budget on usability should improve your conversion rate by 83%. You can probably double the conversion rate by spending less than 15% of your development budget. In most cases, it's far cheaper to use 15% of your development budget than to more than double your advertising budget.
However, as conversion rates double — and later, double again — we'll eventually reach the point where the usability investment for continued improvements becomes much more expensive than current budgets. We'll then need to discover ever-more esoteric ways of satisfying customers, and those ways are unlikely to emerge from the cheap and fast user-testing approaches that dominate today.
We'll soon reach the point where increasing the loyalty rate is the best way to achieve substantial improvements in website business metrics. Whereas we might aptly call the period 2000–2010 the conversion decade for website usability professionals, 2010–2020 will be the loyalty decade.
Sadly, researching loyalty issues and testing design ideas for improving loyalty require more expensive usability methods than those that helped us dispatch impediments to user conversion from the user-hostile sites of the past. For example, usability professionals will have to conduct field research to supplement their lab-based testing. And they'll have to go deeper on even simple user tests to adequately understand the next generation of user needs.
So, as the expected improvement percentages decline and the required usability budgets increase, what will happen to ROI? By definition, ROI relates to two numbers: the return (i.e., the improvement) and the investment (i.e., the budget). As both become less favorable, ROI invariably declines.
Luckily, current usability ROI is so stupendously big (spend 10% to gain 83%) that it can decrease much more and still be a favorable proposition for business executives. Eventually, of course, we'll reach the point where further usability investments will have lower ROI than other ways of spending the company's money. But that point is probably 20 to 30 years into the future.