« Dear (Name Withheld): How to Get Started | Main | User experience reading: expand your skillset »

May 29, 2008


Feed You can follow this conversation by subscribing to the comment feed for this post.

It's important to not throw the baby out with the bathwater, with regard to "usability".

(I'm never quite sure what "usability" means in any given context... here I think you mean "can the user accomplish their task" while leaving out all the touchy-feely aspects of design)

Trying to trace the success of any venture back to "usability" is like trying to trace a tornado in Kansas back to a low pressure area in California 3 days earlier. Did it cause the tornado? Would the tornado have happened without it? There are so many factors involved with the success of a product that it's hard to tease any of them out... for example Apple is a marketing juggernaut. How much of the iPod's success had nothing to do with design and instead had something to do with Apple's ability to convince people that whatever they produce MEANS good design? I don't know. Apple doesn't know either, I'm sure.

Some statements that I think we can say with confidence:
1. Usability, by itself, is rarely enough to result in a successful product.
2. Products can be successful without being useful by any reasonable definition.
3. Having a usabable product will increase its chances for success... but no one knows by how much in any given situation.

So what can we conclude? Being usable is better than being unusable. Just like being well-designed is better than being poorly designed. And being functionally stable is better than being buggy.

What really matters is return on investment. How much does it cost to improve usability versus improving the other factors in a product's success? And I don't think we're particularly close to being able to answer that question.

Thanks for your thoughtful response, Terry. In the context of this post, I define usability to include methodological practice of user research and user-centered design techniques including usability testing, but feel free to provide your own definition. In fact, "why can't we all agree on what the definition of usability is" was another theme raised at this conference.

I think it's pretty damning that after thirty years, user-centered design methodology doesn't have any big tech successes to point to. Less damning but still significant is that statistically valid usability testing can be a lengthy and expensive process, and it's difficult to conceive how usability techniques can provide the same revolutionary ROI that great design can.

Interesting post...

I don't buy Jared Spool's claim that UCD has not been shown to work -- I agree with Terry's points above about it being just one part of the process. Surely one can point to many problems that were caught during usability testing of a product. How is that not success?

Regarding Apple, isn't the difference simply that they stand in for their users during testing? So they're still testing usability but not in the formal sense with external users. It works because they're demanding users and their target customers are people like themselves.

Let's talk specifically about usability testing.

Has anyone "proven" that functional testing works? Most products ship with lots of unfixed functional defects. No product is perfect, and if a product ships without unfixed defects it usually means their test team is so small that they don't know about the defects that exist. Actually, let's make it simpler - regardless of whether a team KNOWS about their functional defects, most products ship with unfixed functional defects.

The question (and the art) is: At what point does finding and fixing more defects become less beneficial to getting the product in the marketplace? Obviously there isn't one answer to that question. Consider launching an "agile" SAAS product as beta (and thereby using your customers as your testers) in a situation where it's no big deal if the product crashes and getting product updates in is simple VERSUS trying to update a huge, legacy banking application that millions of people are using everyday for mission critical work. The answer to the question is very different in those two cases.

Now prove that functional testing works. It's a useless question. Just as useless as whether usability testing works. The interesting question isn't whether usability testing works, the interesting question is at what point does fixing more design problems become not worth the effort? And just like the legacy banking app scenario, the answer is different for every project.

Sometimes I think we're too smart and we try to make things harder than they are.

That said, people have been struggling with the answer to that functional test question for decades, so it's not like we're the only ones with this problem.

Kevin: Catching problems is not success because a top-down design process, including the key factors that Jared cites, has been shown to deliver revolutionary, commercially successful, great design. UCD has no successes to point to on a similar scale. Those key factors, BTW, are 1) a corporate culture that values design, 2) strong design vision from the top, and 3) site visits to customers.

Terry: I think our whole field needs to be less associated with usability testing, because it's expensive and time-consuming to do "right" (that is, get statistically significant results) and it's too incremental and reactive. I have a hard time imagining this equation working in a timely fashion:

mediocre design + good usability testing and iteration = great product release

On the other hand, this equation clearly has been effective:

good design + no usability testing = great product release

Absolutely it's true that updating a legacy enterprise app is a much different design and usability challenge than launching a new software-as-a-service app.

I'm a strange person to defend usability testing. As I've posted on my blog on several occasions (example: http://uxsoapbox.blogspot.com/2007/06/who-cares-about-finding-new-usability.html), I'm not a big fan of usability testing. I definitely agree with you that our entire field needs to be less associated with usability testing. I think spending time on user research and design is far better ROI than usability testing.

I just think this idea that no one has ever proven that product success can be traced to usability testing and therefore usability does not affect success is silly. Spool is presenting a false choice - should we focus on "well understood shared vision, frequent user feedback and a robust tool box of design tricks & techniques" or "design dogma, methodology and formal process"? He says the former is better than the latter. But that's like saying, "Mustard is better than ketchup, so we should choose mustard over ketchup."

The comments to this entry are closed.