I work with a small development team building rich Internet applications for photo printing and sharing. My office follows an informal Agile development methodology, pushing small changes live as frequently as possible.
Last month I ran a conventional set of user tests with seven participants to evaluate our new application. We alternated and compared with a key competitor. Head-to-head, we didn't fare as well as I hoped. It was clear to everyone we needed a major push to enhance the app's usability.
My boss suggested an unusual course to improve the application: test a single participant every week on that week's updates. It's a methodology calculated to motivate the designers and developers, and keep them achieving measureable results on schedule. In addtion, just one "user day" per week leaves me enough time to fulfill all the interaction design of my regular responsibilities.
It's an unorthodox situation, but I have enjoyed the challenge so far of putting the weekly sessions in context. The first week, for example, our participant cruised through almost all of the tasks. Everyone was feeling great about how much we'd improved the app's usability! I pointed out, though, that we hadn't fixed some of the features that this one user quickly figured out. With that perspective I showed that the user proved to be exceptionally capable, and not necessarily representative.
It was an easy case to make, because this team has been great about observing the usability tests. In the past I've had trouble even getting project sponsors and stakeholders to observe one test session -- let alone developers. But this group I didn't even have to bribe with food! We've made some strong incremental usability improvements.
