Edit: Please note the chart at the left is to satirize the "guru factor" in user experience. It does not represent my thoughts, and it's not taken from any site listed here.
The recent redesign of information architecture site Boxes & Arrows puts heavy emphasis on reputation ratings. Online ratings are applied to all registered users on the site (authors and commenters) as well as site articles. I've almost always agreed with the article ratings on Boxes & Arrows, so it seems like a useful feature to me.
Ratings of people, however, is less straightforward. The rating system is heavily weighted towards content contributors, in effect making them "superusers" with scores or hundreds of reputation points. Whenever they post their opinions in comments, their status is obvious. Although rating points are meant to be a broad range, the system as implemented creates a clear binary divide among members.
Taking the ratings trend a step further is a new site, UX Zeitgeist. They attempt to rank books, user experience topics, and even people. Its publishers "believe our ranking system is wildly, brilliantly innovative and one of the most useful features of UX Zeitgeist." Perhaps it is, but they beg the question why is it useful to rank people at all.
UX rating sites bring to my mind the concept of A-list bloggers. Popularity equals wisdom and power in the blogosphere. But the problem is, you can't accurately judge the value of information in a single swift rating. An article useless for one person with one kind of goal, is priceless to the next person with different goals. A-list ratings create a winner-take-all information ecology that can obscure a long tail of personally relevant content. Instead, user experience experts ought to professionalize and promote the field with a system that less resembles the United States homeland security threat indicator.
Scratch the surface of UX Zeitgeist, and there's one immediate hole in the person rankings. Where's Jakob Nielsen? Perhaps he didn't reveal his favorite books -- participation that appears to be required for listing. If so, I think opting out of this attempted Who's Who of UX is laudable.
In one fell swoop, they've equated user experience with threat levels and reinforced the stereotype that we're uptight academics. Even as someone who has a Ph.D., I find that category label a bit on the arrogant side. I've known plenty of stupid people with doctorates (and at least a few marketing people and designers who were geniuses).
Posted by: Dr. Pete | September 10, 2007 at 07:42 PM
Hi Joshua, as its lead designer, I can answer any and all questions about UX Zeitgeist. :-)
First, I should clear up something: UX Zeitgeist does NOT rank people. Never, ever. Our assumption is that it would be a real turn off, reducing participation. Trust me, we debated this a zillion times, and went back and forth before settling on not ranking people. But we DO rank UX books and UX topics. Maybe you're seeing the numbers associated with the list of UX people as rankings that we've come up with, but they actually come from other sources, like UX publication mentions, blog mentions, and so forth. The don't come from us.
As far as Jakob not participating, it's because he hasn't been nominated to participate. Not yet, at least. And when he is, he might chose not to participate. We're not trying to be the "who's who" of UX, BTW; alongside the Steve Krugs, you'll see people that you've never heard of. When we get deeper into our beta phase, we'll actually be issuing about 200 or 300 invitations to members of the UX community to participate. They'll invite more people, and so on. We're just not ready to do that quite yet, as we have to finish working out the kinks in the system that manages our editorial queue (each person's response requires some editing). Check back in a month.
Before passing judgment, you might take a closer look (and wait for us to get out of beta)! In the meantime, I'll put you on the invitation list if you send me your email address. Thanks Joshua.
Posted by: Lou Rosenfeld | September 10, 2007 at 07:59 PM
Dr. Pete: To clarify, the "threat level" graphic is by me, a little satirical joke about the field and its gurus. You won't find it on the UX Zeitgeist site.
Lou Rosenfeld: When I look at your site's UX People Index page, I see a long list of practitioners given scores in various categories. When you sort by "UX Mindshare," you can see that it's a composite index that ranks the people on the list. Ratings seem to be whole point of the list.
Perhaps I'm missing the point of the list -- if so, you might choose to make its purpose clearer in a future revision of your beta.
Thank you both for commenting.
Posted by: Joshua | September 10, 2007 at 10:33 PM
Hi Joshua; it's true that there are UX Mindshare scores, but it's not a composite index at all. It's essentially the # of mentions of each person's name in all UX-related search results (via Yahoo) that we could find. (Mouse over the '?' and it's explained.) We've even explained it: http://www.rosenfeldmedia.com/zeitgeist/blog/2007/05/how_do_we_determine_ux_mindsha.php
If we wanted to do what it sounds like you think we're doing, we'd take all these separate numbers for each person, weight them, and combine them for an overall score, as we do for books and topic.
Hope that clears it up for you; ping me if you need more information.
cheers
Posted by: Lou Rosenfeld | September 10, 2007 at 11:26 PM
Since I am not rated highly on the scale, the system is clearly flawed. =)
Lou: I think what you are doing with Rosenfeld Media is great and needed, and I think the UX Zeitgeist section of the website is an interesting concept, particularly with regard to the topics and books. But regarding people, I'd visited the Zeitgeist page before Joshua's blog post, and I walked away with the exact same impression that he did... that you were basically trying to rank UXers based on some algorithm that calculated "reputation". Like Joshua, I thought this was slightly icky, but at the same time I found it useful because I was trying to find blind spots in my user experience knowledge and by looking at people who were ranked highly in mindshare but whose name I didn't recognize, I was able to quickly find my gaps.
But the bottom line is - I bet the vast majority of people who visit the site will leave with the same impression as Joshua and me. If that's not the impression you intend to be giving, I would give some thought to your design. For example, in your comments above you try to make it clear that you aren't treating people like you do books or topics. But then in the link you give about how you calculate mindshare, you say, "Our intent with UX Mindshare is to show how much penetration each topic or person has in the broader world of UX." I think the subtle distinctions you are making above will be lost on most users.
Consider how people would react differently to the design if instead of people having a rating for "UX Mindshare" they had a rating for "Yahoo Citations". Same algorithm, but "Yahoo Citations" is a dispassionate, calculated numeric value while "UX Mindshare" attaches meaning to the number (and forces people to investigate to discover the algorithm). I'm not saying this should be your design, just making a point that if you were consulting on someone else's website and they were arguing that users are consistently getting a mistaken impression from the website but it's not a problem because they're wrong, I suspect your response would not be "Let's add documentation to the website so fix the problem."
Posted by: Terry Bleizeffer | September 11, 2007 at 08:46 AM
Wow, remind me not to post my end-of-work-day rants before reading things twice :) Kudos to you, then, Joshua, for creating the perfect satire of exactly how too many people view our field.
Posted by: Dr. Pete | September 11, 2007 at 08:53 AM
Terry, that's great input, and I think you've likely solved the problem with your suggestion. I need to mull it over some, but I think we'll follow your advice. Thanks for the constructive input!
Posted by: Lou Rosenfeld | September 11, 2007 at 10:36 AM
Josh,
I have to say that I almost didn't read the article, because I was so put off by your tongue-in-cheek scale. Maybe you should put a question mark at the end of the title, to indicate that it's satirical. I really despise the hierarchical nature of it, and the notion of labeling people's skills and knowledge, based on a single descriptor.
You got my attention,
:)
jen
Posted by: Jen McGinn | September 11, 2007 at 11:04 AM
Thanks for the comments.
Terry, excellent comment and examples to illustrate the point.
Jen: The guru factor seems so prevalent in the field, yet it also can strike a nerve. Perhaps it's because UX folks work with information hierarchies, so they're especially attuned to them.
Clearly though, you're not the first to be put off by the message of the graphic. :-)
Posted by: Joshua | September 11, 2007 at 11:27 AM
I occasionally go to the site. The ranking never weighed in heavily with me as because that wasn't my intent as a user. Quite honestly, I liked it because I'm nosey in nature (a real shocker there...) and I get to browse through the names of people I've met at conferences OR that I've seen posting on ListServs. I can see what they like/dislike, who they are, etc. It's a directory of people in one place!
What you saw is quite different and the dialogue that came as a result of your blog entry generated great ideas...It's hard to strike the right balance.
I'd say join it - you're one of the people I follow - I think you've got a lot of insight to offer!
Posted by: Manya | September 11, 2007 at 05:19 PM