The Case Against Remote Research
by Andy Budd
Even though we’re obviously firm advocates of remote methods, not all UX practitioners agree. Andy Budd is the creative director at Clearleft, a renowned London-based team of UX and Web design experts. Clearleft are the makers of Silverback, in-person usability testing software for interface designers and developers. Andy isn’t such a big fan of remote methods for moderated studies—here’s why.
Full disclosure: I’ve done very little remote testing, and the reason is that I’ve never found a credible need to do it. We’ve always found other ways of testing that didn’t require a remote approach. My issue is less about the negatives of remote testing and more about the positives of in-person testing.
Now, that’s not to say we test in a lab; I find that labs give a veneer of formality and scientific accuracy to studies, which they often don’t have. And testing labs and equipment are often more expensive, and tend to bog things down. So we take a grittier approach—just a meeting room and a video camera or some screen capture software.
We gain a lot of information by being in the room with people. They say 90% of communication is nonverbal. It’s about the cues in people’s tone of voice or posture. When you’re with a test subject, you pick up these signals more easily. With online video conferencing such as Skype, social conventions break down; you’re not able to read the cues that tell you when one person stopped talking or when it’s OK for another person to start talking. You get lag, and people talk over each other. Communicating remotely is difficult to do well, and it’s possible that it’s to do with our ability to use these new tools and technology. I wouldn’t be surprised—give it 20 or 30 years—when video conferencing becomes a norm and we’ve learned how to understand and read these subtle cues better. For now, there’s the potential to lose 90% of the information that’s coming through to you if you’re not testing in person.
On the Shortcomings of Remote Methods
Usability testing is all about empathy. It’s about creating a connection. That kind of empathy is difficult to create through Web conferencing, and it’s that gulf of miscommunication that makes it less attractive to me. I think there are instances where you should use remote moderated testing, often when it’s impossible to recruit users to a specific location. Recently, we were working on a project for a South American site. We wanted to speak to Brazilians, so initially we thought to do some kind of remote testing, but then realized that there was actually a large student and ex-pat Brazilian community here [in Brighton]. So instead we went to a Brazilian café and sat down and just chatted to actual Brazilian people who happened to be living in the UK. Some people said, “How can you possibly say that talking to Brazilian people in a Brighton café is exactly the same as in a favela in São Paulo?” But again, the difference is so subtle as to make little difference on the results, particularly when testing in small numbers. Sure, if you’re doing a scientific test and looking across very large sample sets, these minute effects are going to play a bigger role.
It also depends on what you are testing. There are some obvious cultural differences with the way people use the Web, but there are also universal habits: registering for a service, noticing positioning, etc. It’s unlikely that the Brazilian community in Brighton is not going to pick up something the community in Sao Paulo might.
On the Use of Technology in UX Research
People often try and find technological answers to human problems. A lot of the drive for remote testing is an attempt to find shortcuts. “Oh, it’s difficult to find test subjects, so let’s get technology to help us out.” Today I was reading an article where someone was saying, “How can we do remote ethnographic research? Can we get live cameras streaming?” And I thought, “Do a diary study.” Diary studies are probably the ultimate in remote ethnographic research. You don’t need a webcam streaming back to Mission HQ. People love tinkering with technology because it makes them feel like superheroes; it’s something to show off. I believe in human solutions; I think technology is often used as a crutch.
I think remote testing is still in its infancy. It’s based on the technology that’s available. It’s preferable to have lightweight technology that you can send to a novice user, double-click on it, and it opens and installs. But if you’re looking at an average computer desktop, which is now above 1024×768, and you also want to capture the person’s reactions, you want to send audio and video down that pipe as well—that’s a complicated problem. You need good bandwidth to do this really well. So then you create artificial problems, because you’re limited to people who have got pretty good tech and bandwidth. And so that would probably prevent us going and doing remote testing with somebody in a cybercafé in Brazil.
What I am kind of interested in is unmoderated [i.e., automated] remote testing, because it’s a hybrid between usability testing and statistical analysis or analytics. The benefit is that you can test on a much wider sample set. It complements in-person usability testing.
On the Purpose of User Testing
The point is to develop a core empathetic understanding of what your users’ needs and requirements are, to get inside the heads of your users. And I think the only way you can do that is through qualitative, observational usability testing. There are lots of quantitative tools out there, stuff that can tell you what’s happening, but it can’t necessarily tell you why it’s happening. We’ve all done usability tests where you watch people struggle and have a real problem doing something, and you can see they’re having a problem. Then they’ll go, “Oh, yes. It was easy.” We did a test where a user thought he’d purchased a ticket, and he hadn’t, but he’d left thinking he had. If he’d told you, “Yes, I’ve succeeded,” you would have been mistaken. Watching and observing what users do is very enlightening. Frankly, it’s easier for people to learn from direct experience than through analyzing statistics. There’s nothing like actually watching people and being in the same room.
Moderated vs. Automated
So, you’ve decided it’s worth a shot to try a remote research study. Feels good, doesn’t it? The first thing to know is that remote research can be roughly divided into two very different categories: moderated and automated research.
Figure 1.3
In moderated research, a moderator (aka “facilitator”) speaks directly to the research participants (see Figure 1.3). One-on-one interviews, ethnographies, and group discussions (including the infamous focus group) are all forms of moderated research. All the parties involved in the study—researchers, participants, and observers—are in attendance at the same time, which is why moderated research is also sometimes known as “synchronous” research. Moderated research allows you to gather in-depth qualitative feedback: behavior, tone-of-voice, task and time context, and so on. Moderators can probe at new subjects as they arise over the course of a session, which makes the scope of the research more flexible and enables the researcher to explore behaviors that were unforeseen during the planning phases of the study. Researchers should pay close attention to these
