results you’re going to get, it’s so much richer. If given the choice, I’ll never go back to lab testing again. And there’s the cost savings. Clearly, overall, it’s a less costly proposition. You avoid all the travel costs. There’s always a dud user in every batch of lab participants, and the great thing with usability testing is, if you start talking to someone you want to cut loose, it’s no harm; you can move on to the next person, as the recruiting form is literally filling up before your eyes.
What’s Remote Research GoodFor?
Again, most studies can successfully be done either in person or remotely, but, just as there are times when lab testing is more appropriate, there are also times when it makes more sense to use remote research methods.
Time-Aware Research
Remote research is more appropriate when you want to watch people performing real tasks, rather than tasks you assign to them. The soul of remote research is that it lets you conduct what we call Time-Aware Research (TAR). By now, UX researchers are familiar with the importance of understanding the usage context of an interface—the physical environment where people are normally using an interface. Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest. This is possible because of live recruiting (the subject of Chapter 3), a method that allows you to instantly recruit people who are right in the middle of performing the task you’re interested in, using anything from the Web to text messages. Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.
Consider the difference between these two scenarios:
You’ve been recruited for some sort of computer study. The moderator shows you this online map Web app you’ve never heard of and asks you to use it to find some random place you’ve never heard of. This task is a little tricky, but since you’re sitting in this quiet lab and focusing—and you can’t collect your incentive check and leave until you finish—you figure it out eventually. Not so bad.
You’ve been planning a family vacation for months, but you’ve been busy at work so you procrastinated a bit on the planning, and now it’s the morning of the trip and you’re trying to quickly print out directions between finishing your packing and getting your kids packed. Your coworker told you about this MapTool Web site you’ve never used before, so you decide to give it a shot, and it’s not so bad—that is, until you get stuck because you can’t find the freaking button to print out the directions, and you’re supposed to leave in an hour, but you can’t until you print these damn directions, but your kids are jumping up and down on their suitcases and asking you where everything is. Why can’t they just make this stupid crap easy to use? Isn’t it obvious what’s wrong with it? Haven’t they ever seen a real person use it before?
Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check. Time-awareness is an important concept, so we’ll bring it up again throughout this book to demonstrate how the concept relates to different aspects of the remote research process (recruiting, moderating, and so on).
Note
TAR Keeps You in the Right 1985
Remember that diagram in Back to The Future II? Doc argues that messing with time has sent the world crashing hopelessly toward an alternate reality where things are horrible—the “wrong 1985.” And that’s sort of what happens when you try to assign people a hypothetical task to do at a time when they may or may not actually want to do it: you’re meddling with their time, and it’ll create results that look like the real thing but are all wrong.
When you schedule participants in advance and then ask them to pretend to care, you’re sending your research into the wrong 1985. If you don’t want to create a time paradox—thereby ending the universe—you should do time-aware research.
Other Benefits of Remote Research
Here are some additional benefits of remote research.
Geographic diversity. Even if you do have a lab, the users you want to talk to may not be able to get to it. This is actually the most common scenario: your interface, like most, is designed to be accessed and used all around the world, and you want to talk to users from around the world to get a range of perspectives. Will Chinese players like my video game? Is my online map widget intuitive even for users outside Silicon Valley? Big companies like Nokia and Microsoft are often able to conduct huge, ambitious research projects to address these questions, coordinating research projects in different labs around the world, flying researchers around in first-class. If you don’t have the cash for an international Gorillas-in-the-Mist project, then remote research is a
no-brainer solution. If you can’t get to where your users are, test them remotely.
Ability to test almost anywhere. Remote research has comparatively minimal setup requirements and can reach anywhere that computers and the Internet can go: you can be anywhere; your participants can be anywhere. Lone-wolf consultants and start-up teams working out of cafés can have trouble finding the quiet office space they need to do in-person testing. If it’s too much bother to set up a proper lab, go remote; all you’ll need is a desk.
Some reduced costs. Beyond travel expenses, other costs associated with lab testing may be reduced or eliminated when you test remotely. With live recruiting methods, you can get around third-party recruiting costs, and because the recruiting pool is larger, you may not have to offer as much in the way of incentives as you might otherwise to attract enough participants. Because sessions are conducted through the computer, you can use relatively inexpensive software to replace costly testing accessories, such as video cameras, observation monitors, and screen recording devices. (Note, however, that the overall cost of a remote research study is often comparable to an in-person study for many reasons; see Chapter 10 for reasons why.)
Quicker setup. Closely related to the issue of money, as always, is time. Nearly all existing recruiting methods take many weeks. Recruiting agencies usually require a couple of weeks to gather recruits, and writing out precise recruiting requirements and explaining the study to them can eat up a lot of time. Getting users from your own mailing list can be faster and moderately effective, but what if you don’t have one? Or what if you’ve overfished the list from previous studies, or you don’t want to spam your customers, or you’re looking to test people who’ve never used your interface or heard of your company before? In any of these cases, recruiting your users online makes a lot of sense, since it allows you to do your recruiting as research sessions are ongoing. (We teach you how to do all this in Chapter 3.)
Context-dependent interfaces. Some interfaces just don’t make any sense to test outside their intended usage environment. If you need users to have their own photos and videos to use in a video editing tool, having them bring their laptop or media to a lab will be a tremendous hassle. Or, let’s say you’re testing a recipe Web site that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment.
When to Go Remote
If you have the gumption, you can test almost anything remotely. There are ways to get around nearly any obstacle, but the approach you take is all about what’s most practical
