planning of the study.
Automated (or “unmoderated”) research is the flip side of moderated research: the researcher has no direct contact or communication with the participant and instead uses some kind of tool or service to gather feedback or record user behaviors automatically (see Figure1.4). Typically, automated research is used to gather quantitative feedback from a large sample, often a hundred or more. There’s all sorts of feedback you can get this way: users’ subjective opinions and responses to your site, user clicking behavior, task completion rates, how users categorize elements on your site, and even your users’ behavior on competitors’ Web sites. In contrast to moderated research, automated research is usually done asynchronously: first, the researcher designs and initiates the study; then the participants perform the tasks; then, once all the participants have completed the tasks, the researcher gathers and analyzes the data.
Figure 1.4
When to Use Which Remote Method
There’s plenty of overlap between automated and moderated methods, but Table 1.5 shows how it generally breaks down.
Table 1.5 Moderated vs. Automated Research
Moderated research is qualitative; it allows you to observe how people use interfaces directly. You’ll want a moderated approach when testing an interface with many functions (Photoshop, most homepages) or a process with no rigid flow of tasks (browsing on Amazon, searching on Google) over a small pool of users. Since it provides lots of context and insight into exactly what users are doing and why, moderated methods are good for “formative research” when you’re looking for new ideas to come from behavioral observation. Moderated research can also be used to find usability flaws in an interface. We cover the nitty-gritty of remote moderated research in Chapter 5.
Automated research is nearly always quantitative and is good at addressing more specific questions (“What percentage of users can successfully log in?” “How long does it take for users to find the product they’re looking for?”), or measuring how users perform on a few simple tasks over a large sample. If all you need is raw performance data, and not why users behave the way they do, then automated testing is for you. (Suppose you just want to determine what color your text links should be: testing every different shade on a large sample size to see which performs best makes more sense than closely watching eight users use three different shades.) Also, some automated tools can be used to gather opinion-based market research data as well, so if you’re looking for both opinion-based and behavioral data, you can often gather both in a single study. And certain conceptual UX tasks, like card sorting and A/B testing, are well supported by automated tools. See Chapter 6 for a more thorough look at the various automated research methods.
You don’t necessarily have to choose between moderated and automated testing, or even between lab and remote methods. You can even conduct multiple studies on the same interface, using the findings from one study to add nuance to another. That’s probably excessive for the average study, but for really large-scale projects where you just want to gather every bit of information you can (a new version of a complex software program, an overhauled IA, etc.), being comprehensive can’t hurt.
After reading this chapter, you should have a good idea of whether or not remote research suits you. Give it a try—if it’s not your thing, you can always go back to lab testing. We won’t tell anyone.
Chapter Summary
Do a lab study when you need to use special equipment, keep the interface 100% secure, see the user’s physical movements, or when you can’t use screen sharing tools.
Remote research has its own strengths, the greatest of which is that it enables Time-Aware Research, in which you observe users performing tasks you’re interested in observing right at the moment they’d naturally perform them.
Remote methods can also give you greater user diversity, cut travel costs, allow you to test from anywhere, be quicker to set up, and can be used to test context-dependent interfaces that wouldn’t make sense in a lab.
Currently, remote methods work best for testing functional computer interfaces, prototypes, wireframes, design comps, and mockups. You can test physical products with the right setup, but it’s slightly harder.
There are two kinds of remote research: moderated and automated. Moderated research has the researcher communicating with and observing users as they perform tasks; automated research has researchers using online tools and services to collect behavioral data from users automatically.
Generally, moderated research is good for collecting rich, qualitative behavioral data from a small sample, while automated research is good for collecting quantitative data over a larger sample.
Time-Aware Research.
Recruit someone who’s in the middle of a task.
Observe their behavior.
Chapter 2
Moderated Research: Setup
Gearing Up: Physical Equipment
Drafting the Research Documents
Let’s get down to setting up a typical moderated one-on-one study. We’ll walk you through gathering all the equipment and software you’ll need and explain how to prepare for your first research session. To keep things moving, we’ll stick to bare-bones basics in this chapter, explaining the simplest way to set up a generic moderated research session. Later, you’ll learn about other tools, approaches, and strategies you can use to develop a study that best suits your particularneeds.
Gearing Up: Physical Equipment
Even though you don’t need a lab to do remote research, you’ll still need some equipment to make calls, see your users’ screens, and record the sessions, and there are also a few tools that can make your life easier. Fortunately, you can find a lot of these lying around most offices (see Figure 2.1).
Figure 2.1
