is often a concern for institutions like banks and hospitals, which deal in sensitive information, or companies concerned with guarding certain types of intellectual property. If you’re testing a top-secret prototype, you obviously don’t want to let people access something from their home computer, where it could be saved or screen-captured. On the other hand, you might also be doing a study on users who would be secretive about sharing what’s on their screen—government employees, doctors, or lab technicians, for instance. Either way, you’ll want to test users in a controlled lab environment to keep things confidential, especially if what you’re testing is so hush-hush that you must have your users sign a nondisclosure form.
Inability to use screen sharing. You might also want to use a lab if your users are unable to share their screen over the Internet, for whatever reason. Some studies (of rural users, cybercafe patrons, etc.) may require you to talk to users who don’t have reliable high-speed Internet connections, who own computers too slow or unstable to use screen sharing services effectively, or who have operating systems incompatible with the screen sharing tools you’re using. These restrictions apply only to moderated studies, for which you need to see what’s on your users’ screens.
The need for special equipment. Depending on the interface you’re testing, you may require certain special software or physical equipment to run the study properly, which is most often the case with software that’s still under development. Getting users to install and configure tools to run elaborate software can be a pain (though that’s not unheard of), and requiring users to have certain equipment can make recruiting needlessly difficult.
The importance of seeing the user’s body. Some kinds of research will require you to study certain things about the user that are difficult to gather remotely. UX research has recently begun using eye-tracking studies, and for that kind of study, you’d need to bring the users to the eye-tracking device. Other studies might require you to attend to the participants’ physical movements, which may be difficult to capture with a stationary webcam. And then there are multiuser testing sessions, in which a single research moderator facilitates many participants at once; screen sharing is currently not well suited to sharing multiple desktops at once, though some tools (e.g., GoToMeeting) make it relatively painless to switch from one desktop to another. We want to emphasize, however, that for most studies, seeing the user at all is not actually important; we explain why in Chapters 5, “Moderating,” (see “Ain’t Nothing Wrong with Using the Phone”) and 10, “The Challenges of Remote Testing.”
Although these situations are all compelling reasons to conduct in-person research, part of what we want to demonstrate in this book is that remote research is very broad and adaptable, and even if a study is conducted in a lab, elements of remote methods can be adapted and incorporated to enhance in-person research methods. We’ll get to that in Chapter9, “New Approaches to User Research.”
Why I Went Remote
by Brian Beaver
Old habits die hard, but for any number of reasons—cost, convenience, international testing—a handful of former lab researchers have switched to remote methods and never looked back. Brian Beaver, award-winning creative director at Sony, explains why he decided to go with remote methods after many years of lab testing.
On Going Remote
I have quite a bit of experience, either organizing user research sessions or participating in them, especially at Sony. My research has been pretty varied and the outcomes are always interesting, but I’m a big fan of remote usability testing. It seems to give me the best bang for my buck.
I’d read about it a few years back and had done some work with Adaptive Path when it had a focus on usability. Adaptive Path referred me to Nate [Bolt, coauthor of this book], and when he shared his remote approach with me, I knew we were in sync because a lot of the pain points and skeptical raised eyebrows around results we’d obtained in previous lab testing instantly diminished with remote usability testing.
The pain points always involved recruiting. With a Web site you have such a diverse geographic base that it can be challenging to bring a core group of your users together in one location. Sony tends to be very protective of its customer information, and wouldn’t share it with a research company for the purposes of recruiting, so we’d have to take on that task ourselves, which was always painful.
The raised eyebrows were always about participant motivation and validity of the recruiting process and methodology. There were always questions: How valid are these findings? Are these real users? But when you’re intercepting users who are on your Web site in the middle of performing a task, those questions evaporate.
On Participating in Remote Research
In the past we’d invite our business partners or stakeholders to the lab, but it was difficult to get them to take time out of their day to travel to the lab, and it was a big production. But if they can just bring their laptop to a conference room down the hall and just be there to listen in, it’s fantastic. You’d get the same advantages if you had everyone available to go to the lab testing, and the level of engagement is a lot greater. By having a lot of stakeholders in the room, you get more diverse viewpoints, and the interaction between us observers and the moderator tends to be lively—we chat throughout that whole interview. The ability to observe and discuss things as they come up and then immediately give feedback to the moderator is really powerful.
Because we are involved in the research process, we’ve got our customers and the usability to consider on the one hand, and on the other hand we have a lot of business stakeholders who have strong opinions about how things should be done. So having everyone in the room watching the feedback and engaging with the process is really powerful. We were recently in the middle of a digital camera usability session and were asking the user to go through the features and content we have on the site, and the customer’s going through it and he’s like, “This all seems really impressive, but I really just want to know if it takes great pictures.” And you see this light bulb go off above the product marketing people’s heads. We’re so close to this that we have absolute myopia. It was a real eye-opening moment.
On Benefiting from Remote Methods
One study was about TVs and the TV shopping process. Sony has a broad line of TVs, somewhere around 9 to 10 different series, and each has a dozen size options, so you have a lot of choices. During the study there was an “Ah-ha!” moment, a phenomenon we haven’t seen before: people would often have half a dozen to a dozen sites already open when we contacted them, and they were seamlessly going between sites like Engadget, CNET, Sony, Circuit City, Best Buy, really taking advantage of browser tabs to cross-shop and gather information. We simply wouldn’t have gotten that insight from a lab environment because we wouldn’t have been intercepting people in their natural browsing environment; instead, they’d sit down, have the browser already open, and they’d go. So that behavior would have been completely missed.
The outcome was that, knowing that customers are looking not only for customer reviews but trustworthy, third-party editorial content, we’re actively pursuing ways to bring that content into the SonyStyle site, so that from within the interface they can access that info, instead of relying on the multi-tab approach. In the past, if a product was awarded an editor’s choice, we would have put that on the page as a badge of honor, but I doubt that we would have ever actually included the editorial alongside the product, if it hadn’t been for this study.
Advice for Those Considering Going Remote
If we’re talking about remote testing for Web sites, from my perspective it’s really a nonchoice. Having the benefit of intercepting users that are already coming to your site in order to perform a task already puts you so far ahead of the game because the motivation is there, you’ve got them captive, and you just gain so many
