The numbers didn’t lie—the new search engine, first measured on October 3, was performing worse on each metric than the old engine!
John and his project manager now had the proof they needed to convince the IT folks that the new engine’s poor performance wasn’t just something that came to him after a hard night of partying. The problem was real and serious, dire enough that he thought people could lose their jobs if the new search engine launched as is. IT responded accordingly. While the staff still were obligated to make the same launch deadline, they eliminated some planned features in favor of fixing the problem. Over the coming weeks, they identified the sources of the problems. The primary culprit—a misconfigured configuration file that was missed by Vanguard’s search engine consultant—was fortunately a fairly simple fix. And it wouldn’t have been detected without site search analytics.
You can see how their work progressed to the point where, by launch, they’d at least come close to getting the new search engine (as of October 16) to work about as well as the old one for each of the eight metrics, as shown in Figure 1-5.
Figure 1-5. As the launch date approached, the new search engine’s performance improved dramatically—to the point where it had caught up with the old engine’s performance.
So John’s gut reaction was validated, and he had the numbers to back up his argument that some hard work was in line before the new engine launched. The search experience was measured, a problem was recognized and identified, the search engine was fixed, firings were averted, and egg-on-face was avoided. Since the launch, Vanguard has continued to monitor these metrics and fine-tune the engine’s performance accordingly. It’s now performing much, much better than the original search engine. And that’s where our happy story ends.
Moral of the Story: Be Like John
There’s an important takeaway from this case study: that UX practitioners and other designers should not only pay more attention to the numbers, but it’s their responsibility to employ quantitative approaches to research and evaluate the user experience. If no one at Vanguard had taken on this responsibility, the entire project might have failed miserably.
And though it’s not a lesson, there’s another important point worth remembering: this is just the tip of the SSA iceberg. There’s much more that can and should be done with your site’s search query data.
In this book, I’ll cover many of the ways you can use SSA to better align your site with your business strategy, and I’ll show you how SSA can be used to diagnose problems with your site’s content, metadata, navigation, and search system performance. I’ll do my best to help you integrate SSA, which is an inherently data-driven way to analyze user behavior, into traditional, more qualitative user-centered design methodologies. SSA is a missing link and a goldmine of untapped riches for all kinds of designers. I hope my book will serve as a toolkit to help you mine the data and, like John, achieve a truly better user experience.
Chapter 2. Site Search Analytics in a Nutshell
What Is Site Search Analytics?
George Kingsley Zipf, Harvard Linguist and Hockey Star
Ways to Use SSA (and This Book)
In the last chapter, I showed how Vanguard used (and continues to use) site search analytics to measure, monitor, and optimize its search system’s performance. Not to mention that it improves the overall user experience, as well as saves money, promotes jobs, and avoids disaster. Now it’s your turn to give it a try. The bulk of this book will teach you the nuts and bolts of SSA. Starting with Chapter 3, I’ll show you how to analyze your data, gain actionable insights, and put them to good use so your organization can enjoy some of the same benefits as Vanguard. But before we go deep, we’ll go broad. In this chapter, I’ll briefly cover the nuts-and-bolts aspects of SSA: what it is, how it works, and why you would use it. Think of this chapter as an introduction to SSA in 20 pages or fewer.
What Is Site Search Analytics?
Site search analytics is, at its simplest, the analysis of the search queries entered by users of a specific search system (see Figure 2-1 and Figure 2-2). What did they search? What do their searches tell you about them and their needs? How did their searches go? Does their experience suggest fixes or improvements to your site? Or does it raise follow-up questions to pursue through other forms of user research?
Note that in this book, we’re exploring the searching performed on a Web site or intranet. We are not covering how people search the entire Web using Google or another search engine. There are certainly parallels, but as you’ll see in the table in Figure 2-3, they’re not the same; Referral Queries of the Michigan State University site came from Web search engines like Google; Local Queries were executed on MSU’s own search engine.
Figure
