use. A revision is thus in order. What you will find here is an updated edition of the first volume (with a new edition of the second currently in progress).
The text has been thoroughly reviewed by the authors—Ellen Van Velsor and Jean Brittain Leslie, who wrote the original edition, and John Fleenor, who joined the team for this version—and changes have been made to reflect current understandings in the field—for instance, new information on the proper use of norms and on “item banks” (a collection of instrument items that have been tested for reliability and validity) has been added, and the section on validity has been redone.
You will note that this work has been retitled and that it is no longer referred to as the first volume of a set. We wanted to make this guide, which can help practitioners begin to make sense of the complexity and proliferation of instruments, as visible as possible. The update of the second volume, also to be a stand-alone publication, will aid practitioners by providing in-depth descriptions of selected instruments, including some that were not included in the previous edition, and discussion of key issues in their use. The two can still, of course, be used together, and we invite people to do so.
Walter W. Tornow
Vice President, Research and Publication
Acknowledgments
We would like to express our appreciation to the many reviewers whose comments, ideas, and criticisms greatly improved the quality of this manuscript. These include David DeVries, Janet Spence, Russ Moxley, Michael Hoppe, Maxine Dalton, Lloyd Bond, and members of the Center for Creative Leadership’s Writers’ Group. We are also indebted to Clark Wilson, Walt Tornow, David Campbell, and Philip Benson for several ideas presented here.
Introduction
Many organizations are using 360-degree-feedback instruments to help their managers become better leaders. These instruments are designed to collect information from different sources (or perspectives) about a target manager’s performance. The principal strength of 360-degree-feedback instruments is their use of multiple perspectives. In most cases, the different sources of information (the raters) are the supervisor (or boss), the peers, and the direct reports of the target manager, although some instruments now allow managers to use internal and/or external customers as raters.
This report presents a nontechnical, step-by-step process you can use to evaluate any 360-degree-feedback instrument intended for management or leadership development. Although we have simplified this process as much as possible, it still will require some effort on your part—but effort that will pay off in terms of your having a high-quality instrument that best meets your needs.
The steps in evaluating a 360-degree-feedback instrument are laid out here sequentially. Yet all steps are not equal in complexity or importance. We suggest that you make the most critical decisions early in the process; in this way you can save some effort by eliminating instruments that don’t meet your needs in terms of content and that don’t pass muster when it comes to reliability and validity.
A checklist of the steps is included, for your convenience, at the end of this report. There the reader will also find a glossary of many of the technical words used here and a list of suggested readings.
STEP 1:FIND OUT WHAT IS AVAILABLE
The availability of 360-degree-feedback instruments is increasing at a tremendous pace. You can expect that there are as many promising instruments under development as there are good instruments for sale. So your first task should be to gain some knowledge of what is out there in order to choose the best possible sample of instruments to review.
In the short run, a good way to familiarize yourself with what is available is to search one of several guides that categorize and review instruments. Feedback to Managers, Volume II (Van Velsor & Leslie, 1991) is one such guide. It provides basic descriptive and technical data on 360-degree-feedback instruments available for use for management development. Other guides include Mental Measurements Yearbook (Conoley & Impara, 1995); Business and Industry Testing: Current Practices and Test Reviews (Hogan & Hogan, 1990); Psychware Sourcebook (Krug, 1993); and Tests: A Comprehensive Reference for Assessment in Psychology, Education and Business (Sweetland & Keyser, 1990). These can usually be found in the reference section of libraries. Over time, it may be useful as well to keep a file of the instrument brochures you obtain, because many of the directories are not published often enough to keep you updated on the very newest products.
STEP 2:COLLECT A COMPLETE SET OF MATERIALS
When you have identified several instruments you wish to evaluate, you need to obtain five pieces of information about each of them. You cannot make an informed decision using only a copy of the instrument or a promotional brochure.
Specifically, for each instrument you wish to consider, you should obtain the following:
• A copy of the instrument itself. If the instrument has one form for the individual to rate himself or herself and a separate form for the others who will rate him or her, get both.
• A sample feedback report (a representation of what the manager will receive after the instrument is scored). You can’t tell what type of feedback your managers will actually receive by looking at the instrument they will fill out. The sample could be a complete report, or it could be part of a report such as an example of the feedback display in the technical or trainer’s manual. Either type will do.
• A technical manual or other publication that outlines in detail the developmental and psychometric research done on the instrument.
• Information about any supporting materials that accompany the scored feedback, such as interpretive materials, development guides, goal-planning materials, and the like.
• Information about price, scoring, and whatever certification or training may be required to purchase or use the instrument.
It is not at all unreasonable to request this quantity of information. American Psychological Association guidelines (APA, 1985) require that this information be available upon request when an instrument is offered for sale.
In addition to seeking the recommended information, you should, through all the steps that follow, look for evidence of a commitment to continuous improvement on the part of each instrument’s developer. This is especially true if an instrument has been around for awhile. As we will discuss in the section on validity, research should always be in progress, because no instrument can ever be considered valid once and for all. Expect revisions in the scales over time; these are often made when additional validation studies have been completed. Expect revisions in the presentation of feedback as well; these are often made as the developer learns from the experience of those who have used an instrument. It is not uncommon for good instruments to have more than one copyright date, because even small revisions to content can cause changes in other areas, such as scale weightings or instrument norms.
STEP 3:COMPARE YOUR INTENDED USE TO INSTRUMENT CHARACTERISTICS
It is improbable that one instrument will meet the needs of all managers in an organization. Job demands differ somewhat by organizational level, and even at the same management level, skills that are needed for effectiveness may change over time. In addition, the dimensions on which managers are assessed should be in line with organizational
