Choosing 360: A Guide to Evaluating Multi-rater Feedback Instruments for Management Development

CHOOSING 360

A GUIDE TO EVALUATING MULTI-RATER FEEDBACK INSTRUMENTS FOR MANAGEMENT DEVELOPMENT

CHOOSING 360

A GUIDE TO EVALUATING MULTI-RATER FEEDBACK INSTRUMENTS FOR MANAGEMENT DEVELOPMENT

Ellen Van Velsor

Jean Brittain Leslie

John W. Fleenor

Center for Creative Leadership

Greensboro, North Carolina

The Center for Creative Leadership is an international, nonprofit educational institution founded in 1970 to advance the understanding, practice, and development of leadership for the benefit of society worldwide. As a part of this mission, it publishes books and reports that aim to contribute to a general process of inquiry and understanding in which ideas related to leadership are raised, exchanged, and evaluated. The ideas presented in its publications are those of the author or authors.

The Center thanks you for supporting its work through the purchase of this volume. If you have comments, suggestions, or questions about any CCL Press publication, please contact the Director of Publications at the address given below.

Center for Creative Leadership

Post Office Box 26300

Greensboro, North Carolina 27438-6300

336-288-7210 • www.ccl.org

©1997 Center for Creative Leadership

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of the publisher.

CCL No. 334

Library of Congress Cataloging-in-Publication Data

Van Velsor, Ellen

Choosing 360 : a guide to evaluating multi-rater feedback instruments for management development / Ellen Van Velsor, Jean Brittain Leslie, John W. Fleenor.

    p. cm.

Update ed. of: Feedback to managers, vol. 1. 1991.

Includes bibliographical references (p.    ).

ISBN 1-882197-30-5 [ISBN-13: 978-1-882197-30-9]

1. Organizational effectiveness—Evaluation—Methodology. 2. Feedback (Psychology). 3. Executives—Rating of. I. Leslie, Jean Brittain. II. Fleenor, John W. III. Morrison, Ann M. Feedback to managers. IV. Title.

HD58.9.V36 1997

658.4'03—dc21

97-18853

CIP

Table of Contents

Foreword

Acknowledgments

Introduction

STEP 1: FIND OUT WHAT IS AVAILABLE

STEP 2: COLLECT A COMPLETE SET OF MATERIALS

STEP 3: COMPARE YOUR INTENDED USE TO INSTRUMENT CHARACTERISTICS

STEP 4: EXAMINE THE FEEDBACK SCALES

STEP 5: FAMILIARIZE YOURSELF WITH THE INSTRUMENT-DEVELOPMENT PROCESS

STEP 6: LEARN HOW ITEMS AND FEEDBACK SCALES WERE DEVELOPED

STEP 7: FIND OUT HOW CONSISTENT SCORES TEND TO BE

STEP 8: ASSESS BASIC ASPECTS OF VALIDITY—DOES THE INSTRUMENT MEASURE WHAT IT CLAIMS TO MEASURE?

STEP 9: THINK ABOUT FACE VALIDITY

STEP 10: EXAMINE THE RESPONSE SCALE

STEP 11: EVALUATE THE FEEDBACK DISPLAY

STEP 12: UNDERSTAND HOW BREAKOUT OF RATER RESPONSES IS HANDLED

STEP 13: LEARN WHAT STRATEGIES ARE USED TO FACILITATE INTERPRETATION OF SCORES

STEP 14: LOOK FOR DEVELOPMENT AND SUPPORT MATERIALS

STEP 15: COMPARE COST—VALUE FOR THE PRICE

STEP 16: CONSIDER LENGTH A MINOR ISSUE

Conclusion

References

Suggested Readings

Glossary of Terms

Instrument Evaluation Checklist

Foreword

At the end of 1991, CCL published the two-volume Feedback to Managers (Volume I: A Guide to Evaluating Multi-rater Feedback Instruments; Volume II: A Review and Comparison of Sixteen Multi-rater Feedback Instruments). Since that time, there has been a notable increase of interest in multi-rater, or 360-degree, instruments: More are available; they are being used for a wider range of purposes; and much has been learned about their use. A revision is thus in order. What you will find here is an updated edition of the first volume (with a new edition of the second currently in progress).

The text has been thoroughly reviewed by the authors—Ellen Van Velsor and Jean Brittain Leslie, who wrote the original edition, and John Fleenor, who joined the team for this version—and changes have been made to reflect current understandings in the field—for instance, new information on the proper use of norms and on item banks (a collection of instrument items that have been tested for reliability and validity) has been added, and the section on validity has been redone.

You will note that this work has been retitled and that it is no longer referred to as the first volume of a set. We wanted to make this guide, which can help practitioners begin to make sense of the complexity and proliferation of instruments, as visible as possible. The update of the second volume, also to be a stand-alone publication, will aid practitioners by providing in-depth descriptions of selected instruments, including some that were not included in the previous edition, and discussion of key issues in their use. The two can still, of course, be used together, and we invite people to do so.

Walter W. Tornow

Vice President, Research and Publication