Evaluative Inquiry for Learning in Organizations
Publication Year: 1999
How does evaluative inquiry contribute to organizational learning? How can we practice evaluative inquiry in ways that maximize individual and team learning? This book provides a data-based approach to organizational learning and change and focuses on the use of evaluative inquiry processes with organizations rather than across large-scale, multi-site programs. It contains four illustrative case studies, interview extracts, strategy plans and flow charts, diagrams and advice boxes that consultants can use for implementing their own training and development sessions.
- Front Matter
- Back Matter
- Subject Index
- Chapter 1: Evaluative Inquiry and Organizational Change
- Chapter 2: Learning in Organizations
- Chapter 3: Evaluative Inquiry Learning Processes
- Chapter 4: Focusing the Evaluative Inquiry
- Chapter 5: Carrying Out the Inquiry
- Chapter 6: Applying Learning
- Chapter 7: Building the Infrastructure for Evaluative Inquiry
- Chapter 8: The Practice of Evaluative Inquiry
Copyright © 1999 by Sage Publications, Inc.
All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.
SAGE Publications, Inc.
2455 Teller Road
Thousand Oaks, California 91320
SAGE Publications Ltd.
6 Bonhill Street
London EC2A 4PU
SAGE Publications India Pvt. Ltd.
Greater Kailash I
New Delhi 110048 India
Printed in the United States of America
Library of Congress Cataloging-in-Publication Data
Preskill, Hallie S.
Evaluative inquiry for learning in organizations / by Hallie
Preskill and Rosalie T. Torres.
Includes bibliographical references and index.
ISBN 0-7619-0453-0 (acid-free paper)
ISBN 0-7619-0454-9 (pbk. : acid-free paper)
1. Organizational learning. 2. Inquiry (Theory of knowledge) I. Torres, Rosalie T. II. Title.
HD58.82 .P74 1998
00 01 02 03 04 05 8 7 6 5 4 3 2
Acquiring Editor: C. Deborah Laughton
Editorial Assistant: Eileen Carr
Production Editor: Denise Santoyo
Designer/Typesetter: Danielle Dillahunt
Cover Designer: Candice Harman
List of Strategies, Exhibits, and Figures[Page xi]StrategiesExhibits
- 1.1 Transitioning From the Industrial Era to the Knowledge Era 8
- 1.2 Characteristics of Learning Communities 15
- 2.1 Dispositional Ideals for Effective Team Functioning 28
- 2.2 Characteristics of Effective Teams 30
- 2.3 Self-Monitoring Questions for Teams 32
- 2.4 Characteristics of Ineffective Teams 33
- 2.5 Resolving Team Conflicts 36
- 3.1 Benefits of Dialogue 55
- 3.2 Benefits of Reflection 57
- 3.3 Benefits of Asking Questions 61
- 3.4 Benefits of Identifying and Clarifying Values, Beliefs, Assumptions, and Knowledge 67
- 4.1 Benefits of Focusing the Inquiry 91
- 5.1 Formats for Communicating and Reporting 111
- 5.2 Benefits of Carrying Out the Inquiry 125
- 6.1 Challenges to Successful Implementation of Action Plans 147
- 6.2 Benefits of Applying Learning 150
- 7.1 Organizational Culture That Supports Evaluative Inquiry 157
- 7.2 Organizational Leadership That Supports Evaluative Inquiry 162
- 7.3 Communication Within Organizations That Facilitates Evaluative Inquiry 169
- 7.4 Organizational Systems and Structures That Facilitate Evaluative Inquiry 172
- 4.1 Focusing the Inquiry 73
- 5.1 Sample Format for a Data Collection Plan 114 [Page xiii]
- 5.2 Sample Format for a Communicating and Reporting Plan 115
- 6.1 Applying Learning From the Inquiry 133
- 6.2 Sample Format for an Action Plan 145
- 8.1 Evaluative Inquiry Compared With Organization Development 185
- 8.2 Outcomes Associated With Evaluative Inquiry for Learning 190
Writing a book such as this is not easily accomplished without the support and encouragement of many people. We first wish to thank those we interviewed in our field research. They gave generously of their time and provided us with critical insights into the relationship between organizational learning theory and practice. We have the greatest respect for them, as they represent the true agents of learning and change in organizations. We also wish to thank several of our colleagues who, over the last few years, gave us feedback on our ideas as they evolved. In particular, we would like to thank Michael Q. Patton, Brad Cousins, Valerie Caracelli, Larry Braskamp, Lyn Shulha, Jennifer Martineau, and our anonymous reviewers.
Once again, our editor, C. Deborah Laughton, gave unwavering support and guidance throughout the writing of this book. Her untiring good cheer, belief in the importance of this topic, and concrete suggestions made the writing process all the more enjoyable and worthwhile.
We have saved for last those individuals whose daily support meant being able to complete this book. To Stephen and Benjamin—your patience, sense of humor, and understanding during the last 2 years made writing this book possible.[Page xvi]
The amount of organizational change occurring today is unprecedented. The burgeoning literature that has provided advice, empirical research, case studies, and evaluations on this subject suggests that today's organizations (a) are context bound and driven, (b) are made of up people who experience change differently, (c) have many cultures, (d) include formal and informal communication structures, and (e) are politically charged.
Continuous organizational change is resulting in less organizational stability and a redefinition of who we are and what we do in the workplace. The traditional structures that have given us a feeling of solidity and predictability have vanished. This shift has placed a greater emphasis on the need for fluid processes that can change as an organization and its members' needs change. Instead of the traditional rational, linear, hierarchical approach to managing jobs, which focused on breaking down job tasks and isolating job functions, tomorrow's jobs will be built on establishing networks of relationships. Workers will require listening, communicating, and group facilitation skills to get their work done. As a result, more and more organizational charts, job descriptions, and functional specializations will go by the wayside. [Page xviii]Organizations that make it through this transition will provide “structures that promote the flow of ideas, build trust and a unity of purpose, tap the energy and creativity of the workforce, and translate ideas into new products, processes and services” (Goddard, 1990, p. 4).
The sum of these changes is that tomorrow's organizations will (a) accomplish their work through multidisciplinary teams; (b) have permeable boundaries; (c) be focused on mental tasks; (d) be participative, diverse, and innovative; (e) support a professional culture of commitment and results; and (f) value peer-to-peer relationships. Organizations that have these characteristics will develop the capacity for self-renewal through the interaction of their members and employ a systems thinking approach to how work is accomplished. Such organizations will develop a culture of continuous learning that influences the way the organization approaches its goals—whether they be increasing student achievement scores, improving the satisfaction of clients and customers, placing higher numbers of jobless people in decent-paying jobs, or improving the profit margin on a particular product.
As organizations adapt to new economic and societal requirements, we believe that evaluative inquiry can be a guiding force for organizational growth and success. When we (the authors) started working in the evaluation field some 20 years ago, program evaluation in the United States was still in its adolescence. Having been born in the early 1960s through federal grant requirements, evaluations typically were conducted by social science researchers employed in universities and government agencies. These evaluators were esteemed for their perceived unbiased, objective perspective. In most cases, they had been trained to use a strict social science positivist approach, emphasizing experimental designs and replicable, generalizable findings. They typically developed psychometrically valid instruments, collected data, conducted statistical analyses, and submitted a final report with recommendations to primary stakeholders.
The evaluation's role was to help reduce uncertainty by providing data and information for the purpose of decision making. It was assumed that organizations knew how to use such information and that they would do so, making the appropriate changes. Schwandt (1997) characterizes this mode of problem solving as an “operational intelligence” approach to evaluative activity:[Page xix]
This kind of intelligence is instruction on the status of means and means-end reasoning; it is directed at helping a client get to there from here…. Evaluation aimed at teaching operational intelligence seeks to improve the rationality of practitioner and decision maker's practices by applying knowledge that evaluation has produced. (p. 79)
A rationalist approach to evaluation inquiry assumes that the organization is an independent actor in its environment, that there is only one answer to the question, that everyone thinks rationally on behalf of the organization and will arrive at the same conclusion, and that full implementation follows the discovery of the one best strategy. This approach has also focused on individual learning and short-term solutions rather than helping organizations learn about their practice and the values on which that practice is based.
Although this kind of thinking has served many organizations well over the years, it is inconsistent with the goal of developing a community of practitioners who inquire daily about their progress and use their learning to improve themselves and the organization. We are increasingly hearing from management experts that the key to organizational survival is, and will be, the intellectual capital of its employees. We're not talking just about business organizations—this relates to every kind of organization in every sector of society. Just as organizations need to transform themselves to survive, so must evaluation theory and practice evolve.
Thus, the question becomes: How can evaluative inquiry contribute to the development, maintenance, and growth of organizations in a dynamic, unstable, unpredictable environment? What we propose in this book is that evaluative inquiry can not only be a means of accumulating information for decision making and action (operational intelligence) but that it also be equally concerned with questioning and debating the value of what we do in organizations. This approach is much more aligned with the interpretive perspective of organizational learning. That is, learning from evaluative inquiry is a social construction occurring through the involvement of multiple constituencies each representing different perspectives. It is socially situated and is mediated through participants' previous knowledge and experiences.[Page xx]
We see evaluative inquiry as a kind of public philosophy in which organization members engage in dialogue with clients and other stakeholders about the meaning of what they do and how they do it. In this dialogue, they pay particular attention to the historical, political, and sociological aspects of the objects of inquiry (Schwandt, 1992). Evaluative inquiry for organizational learning and change encompasses
- A focus on program and organizational processes as well as outcomes
- Shared individual, team, and organizational learning
- Education and training of organizational practitioners in inquiry skills
- Modeling the behaviors of collaboration, cooperation, and participation
- Establishing linkages between learning and performance
- Searching for ways to create greater understanding of the variables that affect organizational success and failure
- Using a diversity of perspectives to develop understanding about organizational issues
The approach to evaluative inquiry we present in this book is a result of our cumulative experiences teaching evaluation, conducting research on evaluation practice, and providing consulting services to education, business, health care, government, and nonprofit organizations. Over the years, we have learned that in spite of an organization's interest in evaluation, the findings of our inquiry efforts, often reported in traditional lengthy final reports, have gone unused. The reports may have been read by some, but all in all, the changes called for in these documents were rarely implemented. Reasons for this situation included (a) the changes already had been implemented as program participants were monitoring their own progress, (b) the political climate had changed so that interest in the evaluation and its findings had diminished, or (c) a lack of initial buy-in to the evaluation limited the perceived usefulness of its findings. As we reflected on the reasons why our evaluation efforts failed to have the kinds of organizational impact we [Page xxi]had hoped for, we came to realize that traditional evaluation practice is inadequate for helping today's organizations meet the complex challenges of a global economy and the emergence of the knowledge era. In addition, evaluation, as we and many others have been practicing it, has not paid attention to the learning dimensions of evaluative activity. We first addressed this problem with Mary Piontek in our book on evaluation strategies for communicating and reporting (Torres, Preskill, & Piontek, 1996; also see Torres, Preskill, & Piontek, 1997). This new book represents a more comprehensive reconceptualization of evaluation practice as a set of processes that leads to individual, team, and organizational learning.
Our goals for writing this book are to
- Describe the role of evaluative inquiry in learning organizations
- Provide a framework for conducting evaluative inquiry within an organizational learning context
- Stimulate reflection and conversation among evaluators, researchers, organizational members, and consultants about their own practice
This book offers readers a way of thinking about and conducting evaluative inquiry in every kind of organization. We believe that by integrating evaluative inquiry into an organization's work processes, organization members will be better able to meet the demands of tomorrow's challenges.For whom did we Write this Book?
This book was written for evaluators, managers, administrators, researchers, consultants, trainers, staff development and organization development practitioners, and leaders in education, health care, business and industry, government, and nonprofit organizations. It is for those individuals who wish to use evaluative inquiry as a catalyst for organizational change, growth, and renewal. It is for those who believe learning is a lifelong process that is never finished. Finally, it is [Page xxii]for people who believe in developing communities of practice where work is best accomplished through interpersonal relationships and dialogue.
It is important to recognize that this book addresses both theoretical and practical issues associated with conducting evaluative inquiry. It is an attempt to bridge what research says about individual, team, and organizational learning and evaluation. This book is neither a primer on organizational learning nor a textbook on evaluation. For both of these topics, there are many excellent resources.1 Instead, this book tries to answer the questions “How does evaluative inquiry contribute to organizational learning?” and “How do we practice evaluative inquiry in ways that maximize individual and team learning?” We hope this book provides readers with another way to conceptualize both (a) organizational learning and change and (b) evaluation.
We've chosen to focus on using evaluative inquiry processes within organizations—versus across large-scale, multisite programs or within a public policy arena. The processes described in the book build on local and working knowledge and therefore are most readily applied within a specific organizational context. At the same time, we hope that those working in policy-making environments will also find the ideas and strategies discussed in the book useful.Our Field Research
The ideas expressed in this book represent our collective experience, our reading of the extant literature, and in-depth interviews we conducted in organizations that were attempting to implement organizational learning practices. These include
- Land O' Lakes (Minneapolis, MN)
- Colorado Department of Education—The Prevention Initiatives Unit (Denver, CO)
- Ford Motor Company—Electrical, Fuel, and Handling Division (Ypsilanti, MI) [Page xxiii]
- Presbyterian Hospital and Healthcare Services (Albuquerque, NM)
Specifically, we were interested in knowing (a) how the organization became interested in organizational learning principles and practices; (b) the nature of organizational support needed for implementing organizational learning practices; (c) what specific strategies were being used to facilitate and support individual, team, and organizational learning; and (d) what the organization perceived to be concrete outcomes of implementing organizational learning practices. We also asked our interviewees to explain if, and how, they were evaluating any of these efforts.
During our 1- to 3-day site visits, we interviewed a total of 28 individuals. Throughout the book, we have integrated quotations from our interviewees and from others we have met over the last 2 years with whom we've discussed our ideas. Each of the quotations used is referenced by the person's initials and organization, and by the location of the quotation in our data file. In addition, all quotations were approved for use in this book by the interviewees.Overview of the Book
The first three chapters of this book lay the foundation and context for evaluative inquiry. In Chapter 1, we discuss how organizations are changing and the necessary conditions for organizational survival in the 21st century. We make the case that organizations will need to develop communities of inquirers that capitalize on the knowledge and expertise of those closest to organizational issues, and that evaluative inquiry can be the driving force for this effort.
In Chapter 2, we define and discuss what it means to learn at the individual, team, and organizational levels. If organizations are to benefit from evaluative inquiry, it is important to understand how learning occurs at each of these levels. Chapter 3 describes the four learning processes that facilitate evaluative inquiry. These are dialogue, reflection, asking questions, and identifying and clarifying values, beliefs, [Page xxiv]assumptions, and knowledge. Engaging these processes throughout evaluative inquiry efforts will develop greater insights and understandings about organizational issues. Ultimately, these insights and understandings lead to informed decisions necessary for organizational change.
The next three chapters guide the reader through the three phases of evaluative inquiry. In Chapter 4, we explain how an evaluative inquiry team goes about focusing the inquiry using the four learning processes. The steps described in this chapter include defining the issue, identifying stakeholders, and determining evaluative questions the inquiry will address. In Chapter 5, we describe how to carry out the inquiry, again using the four learning processes to facilitate the tasks of this phase. In this chapter, we cover issues related to the inquiry's design, data collection, analysis and interpretation, recommendations, and communicating and reporting. Chapter 6 focuses on applying learning from the evaluative inquiry. It covers identifying and selecting action alternatives, developing action plans, and implementing and monitoring the actions taken. Through each of these three chapters, we have interwoven an illustrative case to help the reader clearly understand how the four learning processes are used during each of the evaluative inquiry phases. Additionally, Chapters 4, 5, and 6 include specific strategies (in boxed text) for implementing evaluative inquiry.
In Chapter 7, we argue that for evaluative inquiry to be successful, an organization's infrastructure must support and facilitate it. The critical elements of this infrastructure are culture, leadership, communication, and systems and structures. How each of these relates to evaluative inquiry is discussed in detail with examples provided from our research. Chapter 8 addresses additional considerations for the practice of evaluative inquiry. It compares evaluative inquiry for organizational learning and change to more traditional approaches to evaluation and organization development. In addition, it explores evaluator roles and challenges to implementing evaluative inquiry in today's organizations.
The reader can take any of several routes in reading this book. For the fullest understanding of what constitutes evaluative inquiry and why it is an important approach to understanding organizational life, it is best to start with Chapter 1 and read the chapters in sequence. Readers [Page xxv]already familiar with the literature on organizational change and theories of individual, team, and organizational learning may wish to begin with Chapter 3, which describes the four learning processes central to evaluative inquiry. They would then read Chapters 4 through 6, the “how to conduct evaluative inquiry” chapters. After acquiring a thorough understanding of what evaluative inquiry entails, the reader should then read Chapter 7, which addresses what needs to be present in the larger organization infrastructure for successful implementation of evaluative inquiry. Regardless of the road taken, we hope readers will develop an understanding of how evaluative inquiry can be a catalyst for learning in organizations. Although the journey no doubt will be a difficult one, the challenges experienced along the way may reveal unanticipated hidden treasures.Note
1. For example, see the following books, which provide an overview of the theory and practice of evaluation: Herman (1987), Joint Committee on Standards for Educational Evaluation (1994), Patton (1997), Posavac and Carey (1997), Shadish, Cook, and Leviton (1991), and Worthen, Sanders, and Fitzpatrick (1997).
Books on the theory and practice of organizational learning include Argyris (1992), Argyris and Schon (1996), Chawla and Renesch (1995), Cohen and Sproull (1996), DiBella and Nevis (1998), Dixon (1994), Marquardt (1996a), and Watkins and Marsick (1993).[Page xxvi]
Questions for Facilitating Evaluative Inquiry[Page 193]Chapter 4: Focusing the Evaluative InquiryDefining the Evaluative Issue
- What is the history or background of the problem/issue?
- Have any previous inquiries been conducted regarding this topic? If yes, what were the results? How were the findings used?
- Why is it important that we develop new insights into this problem/issue at this time?
- What kinds of decisions would we like to make about the problem/issue?
- What do we know about this problem/issue? What don't we know?
- What organizational activities, procedures, or policies are part of the issue/problem?[Page 194]
- What are the organizational variables that are affecting the context of the problem/issue?
- What organizational politics are affecting the problem/issue?
- What are our worst fears about this evaluative inquiry effort?
- What are our hopes for what this evaluative inquiry effort will accomplish?
- What is the purpose of the evaluative inquiry?
Determining Evaluative Questions
- What individuals or groups were mentioned as we worked to define the issues for this inquiry? What is their involvement with the problem/issue?
- Why do we believe that each of these individuals or groups are stakeholders?
- How might each of these individuals or groups be affected by the outcomes of this inquiry?
- Which individuals or groups might use the findings for policy-making decisions?
- Which individuals or groups might use the findings for making operational decisions?
- Which individuals or groups might be interested in the findings but are not in a decision-making position relative to the subject of the inquiry?
- Who has a “right to know” about the inquiry's findings?
- Which of these individuals or groups should be involved in the process of the inquiry but are not currently on the inquiry team?
Chapter 5: Carrying out the InquiryDesigning the Inquiry
- What questions should the inquiry seek to answer?
- Why are these evaluative questions important?
- What must we know now, versus what can we wait to know?
- Which are the most critical questions?[Page 195]
- What are the consequences if we do not answer these questions soon?
- What do we hope will happen by answering these questions?
- What do we think will happen if the answers to these questions are not in line with what certain stakeholders believe is true?
Methods and Procedures for Data Collection
- What kinds of data does the organization typically respond to? What does it ignore?
- To what extent was the type of data from previous studies considered credible by the organization?
- What are the preferred methods of those responsible for carrying out the data collection and analysis activities for the present inquiry?
- What “political” issues related to the use of different data collection methods have surfaced in the past?
- How might the answers to any of these questions influence decisions about the present inquiry?
Methods and Procedures for Data Analysis and Interpretation
- What kinds of data can best answer the evaluative inquiry questions we seek to answer?
- What type of data will indicate if the activities, procedures, or policies in question are being implemented as intended or will inform us of any unanticipated outcomes?
- What type of data will indicate if the intended outcomes of these activities, procedures, or policies in question are being realized?
- What data already exist that might address the evaluative inquiry questions? Where does this information reside?
- What other kinds of data (e.g., background information) might be available that would address the evaluative inquiry questions?[Page 196]
- Will data be needed on an ongoing basis, or is the need more episodic—that is, particular to a one-time situation?
- What logistical constraints (methodological, time, resources) must be considered?
- What resources (technical/computer expertise, expenses, equipment, personnel time, access to particular personnel, etc.) will be needed to implement each data collection activity?
Methods and Procedures for Communicating and Reporting
- How will each type of data collected be analyzed?
- What expertise/resources will be needed? How much time will it take to analyze the data from each method used?
- In what ways can different individuals and groups contribute to the analysis and interpretation of data?
[Page 197]Implementing Data Collection Efforts
- What are some different analytic or reporting frameworks that might be appropriate for representing findings from the evaluative inquiry (e.g., the original evaluative questions, some organizational or programmatic framework, a new issues-oriented framework determined by the content of the findings)?
- What does each stakeholder group need to know about how the inquiry is progressing and when? What is the best format and channel for communicating this information?
- In addition to the previously identified stakeholders, who else should receive communications about the inquiry?
- Is a comprehensive final report necessary? Will more informal reporting suffice?
Data Analysis and Interpretation
- Is the data collection plan being implemented as specified? If not, why not?
- What new developments threaten successful implementation of the data collection activities?
- How is the organization reacting to carrying out the inquiry?
- What adjustments to the data collection plan need to be made?
- What assumptions, values, beliefs, assumptions, and knowledge are proposed adjustments to the plan based upon?
- What individuals or groups need to be apprised of adjustments made to the data collection plan?
- Are analysis and interpretation activities on schedule?
- Do any new developments from either within or outside the organization threaten successful implementation of analysis and interpretation activities?
- What adjustments to the analysis strategies are necessary? What individuals or groups need to be apprised of these adjustments?
- How well does each possible different framework meet the learning needs of the organization?
- What values, beliefs, assumptions, and knowledge are reflected in the choice of a particular analytic or reporting framework?
- What other perspectives should be considered?
- How should quantitative and qualitative data be presented?
- How should data across departments or sites be integrated and presented?
Communicating and Reporting
- What evidence supports each recommendation?
- Does the set of recommendations represent all findings of the inquiry?[Page 198]
- Do recommendations take into account what is known about organizational context, logistics, and constraints?
- What particular values, beliefs, or assumptions are reflected in each recommendation?
Chapter 6: Applying LearningIdentifying and Selecting Action Alternatives
- Are communications and reports written in a clear, jargon-free style?
- Have tables and figures been used effectively to make information more understandable?
- Has the communication of negative findings been handled productively (e.g., within the context of continuous improvement and learning)?
- Have findings and interpretations been appropriately summarized for different audiences?
- Does the format of each communication/report facilitate easy interpretation and assimilation of its content?
Developing an Action Plan
- What priority should each recommendation be given (low, medium, high)?
- What reasoning is behind each prioritization?
- What values, beliefs, assumptions, and knowledge are reflected in reasons given?
- In what possible (additional) ways can this recommendation be addressed?
- Specifically, what will this action alternative achieve?
- What groups and individuals would be involved in the implementation of this action alternative?
- What other groups and individuals will be affected by it?
- What are potential undesirable consequences of this alternative?[Page 199]
- How much will implementation of this action cost? What other resources will it require?
- What new skills, knowledge, and attitudes will organizational members need to acquire to facilitate this action's success?
- What incentives are there for organizational members to make the changes in their daily practices that it will require?
- To what extent will the organization's existing infrastructure (culture, leadership, modes of communication, other systems and structures) either support or undermine implementation of this action alternative?
- Conversely, what impact will it likely have on any elements of the infrastructure?
- In what groups, teams, departments, and/or individuals is this action alternative likely to be met with resistance?
- What additional obstacles or barriers, if any, might impede successful implementation? What could be done to overcome any of these barriers?
- What has been the organization's experiences with similar change initiatives?
[Page 200]Implementing the Action Plan and Monitoring Progress
- What steps are necessary to carry out this action alternative? Are there several broad categories of action, each of which has smaller substeps?
- How will the critical issues for successful implementation (outlined in the previous phase of identifying and selecting the action alternative) be accounted for in these steps?
- Who should be responsible for carrying out each step/substep?
- What amount of time realistically will be required to do so?
- Are there critical organizational events that must be accounted for in developing the timeline for implementation?
- How will we know if the action is being implemented successfully?
- At what point(s) should dialogues to monitor progress take place?
- Who should be involved in these dialogues?
- How is actual implementation paralleling intended implementation, and what can we learn from any discrepancies?
- What new issues are surfacing that impact the success of the change being made?
- What adjustments to the plan are necessary, and what is the best way to make them?
- What barriers or obstacles are impeding full implementation?
References[Page 201]1997). 12 principles of knowledge management. Training & Development, 51(11), 71–74.(1996). A different sort of corporate merger. Training & Development, 50(6), 8.(American Society for Training and Development. (1996a). Basic skills 1996 survey results. Survey2, 1996. Alexandria, VA: Author.American Society for Training and Development. (1996b). Building a learning organization. Survey 3, 1996. Alexandria, VA: Author.American Society for Training and Development. (1996c). Leadership development 1996 survey results. Survey 1, 1996. Alexandria, VA: Author.American Society for Training and Development. (1997). Intellectual capital. Survey 1, 1997. Alexandria, VA: Author.1985). Strategy, change and defensive routines. Marshfield, MA: Pitman.(1992). On organizational learning. Cambridge, MA: Blackwell Business. http://dx.doi.org/10.1108/02683949510093849(1978). Organizational learning: A theory of action perspective. Reading, MA: Addison-Wesley. http://dx.doi.org/10.2307/40183951, & (1996). Organizational learning II: Theory, method and practice. Reading, MA: Addison-Wesley., & (1995). The boundaryless organization: Breaking the chains of organizational structure. San Francisco: Jossey-Bass., , , & (1994). The change monster. Training, 31(5), 136.. ([Page 202]1995). Creating appreciative learning cultures. Organizational Dynamics, 24(2), 36–48. http://dx.doi.org/10.1016/0090-2616%2895%2990070-5(1998, January 7). Second World Congress on the Management of Intellectual Capital. (Available through e-mail at firstname.lastname@example.org)(1998). The 1998 ASTD state of the industry report. Alexandria, VA: American Society for Training and Development., & (1972). Steps to an ecology of mind. New York: Ballantine.(1997). A tradition that has no name. New York: Basic Books., , & (1994). Organisations: Valuing and learning. In J. G.Burgoyne, M.Pedler, & T.Boydell (Eds.), Toward the learning company: Concepts and practices. Maiden-head, UK: McGraw-Hill.(1996). On dialogue (L. Nichol, Ed.). London: Routledge.(1995). How to conduct self-administered and mail surveys. Thousand Oaks, CA: Sage., & (1997). Managing an organization as a system. Training, 34(2), 68–74., & (1989). Achieving results from training. San Francisco: Jossey-Bass. http://dx.doi.org/10.1002/pfi.4170300212(1995). Becoming a critically reflective teacher. San Francisco: Jossey-Bass.(1998). Talking democratically. San Francisco: Jossey-Bass., & (1994). Power and production of knowledge: Collective team learning in work organizations. Human Resource Development Quarterly, 5(3), 213–235. http://dx.doi.org/10.1002/hrdq.3920050303(Brooks, A., & Watkins, K. E. (Eds.). (1994). The emerging power of action inquiry technologies [Special issue]. New Directions for Adult and Continuing Education, 63(Fall).1995). Dialogue: Capacities and stories. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 153–164). Portland, OR: Productivity Press.(1989). Participatory evaluation: A tool to assess projects and empower people. New Directions in Program Evaluation, 42, 9–17., & (1996). An unused intelligence. Berkeley, CA: Canari., & (1987). Creating the high-performance team. New York: John Wiley & Sons., , & (1992). Creating a learning organization. Royal Society for the Arts Journal, 140(5428), 321–332.(1991). What teachers say about reflection. Educational Leadership, 49(3), 18–21.(1997). Cultural due diligence. Training, 34(11), 67–75.(1996). A new paradigm for learning in organizations. Human Resource Development Quarterly, 7(3), 287–293. http://dx.doi.org/10.1002/hrdq.3920070309(1995). Conclusion: Reflections of learning from a gathering. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 501–508). Portland, OR: Productivity Press.([Page 203]Chawla, S., & Renesch, J. (Eds.). (1995). Learning organizations: Developing cultures for tomorrow's workplace. Portland, OR: Productivity Press.1995). Meetings go high-tech. Training & Development, 49(11), 32–38., & (1996). Making sense of qualitative data: Complementary research practices. Thousand Oaks, CA: Sage., & (Cohen, M. D., & Sproull, L. S. (Eds.). (1996). Organizational learning. Thousand Oaks, CA: Sage.1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397–418., & (1995). The case for participatory evaluation: Theory, research, practice. In J. B.Cousins & L. M.Earl (Eds.), Participatory evaluation in education (pp. 5–18). London: Falmer., & (1994). Understanding and promoting transformative learning. San Francisco: Jossey-Bass.(1993). Action research and organizational development. Westport, CT. Praeger.(1987). How organizations learn: A communication framework. Research in the Sociology of Organizations, 5, 1–36., & (1992). Human resource development: An emerging profession—an emerging purpose. Studies in Continuing Education, 14(2), 187–204. http://dx.doi.org/10.1080/0158037920140208, & (1996). Rumble, rumble. Training & Development, 50(11), 44–45.(1933). Experience and education. New York: Macmillan. http://dx.doi.org/10.1080/00131728609335764(1938). How we think. New York: Heath. http://dx.doi.org/10.1037/10903-000(1998). How organizations learn. San Francisco: Jossey-Bass. http://dx.doi.org/10.5465/AMBPP.1995.17536560, & (1994). The organizational learning cycle: How we can learn collectively. London: McGraw-Hill.(1992). Organizational learning: A review of the literature with implications for HRD professionals. Human Resource Development Quarterly, 3(1), 29–49. http://dx.doi.org/10.1002/hrdq.3920030105(1996). The journey toward becoming a learning organization: Are we almost there? In K. E.Watkins & V. J.Marsick (Eds.), Creating the learning organization (Vol. 1, pp. 67–80). Alexandria, VA: American Society for Training and Development., & (1997). The future that has already happened. Harvard Business Review, 75(5), 19–23.(1996). How to conduct organizational surveys. Thousand Oaks, CA: Sage. http://dx.doi.org/10.4135/9781452231563, , , & (1991). The relationship of accountability and interdependent behavior to enhancing team consequences. Group & Organization Studies, 16(3), 300–312. http://dx.doi.org/10.1177/105960119101600305(1996, August). Involving line managers in training and development. (Available at http://www.fenman.co.uk/index.htm)(1994). Empowerment evaluation. Evaluation Practice, 15(1), 1–15. http://dx.doi.org/10.1016/0886-1633%2894%2990055-8(1996). Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage. http://dx.doi.org/10.4135/9781452243573(1995). How to design surveys. Thousand Oaks, CA: Sage.([Page 204]1985). Organizational learning. Academy of Management Review, 10, 803–813., & (1987). How to analyze data. Newbury Park, CA: Sage., & (1993). Survey research methods. Thousand Oaks, CA: Sage.(1995). Managing projects in organizations. San Francisco: Jossey-Bass.(1995). How to conduct interviews by telephone and in person. Thousand Oaks, CA: Sage., & (1970). Pedagogy of the oppressed. New York: Seabury.(1993). Change forces: Probing the depths of educational reform. London: Falmer.(1996). Signs of the times. Training & Development, 50(2), 32–36., & (1983). Frames of mind. New York: Basic Books.(1984, July 25). Asking questions is the key skill needed for “discussion teaching.”Chronicle of Higher Education, p. 20.(1993). Building a learning organization. Harvard Business Review, 71(4), 78–91.(1996). Learning organizations come alive. Training & Development, 50(12), 35–45., , , and (1990). The rise of the new organization. MW, 19(1), 3–5.(1990). Technology and organizations. San Francisco: Jossey-Bass. http://dx.doi.org/10.1016/S1535-5535%2804%2900038-3, , & Associates. (1997). Creating an environment for successful projects. San Francisco: Jossey-Bass., & (1997). The handbook for focus group research. Thousand Oaks, CA: Sage.(1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 18(5), 574–591.(Greene, J. C., & Caracelli, V. J. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms [Special issue]. New Directions for Evaluation, 74.1996). The project manager's partner. Amherst, MA: Human Resource Development Press.(1996). Teams in organizations: Recent research on performance and effectiveness. Annual Review of Psychology, 47, 307–338. http://dx.doi.org/10.1146/annurev.psych.47.1.307, & (Hamilton-KSA Analysts. (1996). Demand for value in health care, part 3. Perspectives. Atlanta, GA: Author.1994). The team building tool kit. New York: American Management Association.(1994). Human resource development. Fort Worth, TX: The Dreydon Press, Harcourt Brace College Publishers. http://dx.doi.org/10.1108/eb053539, & (1996). The new organization. London: McGraw-Hill. http://dx.doi.org/10.1108/17538350910970228(1995). The web of inclusion: A new architecture for building great organizations. New York: Currency/Doubleday.([Page 205]Herman, J. L. (Ed.). (1987). Program evaluation kit (2nd ed.). Thousand Oaks, CA: Sage.1995). Shared values: Nutrients for learning. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 463–474). Portland, OR: Productivity Press., & (1997). Professional learning communities: Communities of continuous inquiry and improvement. Austin, TX: Southwest Educational Development Laboratory.(1996, March). Team sense-making: An imperative for individual and organizational learning. In E. F.HoltonIII (Ed.), Academy of Human Resource Development Conference Proceedings. Baton Rouge, LA: Academy of Human Resource Development., , , & (1991). Organizational learning: The contributing processes and the literatures. Organizational Science 2(1), 88–115.(1993). Spitwad sutras: Classroom teaching as sublime vocation. Westport, CT. Bergin and Garvey.(1994). Introducing a new approach to experiential learning. In L.Jackson & R. S.Caffarella (Eds.), New directions for adult and continuing education, 55, 17–27., & (1992). Paradoxes of learning. San Francisco: Jossey-Bass.(1995). The role of evaluation in schools as learning organizations. In P. M.Jenlink (Ed.), Systemic change: Touchstones for future schools. Palatine, IL: IRI/Skylight Training and Publishing. http://dx.doi.org/10.1007/BF02812078, & (1996). Population aging and employment policies. In R.Paton, G.Clark, G.Jones, J.Lewis, & P.Quintas (Eds.), The new management reader (pp. 4–12). London: Routledge.(Joint Committee on Standards for Educational Evaluation. (1994). The personnel evaluation standards. Newbury Park, CA: Sage.1997). Workforce 2020: Work and workers in the 21st century. Indianapolis: Hudson Institute., & (1996). Facilitator's guide to participatory decision-making. Gabriola Island, British Columbia, Canada: New Society Publishers.(1982). The action research planner. Geelong, Australia: Deakin University Press., & (1994). Developing reflective judgment. San Francisco: Jossey-Bass. http://dx.doi.org/10.1207/s15326985ep3901_2, & (1980). The modern practice of adult education: From pedagogy to andragogy ((2nd ed.). New York: Cambridge Books.1993). Communities of commitment: The heart of learning organizations. Organizational Dynamics, 22(2), 5–23. http://dx.doi.org/10.1016/0090-2616%2893%2990050-B, & (1984). Experiential learning: Experience as the source of learning and development. Englewood Cliffs, NJ: Prentice Hall. http://dx.doi.org/10.1016/B978-0-7506-7223-8.50017-4(1996). Leading change. Boston, MA: Harvard Business School Press.(1996). Interviews: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage.([Page 206]1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. http://dx.doi.org/10.1017/CBO9780511815355, & (1996, September). Understanding the learning company: A constructivist approach. Paper presented at the Organizational Learning and the Learning Organization Research Symposium, Lancaster University, Lancaster, UK., , & (1991). The managerial implications of changing work force demographics: A scoping study. Human Resource Management, 30(1), 7–29. http://dx.doi.org/10.1002/hrm.3930300102, & (1996). Either.or yields the theory of both. In F.Hesselbein, M.Goldsmith, & R.Beckhard (Eds.), The leader of the future. San Francisco: Jossey-Bass., & (1995). The future, disposable organizations and the rigidities of imagination. Organization, 2(3/4), 427–440. http://dx.doi.org/10.1177/135050849523009(1996a). Building the learning organization. New York: Mc-Graw Hill.(1996b). Cyberlearning: New possibilities for HRD. Training & Development, 50(11), 56–57.(1994). The global learning organization. Burr Ridge, IL: Irwin., & (1994). Designing qualitative research (, & (2nd ed.). Thousand Oaks, CA: Sage.1996). Individuals who learn create organizations that learn. New Directions for Adult and Continuing Education, 72(Winter), 97–104., & (1997). Work-life programs reap benefits. HR Magazine, 42(6), 110–114.(1995). Action learning: A guide for professional, management and educational development. London: Kogan Page., & (1993). Unlearning the organization. Organizational Dynamics, 22(2), 67–79. http://dx.doi.org/10.1016/0090-2616%2893%2990054-5, & (1995). Waves of change. Boston, MA: Harvard Business School Press., , & (1995). The age of participation. San Francisco: Berrett-Koehler., & (1996). The shift to participation. Perspectives on Business and Global Change, 10(1), 47–59., & (1991). Action research: A short modern history. Geelong, Australia: Deakin University Press.(1991). Learning in adulthood. San Francisco: Jossey-Bass., & (1997). Research methods in education and psychology: Integrating diversity with quantitative & qualitative approaches. Thousand Oaks, CA: Sage.(1991). Transformative dimensions of adult learning. San Francisco: Jossey-Bass.(1994). Qualitative data analysis: An expanded sourcebook (, & ([Page 207]2nd ed.). Thousand Oaks, CA: Sage.1994). Open organizations: A model for effectiveness, renewal, and intelligent change. San Francisco: Jossey-Bass., , , & (1995). Designing team-based organizations. San Francisco: Jossey-Bass., , & (Moingeon, B., & Edmondson, A. (Eds.). (1996). Organizational learning and competitive advantage. London: Sage. http://dx.doi.org/10.4135/97814462502281997). The focus group kit. Thousand Oaks, CA: Sage., & (1995). Development strategies for the knowledge era. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 322–335). Portland, OR: Productivity Press.(1987). How to communicate evaluation findings. Newbury Park, CA: Sage., , & (1997). Completing a qualitative project: Details and dialogue. Thousand Oaks, CA: Sage.(1985). Manual: A guide to the development and use of the Myers-Briggs Type Indicator. Palo Alto, CA: Consulting Psychologists Press., & (1994). From groupthink to teamthink: Toward the creation of constructive thought patterns in self-managing work teams. Human Relations, 47(8), 929–949. http://dx.doi.org/10.1177/001872679404700804, & (1997). Using performance measurement to improve programs. In K. E.Newcomer (Ed.), Using performance measurement to improve public and nonprofit programs (New Directions for Evaluation No. 75). San Francisco: Jossey-Bass.(1995). The knowledge-creating company. New York: Oxford University Press. http://dx.doi.org/10.1016/B978-0-7506-7009-8.50016-1, & (1992). Open space technology: A user's guide. Potomac, MD: Abbott.(1987). Community, conflict, and ways of knowing. Change, 19(5), 20–25.(1994). Developmental evaluation. Evaluation Practice, 15(3), 311–319. http://dx.doi.org/10.1016/0886-1633%2894%2990026-4(1997). Utilization-focused evaluation: The new century text ((3rd ed.). Thousand Oaks, CA: Sage.1994). Designing educational project and program evaluations: A practical overview based on research and experience. Boston: Kluwer. http://dx.doi.org/10.1007/978-94-011-1376-2(1987). The different drum. New York: Simon & Schuster.(1996). Invited reaction: An adult educator responds. Human Resource Development Quarterly, 7(2), 131–139. http://dx.doi.org/10.1002/hrdq.3920070204(1992). Liberation management. New York: Knopf.(1997). How to. Training & Development, 51(3), 9., & (1995). Cognitive distortions: Barriers to change. TQM Network News (Spring). Alexandria, VA: American Society for Training and Development., & (1997). Program evaluation methods and case studies (, & ([Page 208]5th ed.). Upper Saddle River, NJ: Prentice Hall.1969). Teaching as a subversive activity. New York: Delacorte., & (1991, October). Metaphors of educational reform implementation: A case study of the Saturn School of Tomorrow. Paper presented at the annual meeting of the American Evaluation Association, Chicago, IL.(1994). Evaluation's role in enhancing organizational learning: A model for practice. Evaluation and Program Planning, 17(3), 291–297. http://dx.doi.org/10.1016/0149-7189%2894%2990008-6(1997, March). HRD evaluation as the catalyst for organizational learning. Paper presented at the annual meeting of the Academy of Human Resource Development, Atlanta, GA.(1997, November). Expanding theoretical conceptions of evaluation misuse: Lessons from practice. Paper presented at the annual meeting of the American Evaluation Association, San Diego, CA., , & (1997). Meeting the postmodern challenge: Pragmatism and evaluative inquiry for organizational learning. Advances in Program Evaluation, 3, 155–169., & (1982). The origin and growth of action learning. Bickly, Kent, UK: Chartwell-Bratt.(1969). Freedom to learn. Columbus, OH: Merrill.(1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage., & (1995). Learning communities: An alternative to the expert model. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 279–291). Portland, OR: Productivity Press.(1994). The centric reflection model: A kaleidoscope for staff developers. Journal of Staff Development, 15(3), 16–20., , & (Sapsford, R., & Jupp, V. (Eds.). (1996). Data collection and analysis. Thousand Oaks, CA: Sage.1996). The cultures of management: The key to organizational learning. Sloan Management Review, 38(1), 9–20.(1983). The reflective practitioner. New York: Basic Books. http://dx.doi.org/10.1080/07377366.1986.10401080(1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.(1989). No more teams!: Mastering the dynamics of creative collaboration. New York: Currency.(1993). From cooperative action to collective self-reflection: A sociodynamic approach to educational research. In M.Schratz (Ed.), Qualitative voices in educational research (Social Research and Educational Series No. 10). London: Falmer.(1995). Learning as an organization: A journey into chaos. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 365–380). Portland, OR: Productivity Press.(1992). Better living through evaluation? Images of progress shaping evaluation practice. Evaluation Practice, 13(2), 135–144. http://dx.doi.org/10.1016/S0886-1633%2805%2980006-6(1997). Evaluation as practical hermeneutics. Evaluation, 3(1), 69–83. http://dx.doi.org/10.1177/135638909700300105(1994). Ground rules for groups. Training & Development, 48(8), 45–53.([Page 209]1990a). The fifth discipline. New York: Doubleday. http://dx.doi.org/10.1108/eb025496(1990b). The leaders' new work: Building learning organizations. Sloan Management Review, 32(1), 19–35.(1996). Leading learning organizations: The bold, the powerful, and the invisible. In F.Hesselbein, M.Goldsmith, & R.Beckhard (Eds.), The leader of the future (pp. 41–57). San Francisco: Jossey-Bass.(1994). The fifth discipline fieldbook. New York: Doubleday., , , , & (1994). Building community in schools. San Francisco: Jossey-Bass.(1991). Foundations of program evaluation: Theories and practice. Newbury Park, CA: Sage., , & (1988). Participatory evaluation: Towards a transformation of assessment for women's studies programs and projects. Educational Evaluation and Policy Analysis, 11(5), 555–590.(1991). Teaching organizations to learn. Organization Development Journal, 9(4), 1–12., & (1996). Share your worth. HR Magazine, 41(6), 86.(1994). Statistics for the social sciences. Thousand Oaks, CA: Sage.(1993). The emergence of learning communities. The systems thinker. Cambridge, MA: Pegasus Communications.(1995). The art of case study research. Thousand Oaks, CA: Sage.(1997). Communities of practice. Training, 34(2), 34–42.(1987). How to focus an evaluation. Thousand Oaks, CA: Sage., & (1997). Intellectual capital. New York: NY: Doubleday/Currency. http://dx.doi.org/10.1002/pfi.4140370713(1995). The digital economy. New York: Basic Books.(1995). The renaissance of learning in business. In S.Chawla & J.Renesch (Eds.), Learning organizations: Developing cultures for tomorrow's workplace (pp. 85–99). Portland, OR: Productivity Press.(1991). Improving the quality of internal evaluation: The evaluator as consultant-mediator. Evaluation and Program Planning, 14(3) 189–198. http://dx.doi.org/10.1016/0149-7189%2891%2990055-L(1996). Evaluation strategies for communicating and reporting: Enhancing learning in organizations. Thousand Oaks, CA: Sage., , & (1997). Communicating and reporting: Concerns of internal and external evaluators. Evaluation Practice, 18(2), 105–125. http://dx.doi.org/10.1016/S0886-1633%2897%2990016-7, , & (1979). The adult's learning projects: A fresh approach to theory and practice in adult learning ((2nd ed.). Toronto: Ontario Institute for Studies in Education.Training Magazine Industry Report. (1994, October). Training Magazine, 31(10), 59–66.Training Magazine Industry Report. (1997, October). Training Magazine, 34(10), 35–75.1981, June). The evolution of socio-technical systems: A conceptual framework and an action research program (Occasional Paper No. 2). Toronto: Ontario Quality of Working Life Centre.(1996). Learning as a way of being. San Francisco: Jossey-Bass. http://dx.doi.org/10.1002/pfi.4140370412([Page 210]1995). Technological innovation, learning, and leadership. Paper presented at the Stern School of Business Research of New York University, New York., & (1996). Of course organizations learnNew Directions for Adult and Continuing Education, 72(Winter), 89–94.(1992). Building the learning organization: A new role for human resource developers. Studies in Continuing Education, 14(2), 115–129. http://dx.doi.org/10.1080/0158037920140203, & (1993). Sculpting the learning organization. San Francisco: Jossey-Bass., & (Watkins, K. E., & Marsick, V. J. (Eds.). (1996). Creating the learning organization (Vol. 1). Alexandria, VA: American Society for Training and Development.1992). Conflict resolution. New York: G. P. Putnam's Sons. http://dx.doi.org/10.1080/07488009408409153(1987). Productive workplaces. San Francisco: Jossey-Bass.(1997). Practice, learning, meaning, identity. Training, 34(2), 38–39.(1994). Leadership and the new science. San Francisco: Berrett-Koehler.(1993). The learning edge. New York: McGraw-Hill., & (1997). The chief learning officer: A case study at Millbrook Distribution Services. In H.Preskill & R.Dilworth (Eds.), HRD in transition: Defining the cutting edge. Washington, DC: International Society for Performance Improvement., & (1994). Transforming qualitative data: Description, analysis, and interpretation. Thousand Oaks, CA: Sage.(1997). Program evaluation: Alternative approaches and practical guidelines (, , & (2nd ed.). New York: Longman.1996). Understanding statistics: An introduction for the social sciences. Thousand Oaks, CA: Sage.(1995). Effective project management: How to plan, manage, and deliver projects on time and within budget. New York: John Wiley & Sons., , & (1996). Helping leaders take effective action: A program evaluation. Greensboro, NC: Center for Creative Leadership., & (1997). The leadership triad: Knowledge, trust, and power. New York: Oxford University Press.(1988). In the age of the smart machine. New York: Basic Books.(
About the Authors