The Practice of Health Program Evaluation

Books

David Grembowski

  • Citations
  • Add to My List
  • Text Size

  • Chapters
  • Front Matter
  • Back Matter
  • Subject Index
  • Dedication

    To Aunt Imogene

    Copyright

    View Copyright Page

    Preface

    In writing The Practice of Health Program Evaluation, I had two purposes in mind. My first purpose was to present the major concepts, methods, and issues of evaluating health programs and services for graduate students in health administration, public health, medicine, nursing, dentistry, pharmacy, social work, public administration, and related health professions. Over the past several years, I have taught graduate courses on health program evaluation to students in the health sciences and social work. My experiences repeatedly suggested to me that both teaching and learning might improve if students had a customized textbook about the evaluation of health programs and services that presented information in a form that most students could understand. Student evaluations of earlier versions of this textbook were promising and indicated that other students may also benefit from it.

    I also had a second audience in mind when writing this book—program administrators, decision makers, and interest groups in public and private organizations, as well as health program evaluators and health services researchers. As more and more money is spent developing programs and services to solve health problems, interest has grown in evaluating their performance. For many reasons, such as improving performance or defending a program's worth, administrators in public or nonprofit agencies often are eager to evaluate how well their programs work. In addition, as the number and size of managed care organizations have grown, so has interest in evaluating the cost-effectiveness of their services and in launching programs to keep their enrolled populations healthy. Policymakers and executive-level administrators also may be interested in this material as an aid to understanding evaluation methods and interpreting findings. Evaluators and health services researchers often apply evaluation methods to assess the performance of health delivery systems and therefore also may be interested in this text. More than ever before, health professionals in both the public and private sectors want management and decision-making tools for improving the performance of health programs and services.

    My second purpose was to teach students how to become evaluators—that is, how to apply research methods in the practice of health evaluation. Designing and conducting a health program evaluation is much more than an exercise in applied research methods. All evaluations are conducted in a political context, and the ability to complete an evaluation successfully depends greatly on the evaluator's ability to navigate the political terrain. In addition, evaluation itself is a process with interconnected steps designed to produce information for decision makers and other groups. Understanding the steps and their interconnections is just as fundamental to evaluation as is knowledge of quantitative and qualitative research methods. To convey these principles, I use the metaphor of evaluation as a three-act play with a variety of actors and interest groups, each having a role and each entering and exiting the stage at different points in the evaluation process. Evaluators are one of several actors in this play, and it is critical for them to understand their role if they are to be successful.

    Applying this principle, the book has three major sections, or “Acts,” that cover basic steps of the evaluation process. “Act I: Asking the Question” occurs in the political realm, where evaluators work with decision makers and other groups to define the questions that the evaluation will answer about a program. Chapter 3 presents material to help students and health professionals develop evaluation questions.

    In “Act II: Answering the Question,” evaluation methods are applied to answer the questions about the program. After the relevant interest groups and evaluator agree on the key questions about a program, the next step is to choose one or more evaluation designs that will answer those questions. Chapter 4 presents experimental and quasi-experimental impact evaluation designs, and Chapter 5 reviews cost-effectiveness analysis, which has become more prevalent and important in health care over the past two decades. Chapter 6 presents methods for designing evaluations of program implementation, or process evaluation.

    Once a design is chosen, quantitative and qualitative methods for conducting the evaluation must be developed and implemented. Chapter 7 presents methods for choosing the populations for the evaluation and sampling members from them. Chapter 8 reviews measurement and data collection issues frequently encountered in quantitative and qualitative evaluations. Finally, Chapter 9 describes data analyses for different impact and implementation designs.

    In “Act III: Using the Answers in Decision Making,” the evaluation returns to the political realm, where findings are disseminated to decision makers, interest groups, and other constituents. A central assumption is that evaluations are useful only when their results are actually used to formulate new policy or improve program performance. Chapter 10 presents methods for developing formal dissemination plans and reviews factors that influence whether evaluation findings are used or not. Chapter 11 reviews major challenges facing the discipline in the next decade.

    In summary, by integrating the evaluation literature about health programs and services from a wide variety of sources, this book is designed to be an educational resource for teachers and students, as well as a reference for health professionals engaged in program evaluation.

    Acknowledgments

    In many ways, this book is a synthesis of what I have learned about evaluation from my evaluation teachers and colleagues, and I am very grateful for what they have given me. I wish to acknowledge and thank my teachers—Marilyn Bergner and Stephen Shortell—who provided the bedrock for my professional career in health program evaluation when I was in graduate school. In those days, their program evaluation class was structured around a new, unpublished book, Health Program Evaluation, by Stephen Shortell and William Richardson, which has since become a classic in our field and provided a model for this work. I also thank Donald Miller, my urban planning professor, for his insights and cutting-edge methods of conducting plan evaluation. I give my special thanks to Ann Blalock, who introduced me to the politics of programs in the real world and who was a role model for navigating this terrain with grace and humor.

    As a new professor and beginning teacher, I benefited greatly from the support and help of other teachers of health program evaluation. I wish to thank Ronald Andersen, who taught health program evaluation at the University of Chicago (now at UCLA) for many years. He was an early role model and shared many insights about how to package course material in ways that could be grasped readily by graduate students. His evaluation course divided the evaluation process into three distinct phases, and I discovered early that his model also worked very well in my own evaluation courses. I also wish to thank Diane Martin and Rita Altamore, who taught this course at the University of Washington and shared their approaches in teaching this subject with me. I especially want to thank the speakers in my evaluation course for their insights about the practice of program evaluation: Tom Wickizer (design and conduct of program evaluation), Sharyne Shiu-Thornton (conducting evaluation in a cultural context), Beti Thompson (community-based program evaluation), Cynthia Curreri (use of evaluation results in decision making), and Dan Lessler and Nicole Urban (cost-effectiveness analysis). I also benefited greatly from “lessons learned” about evaluation through my work with other faculty—Michael Chapko, Douglas Conrad, Paula Diehr, Mary Durham, Louis Fiset, Peter Milgrom, Donald Patrick, and others at the University of Washington, as well as Don Dillman and John Tarnai at Washington State University.

    Several people played important roles in the production of this book. I wish to thank the W. K. Kellogg Foundation for its generous grant award, which created the time and opportunity for me to write this book. I also wish to thank everyone who provided comments on previous versions of this book. I especially want to thank the students in my health program evaluation class, particularly Letitia Reason and Henry Espinoza, for providing detailed comments for each chapter. Many thanks also are extended to the Sage reviewers. Their thoughtful comments significantly improved the quality of this textbook: Ronald Andersen, University of California, Los Angeles; Robin Lin Miller, University of Illinois at Chicago; William L. Miller, MD, Lehigh Valley Hospital, Allentown, Pennsylvania; Lynn Overman, University of Alabama at Birmingham; and Letitia Reason, University of Washington.

    I am extremely grateful for the patience and help that I received from my secretary, Alice Gronski, in producing the final manuscript. Her organizational skills and attention to detail were invaluable in completing the final manuscript. I also thank and acknowledge the efforts of Laura Hammond and Daniel Moore, who provided administrative support for producing earlier versions of the textbook.

    Finally, I thank my family, as well as my mother and father, and siblings Linda and Gary—for their patience, support, and encouragement in this effort.

  • References

    AdamsPF, BensonV.Current estimates from the National Health Interview Survey, 1989. National Center for Health Statistics. Vital and Health Statistics1990; 10(176).
    AdayLA.Designing and Conducting Health Surveys. San Francisco: Jossey-Bass; 1996.
    AdayLA, BegleyC, LairsonDR, SlaterC.Evaluating the Healthcare System: Effectiveness, Efficiency, and Equity.
    2nd ed
    . Chicago: Health Administration Press; 1998.
    AffholterDP.Outcome monitoring. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:96-118.
    AikenLS, WestSG, SchwalmDE, CarrollJL, HsiungS.Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation. Evaluation Review1998; 22(2):207-244. http://dx.doi.org/10.1177/0193841X9802200203
    American Evaluation Association, Task Force on Guiding Principles for Evaluators. Guiding principles for evaluators. New Directions for Program Evaluation1995; 66:19-26.
    American Psychological Association. Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association; 1985.
    AndersenRM.Unpublished course syllabus, Evaluation Research: Problems and Techniques in Health Services (Business 589). Chicago: University of Chicago, Graduate School of Business, Center for Health Administration Studies, Graduate Program in Health Administration; 1988 (Autumn).
    AndersonOW.Health Services in the United States: A Growth Enterprise Since 1875. Ann Arbor, MI: Health Administration Press; 1985.
    AstinJA, HarknessE, ErnstE.The efficacy of “distant healing”: A systematic review of randomized trials. Annals of Internal Medicine2000; 132(11):903—910. http://dx.doi.org/10.7326/0003-4819-132-11-200006060-00009
    BanduraA.Self-efficacy mechanism in psychobiologic functioning. In: SchwarzerR, ed. Self-Efficacy: Thought Control of Action. Washington, DC: Hemisphere Publishing Company; 1992:355-394.
    BantaHD, LuceBR.Assessing the cost-effectiveness of prevention. Journal of Community Health1983; 9(2):145-165. http://dx.doi.org/10.1007/BF01349877
    BaronRM, KennyDA.The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology1986; 51(6):1173-1182. http://dx.doi.org/10.1037/0022-3514.51.6.1173
    BeckerH.Whose side are you on?Social Problems1967; 14:239-248. http://dx.doi.org/10.2307/799147
    BellJB.Managing evaluation projects step-by-step. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:510-533.
    BerkRA.An introduction to sample selection bias in sociological data. American Sociological Review1983; 48 (June):386-398. http://dx.doi.org/10.2307/2095230
    BersteneTG.Commentary: Process is important. American Journal of Evaluation1999;20(1):116-118.
    BlalockHM.Causal Inferences in Nonexperimental Research. New York: W. W. Norton & Company; 1964.
    BlalockHM Jr.Social Statistics. New York: McGraw-Hill; 1972.
    BlumenthalD.The variation phenomenon in 1994. New England Journal of Medicine1994;331(15):1017-1018. http://dx.doi.org/10.1056/NEJM199410133311511
    BorensteinPE, HarvilchuckJD, RosenthalBH, SantelliJS.Patterns of ICD-9 diagnoses among adolescents using school-based clinics: Diagnostic categories by school level and gender. Journal of Adolescent Health1996; 18(3):203-210. http://dx.doi.org/10.1016/1054-139X%2895%2900013-I
    BoudonR, BourricaudF.A Critical Dictionary of Sociology. Chicago: University of Chicago Press; 1986.
    BourqueLB, ClarkVA.Processing Data: The Survey Example. Sage Series in the Quantitative Applications in the Social Sciences, No. 85. Thousand Oaks, CA: Sage Publications; 1992.
    BrandonPR.Involving program stakeholders in reviews of evaluators' recommendations for program revisions. Evaluation and Program Planning1999; 22:363-372. http://dx.doi.org/10.1016/S0149-7189%2899%2900030-0
    BrownJB, AdamsME.Patients as reliable reporters of medical care process: Recall of ambulatory encounter events. Medical Care1992; 30(5):400-411. http://dx.doi.org/10.1097/00005650-199205000-00003
    BrownNL, PenneylegionMT, HillardP.A process evaluation of condom availability in the Seattle, Washington public schools. Journal of School Health1997; 67(8):336-340. http://dx.doi.org/10.1111/j.1746-1561.1997.tb03469.x
    BuchnerD, CaspersonC, CarterW, WagnerE, MartinM.A New Physical Activity Questionnaire for Older Adults. Unpublished manuscript. Seattle: University of Washington; 1991.
    BurtMR.Practical Methods for Counting the Homeless: A Manual for State and Local Jurisdictions. Washington, DC: The Urban Institute; 1996.
    CampbellDT, BoruchRF.Making the case for randomized assignment to treatments by considering the alternatives. In: BennettCA, LumsdaineAA, eds. Evaluation and Experiment. New York: Academic Press; 1975:195-297.
    CampbellDT, StanleyJC.Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin; 1963.
    CarminesEG, ZellerRA.Reliability and Validity Assessment. Thousand Oaks, CA: Sage Publications; 1979.
    CaudleSL.Using qualitative approaches. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:69-95.
    Centers for Disease Control and Prevention. Framework for program evaluation in public health. Morbidity and Mortality Weekly Report1999; 48(No. RR-11; September 17).
    CharnsMP, SchaeferMJ.Health Care Organizations: A Model for Management. Englewood Cliffs, NJ: Prentice-Hall; 1983.
    ChassinMR, KosecoffJ, ParkRE, WinslowCM, KahnKL, MerrickNJ, FinkA, KeeseyJ, SolomonDH, BrooksRH.The Appropriateness of Selected Medical and Surgical Procedures. Ann Arbor, MI: Health Administration Press; 1989.
    ChelimskyE.The politics of program evaluation. Society1987; 25(1):24-32. http://dx.doi.org/10.1007/BF02695393
    ChenHT.Theory-Driven Evaluations. Newbury Park, CA: Sage Publications; 1990.
    ChenHT, RossiPH.Evaluating with sense: The theory-driven approach. Evaluation Review1983; 7(3):283-302. http://dx.doi.org/10.1177/0193841X8300700301
    CherkinDC, GrothausL, WagnerEH.The effect of office visit copayments on utilization in a health maintenance organization. Medical Care1989; 27(11):1036-1045. http://dx.doi.org/10.1097/00005650-198911000-00005
    ClancyCM, EisenbergJM.Outcomes research: Measuring the end results of health care. Science. October 9, 1998:245-246. http://dx.doi.org/10.1126/science.282.5387.245
    CohenJ.Statistical Power Analysis for the Behavioral Sciences.
    Rev. ed
    . New York: Academic Press; 1977.
    CohenJ.Statistical Power Analysis for the Behavioral Sciences.
    2nd ed
    . Hillsdale, NJ: Lawrence Erlbaum; 1988.
    CoileRC.The Five Stages of Managed Care: Strategies for Providers, HMOs, and Suppliers. Chicago: Health Administration Press; 1997.
    ColeGE.Advancing the development and application of theory-based evaluation in the practice of public health. American Journal of Evaluation1999; 20(3):453-470.
    ColtonD.The design of evaluations for continuous quality improvement. Evaluation & the Health Professions1997; 20(3):265-285. http://dx.doi.org/10.1177/016327879702000302
    ConnerRF.Selecting a control group: An analysis of the randomization process in twelve social reform programs. Evaluation Quarterly1977; 1(2):195-244. http://dx.doi.org/10.1177/0193841X7700100201
    ConradDA, MaynardC, CheadleA, et al. Primary care physician compensation method in medical care groups: Does it influence the use and cost of health services for enrollees in managed care organizations?JAMA1998; 279(11):853-858. http://dx.doi.org/10.1001/jama.279.11.853
    CookTD.A quasi-sampling theory of the generalization of causal relationships. New Directions for Evaluation1993; 57(Spring):39-82.
    CookTD, CampbellDT.Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin; 1979.
    CordrayDS, FischerRL.Synthesizing evaluation findings. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:198-231.
    CousinsJB, WhitmoreE.Framing participatory evaluation. New Directions for Evaluation1998; 80(Winter):5-23. http://dx.doi.org/10.1002/ev.1114
    CronbachLJ.Coefficient alpha and the internal structure of tests. Psychometrika1951; 16:297-334. http://dx.doi.org/10.1007/BF02310555
    CronbachLJ.Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass; 1982.
    CummingsSR, RubinSM, OsterG.The cost-effectiveness of counseling smokers to quit. JAMA1989; 261(1):75-79. http://dx.doi.org/10.1001/jama.1989.03420010085038
    Curbing Native American smoking [transcript]. “Morning Edition.” National Public Radio. December 9, 1998.
    DavidoffAJ, PoweNR.The role of perspective in defining economic measures for the evaluation of medical technology. International Journal of Technology Assessment in Health Care1996; 12(1):9-21. http://dx.doi.org/10.1017/S026646230000934X
    DavisJE.Commentary: Advocacy, care, and power. American Journal of Evaluation1999;20(1):119-122.
    DeanDL.How to use focus groups. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:338-349.
    DenistonOL, RosenstockIM.The validity of nonexperimental designs for evaluating health services. Health Services Reports1973; 88(2):153-164. http://dx.doi.org/10.2307/4594755
    DennisML.Ethical and practical randomized field experiments. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:155-197.
    DenzinNK.The logic of naturalistic inquiry. In DenzinNK, ed. Sociological Methods: A Sourcebook. New York: McGraw-Hill; 1978:6-29.
    DetskyAS, RedelmeierDA.Measuring health outcomes—putting gains into perspective. New England Journal of Medicine1998; 339(6):402-404. http://dx.doi.org/10.1056/NEJM199808063390611
    DeVellisRF.Scale Development: Theory and Applications. Newbury Park, CA: Sage Publications; 1991.
    DeyoRA, DiehrP, PatrickDL.Reproducibility and responsiveness of health status measures. Controlled Clinical Trials1991; 12:142S-158S. http://dx.doi.org/10.1016/S0197-2456%2805%2980019-4
    DeyoRA, PsatyBM, SimonG, WagnerEH, OmennGS.The messenger under attack—intimidation of researchers by special-interest groups. New England Journal of Medicine1997; 336(16):1176-1180. http://dx.doi.org/10.1056/NEJM199704173361611
    Di LimaSN, SchustCS.Community Health Education and Promotion: A Guide to Program Design and Evaluation. Gaithersburg, MD: Aspen; 1997.
    DiehrPK, RichardsonWC, ShortellSM, LoGerfoJP.Increased access to medical care: The impact on health. Medical Care1979; 27(10):989-999. http://dx.doi.org/10.1097/00005650-197910000-00002
    DixonAS.The evolution of clinical policies. Medical Care1990; 28:201-220. http://dx.doi.org/10.1097/00005650-199003000-00002
    DonabedianA.Aspects of Medical Care Administration Specifying Requirements for Health Care. Cambridge, MA: Harvard University Press; 1973.
    DoubiletP, WeinsteinM, McNeilBJ.Use and misuse of the term “cost-effectiveness” in medicine. New England Journal of Medicine1986; 314:253-256. http://dx.doi.org/10.1056/NEJM198601233140421
    DrummondMF, StoddardGL, TorranceGW.Methods for Economic Evaluation of Health Programs. New York: Oxford University Press; 1997.
    DuanN, ManningWG Jr., MorrisCN, NewhouseJP.A comparison of alternative models for the demand for medical care. Journal of Business & Economic Studies1983; 1(2):115-126.
    DurhamM, BeresfordS, DiehrP, GrembowskiD, HechtJ, PatrickD.Participation of higher users in a randomized trial of Medicare reimbursement for preventive services. The Gerontologist1991; 31(5):603-606. http://dx.doi.org/10.1093/geront/31.5.603
    EastmondN.Commentary: When funders want to compromise your design. American Journal of Evaluation1998; 19(3):392-395.
    EdelsonJT, WeinsteinMC, TostesonA, WilliamsL, LeeTH, GoldmanL.Long-term cost-effectiveness of various initial monotherapies for mild or moderate hypertension. JAMA1990;263(3):407-413. http://dx.doi.org/10.1001/jama.1990.03440030094028
    EisenbergJM.Clinical economics: A guide to the economic analysis of clinical practice. JAMA1989; 262(20):2879-2886. http://dx.doi.org/10.1001/jama.1989.03430200123038
    ElixhauserA, HalpernM, SchmierJ, LuceBR.Health care CBA and CEA from 1991 to 1996: An updated bibliography. Medical Care1998; 36(Suppl. 5):MS1-MS9.
    EmeryDD, SchneidermanLJ. Cost-effectiveness analysis in health care. Hastings Center Report1989; 19(4):8-13. http://dx.doi.org/10.2307/3562293
    FaludiA.Planning Theory. Oxford, UK. Pergamon Press; 1973.
    FettermanDM.Ethnography. In: BickmanL, RogDJ, eds. Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications; 1998: 473-504.
    FettermanDM, KafarianSJ, WandersmanA.Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, CA: Sage Publications; 1996. http://dx.doi.org/10.4135/9781452243573
    FinkA.Evaluation Fundamentals: Guiding Health Programs, Research, and Policy. Newbury Park, CA: Sage Publications; 1993.
    FlemingGV, AndersenRA.The Municipal Health Services Program: Improving access to primary care without increasing expenditures. Medical Care1986; 24:565-579. http://dx.doi.org/10.1097/00005650-198607000-00001
    FlemingNS, BeckerER.The impact of the Texas 1989 motorcycle helmet law on total and head-related fatalities, severe injuries, and overall injuries. Medical Care1992; 30(9):832-845. http://dx.doi.org/10.1097/00005650-199209000-00007
    FloydRL, RimerBK, GiovinoGA, MullenPD, SullivanS.A review of smoking in pregnancy: Effects on pregnancy outcomes and cessation efforts. Annual Review of Public Health1993; 14:379-411. http://dx.doi.org/10.1146/annurev.pu.14.050193.002115
    FriedlanderD, RobinsP.Estimating the Effect of Employment and Training Programs: An Assessment of Some Nonexperimental Techniques. New York: Manpower Demonstration Research Corporation; 1994.
    FriedmanGD.Primer of Epidemiology. New York: McGraw-Hill; 1994.
    FuchsVR.Who Shall Live? Health, Economics and Social Choice. New York: Basic Books; 1974.
    FuchsVR.The Health Economy. Cambridge: Harvard University Press; 1986.
    GarberAM, WeinsteinMC, TorranceGW, KamletMS.Theoretical foundations of cost-effectiveness analysis. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:25-53.
    GehlbachS.Interpreting the Medical Literature. New York: McGraw-Hill; 1993.
    GlasgowRE, VogtTM, BolesSM.Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health1999;89(9):1322-1327. http://dx.doi.org/10.2105/AJPH.89.9.1322
    GoldMR, PatrickDL, TorranceGW, FrybackDG, HadhornDC, KamletMS, DanielsN, WeinsteinMC.Identifying and valuing outcomes. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:82-134.
    GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press, 1996.
    GordonMJ.Research workbook: A guide for initial planning of clinical, social, and behavioral research projects. Journal of Family Practice1978; 7(1): 145-160.
    GortmakerSL, ClarkCJG, GravenSN, SobolAM, DeronimusA.Reducing infant mortality in rural America: Evaluation of the Rural Infant Care Program. Health Services Research1987; 22:91-116.
    GottfredsonDC, GottfredsonGD, SkrobanS.Can prevention work where it is needed most?Evaluation Review1998; 22(3):315-340. http://dx.doi.org/10.1177/0193841X9802200301
    GrahamJD, CorsoPS, MorrisJM, Segui-GomezM, WeinsteinMC.Evaluating the cost-effectiveness of clinical and public health measures. Annual Review of Public Health1998; 19:125-152. http://dx.doi.org/10.1146/annurev.publhealth.19.1.125
    GreenLW, KreuterMW.Health promotion planning: An educational and ecological approach. Mountain View, CA: Mayfield Publishing Company; 1999.
    GreenLW, LewisFM.Measurement and Evaluation in Health Education and Health Promotion. Palo Alto, CA: Mayfield Publishing Company; 1986.
    GreeneJC.Evaluation as advocacy. Evaluation Practice1997; 18(1):25-35. http://dx.doi.org/10.1016/S0886-1633%2897%2990005-2
    GrembowskiD.Causal models in plan evaluation. Socio-Economic Planning Sciences1984; 18(4):255-261. http://dx.doi.org/10.1016/0038-0121%2884%2990006-5
    GrembowskiD, BlalockAB.Evaluating program implementation. In: BlalockAB, ed. Evaluating Social Programs at the State and Local Level: The JTPA Evaluation Design Project. Kalamazoo, MI: W. E. Upjohn Institute for Employment Research; 1990:229-297.
    GrembowskiD, CookK, PatrickDL, RousselAE.Managed care and physician referral. Medical Care Research and Review1998; 55(1):3-31. http://dx.doi.org/10.1177/107755879805500101
    GrembowskiD, PatrickD, DiehrP, DurhamM, BeresfordS, KayE, HechtJ.Self-efficacy and health behavior among older adults. Journal of Health and Social Behavior1993;34:89-104. http://dx.doi.org/10.2307/2137237
    GriffinJF, HoganJW, BuechnerJS, LeddyTM.The effect of a Medicaid managed care program on the adequacy of prenatal care utilization in Rhode Island. American Journal of Public Health1999; 89(4):497-501. http://dx.doi.org/10.2105/AJPH.89.4.497
    GrossmanJ, TierneyJP.The fallibility of comparison groups. Evaluation Review1993; 17(5):556-571. http://dx.doi.org/10.1177/0193841X9301700505
    GuyattG, WalterS, NormanG.Measuring change over time: Assessing the usefulness of evaluative instruments. Journal of Chronic Disease1987; 40(2): 171-178. http://dx.doi.org/10.1016/0021-9681%2887%2990069-5
    HaddixAC, TeutschSM, ShafferPA.Decision Analysis and Economic Evaluation. New York: Oxford University Press; 1996.
    HallamJ, PetosaR.A worksite intervention to enhance social cognitive theory constructs to promote exercise adherence. American Journal of Health Promotion1998;13(1):4-7. http://dx.doi.org/10.4278/0890-1171-13.1.4
    HannanEL, KilburnH, RaczM, ShieldsE, ChassinMR.Improving the outcomes of coronary artery bypass surgery in New York State. JAMA1994; 271(10):761-766. http://dx.doi.org/10.1001/jama.1994.03510340051033
    HatziandreuEJ, SacksJJ, BrownR, TaylorWR, RosenbergML, GrahamJD.The cost effectiveness of three programs to increase use of bicycle helmets among children. Public Health Reports1995; 110(3):251-259.
    HayesR, BakerDW.Methodological problems in comparing English-speaking and Spanish-speaking patients' satisfaction with interpersonal aspects of care. Medical Care1998; 36(2):230-236. http://dx.doi.org/10.1097/00005650-199802000-00011
    HedrickSC, KoepsellTD, InuiT.Meta-analysis of home care effects on mortality and nursing home placement. Medical Care1989; 27:1015-1026. http://dx.doi.org/10.1097/00005650-198911000-00003
    HendricksM.Making a splash: Reporting evaluation results effectively. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:549-575.
    HendricksM, HandleyEA.Improving the recommendations from evaluation studies. Evaluation and Program Planning1990; 13:109-117. http://dx.doi.org/10.1016/0149-7189%2890%2990038-X
    HendricksM, PapagiannisM.Do's and Don'ts for offering effective recommendations. Evaluation Practice1990; 11(2):121-125. http://dx.doi.org/10.1016/0886-1633%2890%2990040-K
    HimmelsteinDU, LewontinJP, WoolhandlerS.Who administers? Who cares? Medical administrative and clinical employment in the United States and Canada. American Journal of Public Health1996; 86:172-178. http://dx.doi.org/10.2105/AJPH.86.2.172
    HulscherM, WensingM, GrolR, Van der WeijdenT, Van WeelC.Interventions to improve the delivery of preventive services in primary care. American Journal of Public Health1999; 89(5):737-746. http://dx.doi.org/10.2105/AJPH.89.5.737
    HuntingtonJ, ConnellFA.For every dollar spent—the cost-savings argument for prenatal care. New England Journal of Medicine1994; 331(19):1303-1307. http://dx.doi.org/10.1056/NEJM199411103311910
    JaccardJ, TurrisiR, WanCK.Interaction Effects in Multiple Regression. Sage Series in the Quantitative Applications in the Social Sciences, No. 72. Thousand Oaks, CA: Sage Publications; 1990.
    Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. Thousand Oaks, CA: Sage Publications; 1994.
    JoyceT, CormanH, GrossmanM.A cost-effectiveness analysis of strategies to reduce infant mortality. Medical Care1988; 26(4):348-360. http://dx.doi.org/10.1097/00005650-198804000-00004
    KaneRL, ed. Understanding Health Outcomes Research. Gaithersburg, MD: Aspen; 1997.
    KaneRL, HensonR, DenistonOL.Program evaluation: Is it worth it? In: KaneRL, ed. The Challenges of Community Medicine. New York: Springer; 1974:213—223.
    KaplanA.The Conduct of Inquiry: Methodology for Behavioral Science. San Francisco: Chandler Publishing Company; 1964.
    KaplanRM, BushJW, BerryCC.Health status: Types of validity and the Index of Well-Being. Health Services Research1976; 11:478-507.
    KingG, KeohaneRO, VerbaS.Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press; 1994.
    KingJA, MorrisLL, Fitz-GibbonCT.How to Assess Program Implementation. Newbury Park, CA: Sage Publications; 1987.
    KirbyD, BrenerND, BrownNL, PeterfreundN, HillardP, HarristR.The impact of condom distribution in Seattle schools on sexual behavior and condom use. American Journal of Public Health1999; 89(2):182-187. http://dx.doi.org/10.2105/AJPH.89.2.182
    KirkJ.Reliability and Validity in Qualitative Research. Thousand Oaks, CA: Sage Publications; 1986.
    KishL.Survey Sampling. New York: John Wiley and Sons; 1965.
    KolataG.Researchers ask whether prenatal care truly saves money. New York Times. November 10, 1994.
    KraemerHC, ThiemannS.How Many Subjects? Statistical Power Analysis in Research. Newbury Park, CA: Sage Publications; 1987.
    KriegerJW, ConnellFA, LoGerfoJP.Medicaid prenatal care: A comparison of use and outcomes in fee-for-service and managed care. American Journal of Public Health1992;82:185-190. http://dx.doi.org/10.2105/AJPH.82.2.185
    KristalAR, ShattuckAL, HenryHJ, FowlerAS.Rapid assessment of dietary intake of fat, fiber, and saturated fat: Validity of an instrument suitable for community intervention research and nutritional surveillance. American Journal of Health Promotion1990; 4:288-295. http://dx.doi.org/10.4278/0890-1171-4.4.288
    KruegerRA.Focus Groups: A Practical Guide for Applied Research. Thousand Oaks, CA: Sage Publications; 1994.
    KrugEG, KresnowM, PeddicordJP, DahlbergLL, PowellKE, CrosbyAE, AnnestJL.Suicide after natural disasters. New England Journal of Medicine1998; 338(6):373-378. http://dx.doi.org/10.1056/NEJM199802053380607
    LandisKR, KochGG.The measurement of observer agreement for categorical data. Biometrics1977; 33(March):159-174. http://dx.doi.org/10.2307/2529310
    LeePR, EstesCL.The Nation's Health. Boston: Jones and Bartlett; 1994.
    LeorJ, PooleK, KlonerRA.Sudden cardiac death triggered by an earthquake. New England Journal of Medicine1996; 334:413-419. http://dx.doi.org/10.1056/NEJM199602153340701
    LevitKR, LazenbyHC, BradenBR, et al. National health expenditures, 1996. Health Care Financing Review1997; 19(1):161-200.
    LevitKR, LazenbyHC, LetschSW, CowanCA.National health care spending, 1989. Health Affairs1991; 10(1):117-130. http://dx.doi.org/10.1377/hlthaff.10.1.117
    LimaS, SchustCS.Community Health Education and Promotion; A Guide to Program Design and Evaluation. Gaithersburg, MD: Aspen; 1997.
    LipscombJ, WeinsteinMC, TorranceGW.Time preference. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:214-235.
    LipseyMW.Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Publications; 1990.
    LipseyMW.Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. In: CookTD, CooperH, CordrayDS, HartmannH, HedgesLV, LightRJ, LouisTA, MostellerF, eds. Meta-Analysis for Explanation: A Casebook. New York: Russell Sage Foundation; 1992:83-127.
    LipseyMW.Theory as method: Small theories of treatments. New Directions for Program Evaluation1993; 57(Spring):5-38. http://dx.doi.org/10.1002/ev.1637
    LipseyMW.What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. New Directions for Evaluation1997; 76(Winter):7-39. http://dx.doi.org/10.1002/ev.1084
    LipseyMW.Design sensitivity: Statistical power for applied experimental research. In: BickmanL, RogDJ, eds. Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications; 1998:39-68.
    LipseyMW, WilsonDB.The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist1993; 48(12):1181-1209. http://dx.doi.org/10.1037/0003-066X.48.12.1181
    LoflandJ, LoflandLH.Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, CA: Wadsworth Publishing; 1984.
    LoftinC, McDowallD, WiersemaB, CotteyTJ.Effects of restrictive licensing of handguns on homicide and suicide in the District of Columbia. New England Journal of Medicine1991; 325(23):1615-1620. http://dx.doi.org/10.1056/NEJM199112053252305
    LohrB.An Historical View of the Research on the Factors Related to the Utilization of Health Services. Rockville, MD: Bureau for Health Services Research and Evaluation, Social and Economic Analysis Division; January 1972.
    LuceBR, ManningWG, SiegelJE, LipscombJ.Estimating costs in cost-effectiveness analysis. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:176-213.
    LuceBR, SimpsonK.Methods of cost-effectiveness analysis: Areas of consensus and debate. Clinical Therapeutics1995; 17(1):109-125. http://dx.doi.org/10.1016/0149-2918%2895%2980012-3
    LuepkerRV, PerryCL, McKinlaySM, NaderPR, ParcelGS, StoneEJ, WebberLS, ElderJP, FeldmanHA, JohnsonCC, KelderSH, WuM.Outcomes of a field trial to improve children's dietary pattern and physician activity: The Child and Adolescent Trial for Cardiovascular Health (CATCH). JAMA1996; 275:768-776. http://dx.doi.org/10.1001/jama.1996.03530340032026
    LuftHS.Health services research as a scientific process: The metamorphosis of an empirical research project from grant proposal to final report. Health Services Research1986; 21:563-584.
    LundbergGD, WennbergJE.JAMA theme issue on quality of care: A new proposal and call to action. JAMA1997; 278:1615-1616. http://dx.doi.org/10.1001/jama.1997.03550190079049
    MacMahonB, TrichopoulusD.Epidemiology: Principles and Methods. Boston: Little, Brown and Company; 1996.
    MaddalaGS.Limited-Dependent and Qualitative Variables in Econometrics. Cambridge, UK: Cambridge University Press; 1983. http://dx.doi.org/10.1017/CBO9780511810176
    MarcantonioRJ, CookTD.Convincing quasi-experiments: The interrupted time series and regression-discontinuity designs. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:133-154.
    MarkDB, HlatkyMA, CaliffRM, et al. Cost effectiveness of thrombolytic therapy with tissue plasminogen activator as compared with streptokinase for acute myocardial infarction. New England Journal of Medicine1995; 332:1418-1424. http://dx.doi.org/10.1056/NEJM199505253322106
    MarkMM, ShotlandRL.Multiple Methods in Program Evaluation. New Directions for Program Evaluation, No. 35. San Francisco: Jossey-Bass; 1987.
    MarmorTR.Forecasting American health care: How we got here and where we might be going. Journal of Health Politics, Policy, and Law1998; 23(3):551-571.
    MartinD, DiehrP, PriceKF, RichardsonWC.Effect of a gatekeeper plan on health services use and charges: A randomized trial. American Journal of Public Health1989; 79:1628-1632. http://dx.doi.org/10.2105/AJPH.79.12.1628
    MastersBA.Bid to help women saving men's lives; Study: Domestic-violence programs keep batterers alive. Seattle Times (from the Washington Post), March 15, 1999:A1.
    MaxwellJA.Designing a qualitative study. In: BickmanL, RogDJ, eds. Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications; 1998:69-100.
    McCainLJ, McClearyR.The statistical analysis of the simple interrupted time-series quasi-experiment. In: CookTD, CampbellDT, eds. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston: Houghton Mifflin; 1979:233-293.
    McDowallD, McClearyR, MeidingerEE, HayRA.Interrupted Time Series Analysis. Sage Series in the Quantitative Applications in the Social Sciences, No. 21. Thousand Oaks, CA: Sage Publications; 1980.
    McDowellCL.Standardized tests and program evaluation: Inappropriate measures in critical times. New Directions for Program Evaluation1992; 53:45-54. http://dx.doi.org/10.1002/ev.1600
    McGinnisJM, FoegeWH.Actual causes of death in the United States. JAMA1993; 270(18):2207-2212. http://dx.doi.org/10.1001/jama.1993.03510180077038
    McGrawSA, SellersDE, StoneEJ, BebchukJ, EdmundsonEW, JohnsonCC, BachmanKJ, LuepkerRV.Using process data to explain outcomes: An illustration from the Child and Adolescent Trial for Cardiovascular Health (CATCH). Evaluation Review1996; 20(3):291-312. http://dx.doi.org/10.1177/0193841X9602000304
    MeischkeH, FinneganJ, EisenbergM.What can you teach about cardiopulmonary resuscitation (CPR) in 30 seconds?Evaluation and the Health Professions1999; 22(1):44-59.
    MellorS, MarkMM.A quasi-experimental design for studies on the impact of administrative decisions: Applications and extensions of the regression-discontinuity design. Organizational Research Methods1998; 1(3):315-333. http://dx.doi.org/10.1177/109442819813003
    MerriamSB.Qualitative Research and Case Study Applications in Education. San Francisco: Jossey-Bass; 1998.
    MilesMB, HubermanAM.Qualitative Data Analysis: A Sourcebook of New Methods. Thousand Oaks, CA: Sage Publications; 1984.
    MilesMB, HubermanAM.Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, CA: Sage Publications; 1994.
    MillerDC.Handbook of Research Design and Social Measurement. Newbury Park, CA: Sage Publications; 1991.
    MintzbergH.Structure in Fives: Designing Effective Organizations. Englewood Cliffs, NJ: Prentice-Hall; 1983.
    MohrLB.Impact Analysis for Program Evaluation. Chicago: The Dorsey Press; 1988.
    MohrLB.Impact Analysis for Program Evaluation. Thousand Oaks, CA: Sage Publications; 1995.
    MorrisLL, Fitz-GibbonCT.How to Measure Program Implementation. Thousand Oaks, CA: Sage Publications; 1978.
    MorrisM.Ethical challenges. American Journal of Evaluation1998; 19(3):381-384.
    MorrisM.The case of the desperate staff. American Journal of Evaluation1999; 20(1):115.
    MorrisM.The off-the-record case. American Journal of Evaluation2000; 21(1):121.
    MoskowitzDB.Looking for new ways to measure the payoff from employer health benefits. Perspectives on the Marketplace (suppl. to Medicine and Health). May 4, 1998:1-2.
    NamboodiriNK, CarterLF, BlalockHM.Applied Multivariate Analysis and Experimental Design. New York: McGraw-Hill; 1975.
    NeuhauserD, YinXP.Deciding whether a new test/measure is useful. Medical Care1991;29(8):685-689. http://dx.doi.org/10.1097/00005650-199108000-00001
    NewhouseJP.Controlled experimentation as research policy. In: GinzbergE, ed. Health Services Research: Key to Health Policy. Cambridge, MA: Harvard University Press; 1991:161-194.
    NewhouseJP, McClellanM.Econometrics in outcomes research: The use of instrumental variables. Annual Review of Public Health1998; 19:17-34. http://dx.doi.org/10.1146/annurev.publhealth.19.1.17
    NunnallyJC.Educational Measurement and Evaluation. New York: McGraw-Hill; 1972.
    OrwinRG, SonnefeldLJ, CordrayDS, PionGM, PerlHI.Constructing quantitative implementation scales from categorical services data: Examples from a multisite evaluation. Evaluation Review1998; 22(2):245-288. http://dx.doi.org/10.1177/0193841X9802200204
    OstromCW.Time Series Analysis: Regression Techniques. Sage Series in the Quantitative Applications in the Social Sciences, No. 9. Thousand Oaks, CA: Sage Publications; 1978.
    O'SullivanE, BurlesonGW, LambWE.Avoiding evaluation cooptation: Lessons from a renal dialysis evaluation. Evaluation and Program Planning1985; 8(3):255-259. http://dx.doi.org/10.1016/0149-7189%2885%2990047-3
    PalumboDJ, ed. The Politics of Program Evaluation. Newbury Park, CA: Sage Publications; 1987.
    PardesH, MantonKG, LanderES, TolleyHD, UllianAD, PalmerH.Effects of medical research on health care and the economy. Science. January 1, 1999:36-37. http://dx.doi.org/10.1126/science.283.5398.36
    PatrickDL, BeresfordSA, EhrethJ, DiehrP, PiccianoJ, DurhamM, GrembowskiD.Interpreting excess mortality in a prevention trial of older adults. International Journal of Epidemiology1995; 24(Suppl. 1 Pt. 3):S27-S33.
    PatrickDL, EricksonP.Health Status and Health Policy: Quality of Life in Health Care Evaluation and Resource Allocation. New York: Oxford University Press; 1993.
    PatrickDL, GrembowskiD, DurhamML, BeresfordSA, DiehrP, EhrethJ, HechtJ, PiccianoJ, BeeryW.Cost and outcomes of Medicare reimbursement for HMO preventive services. Health Care Financing Review1999; 20(4):25-43.
    PattonMQ.How to Use Qualitative Methods in Evaluation. Thousand Oaks, CA: Sage Publications; 1987.
    PattonMQ.Qualitative Evaluation and Research Methods. Thousand Oaks, CA: Sage Publications; 1990.
    PattonMQ.Utilization-Focused Evaluation: The New Century Text. Thousand Oaks, CA: Sage Publications; 1997.
    PentzMA, TrebowEA, HansenWB, MacKinnonDP, DwyerJH, JohnsonCA, FlayBR, DanielsS, CormackC.Effects of program implementation on adolescent drug use behavior: The Midwestern Prevention Project (MPP). Evaluation Review1990; 14(3):264-289. http://dx.doi.org/10.1177/0193841X9001400303
    PetittiDB.Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis. New York: Oxford University Press; 1994.
    PhillipsKA, HoltgraveDR.Using cost-effectiveness/cost-benefit analysis to allocate health resources: A level playing field for prevention?American Journal of Preventive Medicine1997; 13(1):18-25.
    PierceJP, GilpinEA, EmerySL, WhiteMM, RosbrookB, BerryCC.Has the California tobacco control program reduced smoking?JAMA1998; 280(10):893-899. http://dx.doi.org/10.1001/jama.280.10.893
    President and Fellows of Harvard College. Lead Poisoning, Parts A and B (Case Program, John F. Kennedy School of Government, Documents C14-75-123.0 and C14-75-124.0). Cambridge, MA: Harvard University; 1975.
    PressmanJL, WildavskyA.Implementation. Berkeley: University of California Press; 1984.
    RadloffLS.The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement1977; 1:385-401. http://dx.doi.org/10.1177/014662167700100306
    ReichardtCS.Estimating the effects of community prevention trials: Alternative designs and methods. In: HolderHD, HowardJM, eds. Community Prevention Trials for Alcohol Problems: Methodological Issues. Westport, CT: Praeger; 1992:137-158.
    ReichardtCS, CookTS.Beyond qualitative versus quantitative methods. In: CookTS, ReichardtCS, eds. Qualitative and Quantitative Methods in Evaluation Research. Thousand Oaks, CA: Sage Publications; 1979:7-32.
    ReichardtCS, MarkMM.Quasi-experimentation. In: BickmanL, RogDJ, eds. Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications; 1998:193-228.
    ReidRJ.A cost-benefit analysis of syringe exchange programs. Journal of Health & Social Policy2000; 11(4):41-57. http://dx.doi.org/10.1300/J045v11n04_03
    RelmanAS.Assessment and accountability: The third revolution in medical care. New England Journal of Medicine1988; 319:1220-1222. http://dx.doi.org/10.1056/NEJM198811033191810
    ReynoldsAJ.Confirmatory program evaluation: A method for strengthening causal inference. American Journal of Evaluation1998; 19(2):203-221.
    Roberts-GrayC, BullerA, SparkmanA.Linking data with action: Procedures for developing recommendations. Evaluation Review1987; 11:678-684. http://dx.doi.org/10.1177/0193841X8701100507
    Rog, DJ.Constructing natural “experiments.” In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:119-132.
    RogersLW, BergmanAB, RivaraFP.Promoting bicycle helmets to children: A campaign that worked. Journal of Musculoskeletal Medicine1991; 8:64-77.
    RossiPH.Advances in quantitative evaluation, 1987-1996. New Directions for Evaluation1997; 76(Winter):57-68. http://dx.doi.org/10.1002/ev.1087
    RossiPH, FreemanHE.Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications; 1993.
    RossiPH, FreemanHE, LipseyMW.Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage Publications; 1999.
    RoyseD, ThyerBA.Program Evaluation: An Introduction. Chicago: Nelson-Hall; 1996.
    RussellLB.The role of prevention in health reform. New England Journal of Medicine1993; 329(5):252-254. http://dx.doi.org/10.1056/NEJM199307293290511
    RussellLB, SiegelJE, DanielsN, GoldMR, LuceBR, MandelblattJS.Cost-effectiveness analysis as a guide to resource allocation in health: Roles and limitations. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine, New York: Oxford University Press; 1996:3-24.
    St. PierreRG.Management and Organizations for Program Evaluation. New Directions for Program Evaluation, No. 18. San Francisco: Jossey-Bass; 1983.
    SalantP, DillmanDA.How to Conduct Your Own Survey. New York: John Wiley and Sons; 1994.
    ScheirerMA.Program Implementation. Thousand Oaks, CA: Sage Publications; 1981.
    ScheirerMA.Designing and using process evaluation. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:40-68.
    ScheirerMA.Commentary: Evaluation planning is the heart of the matter. American Journal of Evaluation1998; 19(3):385-391.
    ScheirerMA, ShediacMC, CassadyCE.Measuring the implementation of health promotion programs: The case of the Breast and Cervical Cancer Program in Maryland. Health Education Research1995; 10(1):11-25. http://dx.doi.org/10.1093/her/10.1.11
    SchlesingerM.On the limits of expanding health care reform: Chronic care in prepaid settings. The Milbank Quarterly1986; 64(2):189-215. http://dx.doi.org/10.2307/3349970
    SchlesingerM, GrayB.A broader vision for managed care, part 1: Measuring the benefit to communities. Health Affairs1998; 17(3):152-168. http://dx.doi.org/10.1377/hlthaff.17.3.152
    SchnelleJF, KirchnerRE, MacraeJW, McNessMP, EckRH, SnodgrassS, CaseyJD, UseltonPH Jr.Police evaluation research: An experimental and cost-benefit analysis of a helicopter patrol in a high-crime area. Journal of Applied Behavior Analysis1978;11:11-21. http://dx.doi.org/10.1901/jaba.1978.11-11
    SchulmanKA, BerlinJA, HarlessW, KernerJF, SistrunS, GershBJ, DubeR, TaleghaniCK, BurkeJE, WilliamsS, EisenbergJM, EscarceJJ.The effect of race and sex on physicians' recommendations for cardiac catheterization. New England Journal of Medicine1999; 340(8):618-626. http://dx.doi.org/10.1056/NEJM199902253400806
    SchweitzerSO, AtchisonKA, LubbenJE, Mayer-OakesSA, De JongFJ, MatthiasRE.Health promotion and disease prevention for older adults: Opportunity for change or preaching to the converted?American Journal of Preventive Medicine1994; 10(4):223-229.
    ShadishWR Jr., CookTD, LevitonLC.Foundations of Program Evaluation. Newbury Park, CA: Sage Publications; 1991.
    SheehanTJ.Stress and low birth weight: A structural modeling approach using real life stressors. Social Science and Medicine1998; 47(10):1503-1512. http://dx.doi.org/10.1016/S0277-9536%2898%2900236-6
    ShortellSM.Suggestions for improving the study of health program implementation. Health Services Research1984; 19(1):117-125.
    ShortellSM.The emergence of qualitative methods in health services research. Health Services Research1999; 34(Suppl., Pt. 2):1083-1090.
    ShortellSM, KaluznyAD.Health Care Management: Organization Design and Behavior. Albany, NY: Delmar Thompson Learning; 2000.
    ShortellSM, McNerneyWJ.Criteria and guidelines for reforming the U.S. health care system. New England Journal of Medicine1990; 322(7):463-467. http://dx.doi.org/10.1056/NEJM199002153220710
    ShortellSM, RichardsonWC.Health Program Evaluation. St. Louis, MO: C. V. Mosby Company; 1978.
    ShroutPE, YagerTJ.Reliability and validity of screening scales: Effect of reducing scale length. Journal of Clinical Epidemiology1989; 42(1):69-78. http://dx.doi.org/10.1016/0895-4356%2889%2990027-9
    SiegelJE, WeinsteinMC, RussellLB, GoldMR.Recommendations for reporting cost-effectiveness analyses. JAMA1996; 276(16):1339-1341. http://dx.doi.org/10.1001/jama.1996.03540160061034
    SiegelJE, WeinsteinMC, TorranceGW.Reporting cost-effectiveness studies and results. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:276-303.
    SiegelKL, TuckelP.The utilization of evaluation research: A case analysis. Evaluation Review1985; 9(3):307-328. http://dx.doi.org/10.1177/0193841X8500900304
    SilvermanM, RicciEM, GunterMJ.Strategies for increasing the rigor of qualitative methods in evaluations of health care programs. Evaluation Review1990; 14(1):57-74. http://dx.doi.org/10.1177/0193841X9001400104
    SimonJL.Basic Research Methods in Social Science: The Art of Empirical Investigation. New York: Random House; 1969.
    SiskJE.The cost of prevention: Don't expect a free lunch. JAMA1993; 269(13):1710, 1715. http://dx.doi.org/10.1001/jama.269.13.1710
    SiskJE, GormanSA, ReisingerAL, GliedSA, DuMouchelWH, HynesMM.Evaluation of Medicaid managed care: Satisfaction, access, and use. JAMA1996; 276:50-55. http://dx.doi.org/10.1001/jama.1996.03540010052030
    SnedecorGW, CochranWG.Statistical Methods. Ames: Iowa State University Press; 1980.
    SolomonMA, ShortellSM.Designing health policy research for utilization. Health Policy Quarterly1981; 1(3):216-237.
    SonnichsenRC.Evaluators as change agents. In: WholeyJS, HatryHP, NewcomerKE, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass; 1994:534-548.
    SrinivasanS, LevittL, LundyJ.Wall Street's love affair with health care. Health Affairs1998; 17(4):126-131. http://dx.doi.org/10.1377/hlthaff.17.4.126
    StewartDW, ShamdasaniPN.Focus group research: Exploration and discovery. In: BickmanL, RogDJ, eds. Handbook of Applied Social Research Methods. Thousand Oaks, CA: Sage Publications; 1998:505-526.
    Stouthamer-LoeberM, van KammenWB.Data Collection and Management: A Practical Guide. Thousand Oaks, CA: Sage; 1995.
    SuchmanEA.Evaluative Research: Principles and Practice in Public Service & Social Action Programs. New York: Russell Sage Foundation; 1967.
    SudmanS, BradburnNM.Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass; 1982.
    TengsTO, AdamsME, PliskinJS, SafranDG, SiegelJE, WeinsteinMC, GrahamJD.Five-hundred life-saving interventions and their cost-effectiveness. Risk Analysis1995; 15(3):369-390. http://dx.doi.org/10.1111/j.1539-6924.1995.tb00330.x
    TeschR.Qualitative Research: Analysis Types and Software Tools. London: Falmer Press; 1990.
    TorranceGW, SiegelJE, LuceBR.Framing and designing the cost-effectiveness analysis. In: GoldMR, SiegelJE, RussellLB, WeinsteinMC, eds. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press; 1996:54-81.
    TrochimW.The regression-discontinuity design. In: SechrestL, PerrinE, BunkerJ, eds. Research Methodology: Strengthening Causal Interpretations of Nonexperimental Data (DHHS Publication No. (PHS) 90-3454). Rockville, MD: Agency for Health Care Policy and Research; May 1990:119-139.
    UdvarhelyiIS, JennisonK, PhillipsRS, EpsteinAM.Comparison of the quality of ambulatory care for fee-for-service and prepaid plans. Annals of Internal Medicine1991;115(5):394-400. http://dx.doi.org/10.7326/0003-4819-115-5-394
    UnutzerJ, PatrickDL, SimonG, GrembowskiD, WalkerE, RutterC, KatonW.Depressive symptoms and the cost of health services in HMO patients aged 65 years and older: A 4-year prospective study. JAMA1997; 277(20):1618-1623. http://dx.doi.org/10.1001/jama.1997.03540440052032
    UrbanN.Cost-Effectiveness Analysis. Class presentation in course HSERV 522, Health Program Evaluation, University of Washington, January 25, 1998.
    U.S. Department of Health and Human Services (DHHS). Healthy People 2000 (DHHS Publication No. (PHS) 91-50212). Washington, DC: Superintendent of Documents, Government Printing Office; 1991.
    Van MondfransA.Guidelines for Reporting Evaluation Findings. Unpublished manuscript. Provo, UT: Brigham Young University; 1985.
    VeneyJE, KaluznyAD.Evaluation and Decision Making for Health Services. Chicago: Health Administration Press; 1998.
    WagenaarAC.Effects of the raised legal drinking age on motor vehicle accidents in Michigan. HSRI Research Review1981; 11(4):1-8.
    WagenaarAC.Preventing highway crashes by raising the legal minimum age for drinking: The Michigan experience 6 years later. Journal of Safety Research1986; 17:101-109. http://dx.doi.org/10.1016/0022-4375%2886%2990024-1
    WalterHJ, VaughanRD, ArmstrongB, KrakoffRY, TiezziL, McCarthyJF.Characteristics of users and nonusers of health clinics in inner-city junior high schools. Journal of Adolescent Health1996; 18(5):344-348. http://dx.doi.org/10.1016/1054-139X%2895%2900171-N
    WareJE Jr., BaylissMS, RogersWH, KosinskiM, TarlovAR.Differences in 4-year health outcomes for elderly and poor, chronically ill patients treated in HMO and fee-for-service systems. JAMA1996; 276(13):1039-1047. http://dx.doi.org/10.1001/jama.1996.03540130037027
    WarnerKE.Issues in cost effectiveness in health care. Journal of Public Health Dentistry1989;49(5):272-278. http://dx.doi.org/10.1111/j.1752-7325.1989.tb02085.x
    WarnerKE, LuceBR.Cost-Benefit and Cost-Effectiveness Analysis in Health Care: Principles, Practice and Potential. Ann Arbor, MI: Health Administration Press; 1982.
    WebbEJ, CampbellDT, SchwartzRD, SechrestL.Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago: Rand McNally College Publishing Company; 1966.
    WeinsteinMC, StasonWB.Foundations of cost-effectiveness analysis for health and medical practices. New England Journal of Medicine1977; 296(13):716-721. http://dx.doi.org/10.1056/NEJM197703312961304
    WeissCH.Evaluation Research. Englewood Cliffs, NJ: Prentice-Hall; 1972.
    WeissCH.Ideology, interests, and information: The basis of policy positions. In: CallahanD, JenningsB, eds. Ethics, the Social Sciences, and Policy Analysis. New York: Plenum Press; 1983:213-245. http://dx.doi.org/10.1007/978-1-4684-7015-4_9
    WeissCH.Nothing as practical as a good theory: Exploring the theory-based evaluation for comprehensive community initiative for children and families. In: ConnelJP, KubischAC, SchorrLB, WeissCH, eds. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: The Aspen Institute; 1995:65-92.
    WeissCH.Theory-based evaluation: Past, present, and future. New Directions for Evaluation1997; 76(Winter):41-55. http://dx.doi.org/10.1002/ev.1086
    WeissCH.Evaluation: Methods for Studying Programs and Policies. Upper Saddle River, NJ: Prentice Hall; 1998a.
    WeissCH.Have we learned anything new about the use of evaluation?American Journal of Evaluation1998b; 19(1):21-33.
    WilderCS.Physician Visits, Volume, and Interval Since Last Visit, U.S., 1969 (Series 10, No. 75, DHEW Publication No. (HSM) 72-1064). Rockville, MD: National Center for Health Statistics; July 1972.
    WilsonDB.The Role of Method in Treatment Effect Estimates: Evidence From Psychological, Behavioral, and Educational Meta-Analyses [dissertation]. Claremont, CA: Claremont Graduate School; 1995.
    WindsorRA, CutterG, MorrisJ, ReeseY, ManzellaB, BartlettEE, SamuelsonC, SpanosD.Effectiveness of self-help smoking cessation interventions for pregnant women in public health maternity clinics: A randomized trial. American Journal of Public Health1985; 75(12):1389-1392. http://dx.doi.org/10.2105/AJPH.75.12.1389
    WindsorRA, WarnerKE, CutterGR.A cost-effectiveness analysis of self-help smoking cessation methods for pregnant women. Public Health Reports1988; 103(1):83-88.
    WorthenBR, SandersJR, FitzpatrickJL.Program Evaluation: Alternative Approaches and Practical Guidelines. New York: Longman; 1997.
    WrightJC, WeinsteinMC.Gains in life expectancy from medical interventions—standardizing data or outcomes. New England Journal of Medicine1998; 339(6):380-386. http://dx.doi.org/10.1056/NEJM199808063390606

    Author Index

    About the Author

    David Grembowski, Ph.D., is a professor in the Department of Health Services, School of Public Health and Community Medicine, and the Department of Dental Public Health Sciences, School of Dentistry, at the University of Washington. He has taught health program evaluation to graduate students for more than 10 years. His evaluation interests are the performance of health programs and health care systems. He is conducting an evaluation of managed care and its influence on physician referrals and outcomes of care. He is also directing an evaluation examining associations between managed care, the patient-physician relationship, and health outcomes. His prevention studies have addressed the cost-effectiveness of preventive services for older adults, the effects of fluoridation on oral health and dental expenditures, and the effects of financial incentives on dentist adoption of preventive technologies. Other work includes the evaluation of a partnership program to boost access to dental care among Medicaid-covered preschool children and studies estimating the impact of insurance on the demand for dental care.


    • Loading...
Back to Top