Health Intervention Research: Understanding Research Design and Methods
Publication Year: 2015
At a time when evidence-based practice is the standard bearer for understanding health behaviour, problems and interventions, ensuring that researchers know the appropriate designs and methods for their research is more paramount than ever. Health Intervention Research will equip those doing research in these communities with the knowledge and tools they need to inform their methodological decisions when planning and conducting studies. This book describes both commonly used (e.g., randomized clinical trials) and advanced (e.g. preference trials, pragmatic trials) designs and methods for health intervention research. It outlines the theoretical reasoning underlying these different approaches, and synthesizes the evidence which supports or disputes different designs and methods. To achieve its aims, the book is divided into three main sections. The first section points to the ...
- Front Matter
- Back Matter
- Subject Index
- Chapter 1: AN OVERVIEW OF HEALTH INTERVENTION RESEARCH
- Chapter 2: VALIDITY IN INTERVENTION RESEARCH
- Chapter 3: PHASES OF INTERVENTION EVALUATION RESEARCH
- Chapter 4: EXPERIMENTAL DESIGNS OR RANDOMIZED CONTROLLED TRIALS: CHARACTERISTIC FEATURES
- Chapter 5: EXPERIMENTAL DESIGNS OR RANDOMIZED CONTROLLED TRIALS: LIMITATIONS
- Chapter 6: ADVANCES IN INTERVENTION EVALUATION DESIGNS: EXTENSIONS OF EXPERIMENTAL DESIGNS
- Chapter 7: ADVANCES IN INTERVENTION EVALUATION DESIGNS: PRAGMATIC AND PREFERENCE TRIALS
- Chapter 8: SELECTION OF PARTICIPANTS
- Chapter 9: RETENTION OF PARTICIPANTS
- Chapter 10: ASSIGNMENT OF PARTICIPANTS TO STUDY GROUPS
- Chapter 11: IMPLEMENTATION OF THE INTERVENTION
- Chapter 12: PRINCIPLES OF OUTCOME MEASUREMENT AND ANALYSIS
SAGE Publications Ltd
1 Oliver’s Yard
55 City Road
London EC1Y 1SP
SAGE Publications Inc.
2455 Teller Road
Thousand Oaks, California 91320
SAGE Publications India Pvt Ltd
B 1/I 1 Mohan Cooperative Industrial Area
New Delhi 110 044
SAGE Publications Asia-Pacific Pte Ltd
3 Church Street
#10-04 Samsung Hub
© Souraya Sidani 2015
First published 2014
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act, 1988, this publication may be reproduced, stored or transmitted in any form, or by any means, only with the prior permission in writing of the publishers, or in the case of reprographic reproduction, in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers.
Library of Congress Control Number: 2014936874
British Library Cataloguing in Publication data
A catalogue record for this book is available from the British Library
ISBN 978-1-4462-5617-6 (pbk)
Editors: Michael Carmichael and Jai Seaman
Editorial assistants: Keri Dickens and Lily Mehrbod
Production editor: Katie Forsythe
Copyeditor: Jane Fricker
Proofreader: Sarah Cooke
Indexer: Silvia Benvenuto
Marketing manager: Camille Richmond
Cover design: Wendy Scott
Typeset by: C&M Digitals (P) Ltd, Chennai, India
Printed and bound by CPI Group (UK) Ltd, Croydon, CR0 4YY
About the Author
In an era characterized by an emphasis on evidence-informed practice, research plays a central role in helping us understand human conditions, behaviors, and emerging problems; and identifying interventions that effectively address the problems and promote well-being. Empirical evidence on the effects of treatments guides decision making regarding the selection, implementation, and evaluation of interventions to improve various domains of life at the local, national, and international levels. However, to be useful, research studies must be well planned and executed. Careful planning involves the selection of research designs and methods that 1) are appropriate to capture the problem under investigation, 2) are consistent with the study’s purpose and aims, 3) ensure validity of the inferences and minimize potential biases, and 4) are feasible within the context in which they are applied. Careful execution consists of developing a detailed study protocol, adhering to it when carrying out research activities (i.e., recruitment, data collection, implementation of the intervention, and data analysis), and closely monitoring the study in order to ensure quality of performance, and to identify and remedy any challenges or deviations.
A range of research designs and methods is available to study the effects of interventions. Some have been commonly considered as the most useful in generating credible evidence, and others have been advanced as plausible alternatives in response to recent critique of commonly used designs and methods. The critique was prompted by the realization that most, if not all, designs and methods are based on assumptions and recommendations which have been taken for granted and not been systematically and critically evaluated. These are derived from logic that may no longer be tenable in light of accumulating experience and emerging empirical evidence. Specifically, the adoption of the experimental or randomized controlled trial as the ‘gold standard’ for determining the effects of interventions was based on theoretical reasons and intuitive attractiveness rather than a compelling evidence base of data. Empirical evidence derived from meta-analyses shows that results of randomized trials and well-designed non-randomized studies evaluating the same interventions are comparable in determining the success of the intervention. These findings raise questions about the necessity and utility of randomization in reducing selection bias and enhancing validity of causal inferences. Randomization increases the likelihood that study groups are similar at baseline, but it does not guarantee it. Further, it introduces biases related to who takes part [Page xiii]in studies and the influence of their perception of the intervention on treatment adherence and outcomes. Practical, pragmatic trials and partially randomized clinical or preference trials have been proposed to enhance representativeness of the sample, account for participants’ treatment preferences, and reduce attrition. Similarly, evidence is emerging that questions the utility of other methods such as the use of placebo.
This book represents a compendium of research designs and methods, encompassing commonly used ones and recent advances that can be used in the evaluation of interventions. The book content describes the theoretical, empirical, and practical knowledge required in choosing among designs and methods for intervention evaluation. Theoretical knowledge covers the logic underlying different designs and methods; it provides the rationale or the ‘why’ for methodological decisions. Empirical knowledge looks at the results of studies that investigate the effectiveness, utility, or efficiency of different methods; it informs the ‘what’, ‘when’, and ‘where’ of methodological decisions. Practical knowledge involves descriptions of the procedure for implementing different research methods; it points to the ‘how’ for carrying out selected methods. The aim is to inform researchers of the nature and effectiveness of various designs and methods. This information is essential to 1) make researchers aware of different designs and methods, each having its strengths and limitations at the theoretical and empirical levels, 2) assist researchers in making appropriate decisions related to the selection of most suitable methods that best fit the context of particular studies, 3) help researchers recognize that methodological decisions should be based on evidence rather than mere traditional recommendations which may not be well supported logically and empirically, and ultimately 4) move the research enterprise out of the ‘inertia’ of using commonly recommended designs and methods that produce empirical evidence of limited utility to decision making and policy development, and into the world of generating, testing, and using alternative relevant designs and methods.
The author gratefully acknowledges the encouragement, support, and informal feedback from colleagues and students, and the instrumental support of staff at the Health Intervention Research Center (Sarah I. and Laura C.).
References[Page 212]1999) The role of randomization in clinical studies: Myths and beliefs. Journal of Clinical Epidemiology, 52: 487–497.and (2004) Psychotherapy process: The missing link: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 130: 664–668.and (2005) Therapy preference and treatment outcome in clients with mild to moderate alcohol dependence. Drug and Alcohol Review, 24: 209–216., and (2004) Are experiments the only option? A look at dropout prevention programs. The Review of Economics and Statistics, 86: 180–194.and (2005) Methodological issues in the affects of attrition: Simple solutions for social scientists. Field Methods, 17: 53–69.and (1998) Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review, 22 (2): 207–244., , , and (1991) The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50 (2): 179–211.(2008) Development and feasibility of a parental support intervention for adolescent smokers. Substance Use and Misuse, 43: 497–511., , , , , , , and (2004) A failed RCT to determine if antibiotics prevent mastitis: Cracked nipples colonized with Staphylococcus aureus: A randomized treatment trial [ISRCTN65289389]. BMC Pregnancy and Childbirth, 4: 19., and (1987) Length of time on a waiting list and attrition after intake. Journal of Counseling Psychology, 34: 93–95., and (2005) Content validation of self-report measurement instruments: An illustration from the development of a brain tumor module of the M.D. Anderson Symptom Inventory. Oncology Nursing Forum, 3: 669–676., , and ([Page 213] , , and (1995) Comparing prospective and retrospective measures of treatment outcomes. Medical Care, 33 (Suppl): AS–67–76.2012) Placebo and other psychological interactions in headache treatment. Journal of Headache and Pain, 13: 191–198., and (2000) Determinants of patients’ treatment preferences in a clinical trial. Community Dental Oral Epidemiology, 28: 119–125., , and (1975) Dropping out of treatment: A critical review. Psychological Bulletin, 82: 738–783.and (2005) Factors that influence women’s decisions about whether to participate in research: An exploratory study. Birth, 32: 60–66., and (2007) Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75 (6): 842–852., and (2002) Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA: The Journal of the American Medical Association, 287: 2973ȁ2982., , , , , and (2007) Using strong evaluation designs in developing countries: Experience and challenges. Journal of MultiDisciplinary Evaluation, 4 (8): 58–73.and (2005) Intervention research: Selecting control interventions for clinical outcome studies. Western Journal of Nursing Research, 27(3): 346–363., and (2002) A nurses’ primer on recruiting participants for clinical trials. Oncology Nursing Forum, 29: 1091–1096.(1986) The dubious effects of type V errors in hypothesis testing on health education practice and theory. Health Education Research, 1 (4): 299–305.and (2000) Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71 (6): 973–979., , and (1990) Coaching for the Scholastic Aptitude Test: Further synthesis and appraisal. Review of Educational Research, 60: 373–417.(2008) Tricks of the Trade: How to Think about Your Research While You’re Doing it. Chicago: University of Chicago Press.(2003) Explanations for improvement in both experimental and control groups. Western Journal of Nursing Research, 25: 746–755., and (2000) Assessing effectiveness of treatment of depression in primary care: Partially randomized preference trial. British Journal of Psychiatry, 177: 312–318., , , , , , , , , and ([Page 214] , , , , , et al. (2004) Enhancing treatment fidelity in health behaviour change studies: Best practices and recommendations from the NIH behaviour change consortium. Health Psychology, 23 (5): 443–451.2000) A comparison of observational studies and randomized, controlled trials. The New England Journal of Medicine, 342 (25): 1878–1886.and (2008) The science of improvement. The Journal of the American Medical Association,299 (10): 1182–1184.(2005) Using experiments to assess non-experimental comparison-group methods for measuring program effects. In (ed.), Learning More from Social Experiments. New York: Russell Sage Foundation, pp. 173–235., and (2011) Acceptability of behavioral treatments for insomnia. Journal of the American Board of Family Medicine, 24: 272–280., and (2005) Understanding placebo, nocebo, and iatrogenic treatment effects. Journal of Clinical Psychology, 61 (7): 871–880.and (2010) Bias in experimental nursing research: Strategies to improve the quality and explanatory power of nursing science. International Journal of Nursing Studies, 47: 123–128.and (1993) Between-group therapy outcome research: Design and methodology. In , and (eds), Behavioral Treatments for Drug Abuse and Dependence. National Institute on Drug Abuse (NIDA) Research Monograph Series 137. Washington, DC: US Department of Health and Human Services, pp. 249–289.(2011) Relationship between treatment preference and weight loss in the context of a randomized controlled trial. Obesity, 20(6): 1218–1222., , , , , , , , and (2005) A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behaviour research. Journal of Consulting and Clinical Psychology, 73 (5): 852–860., , , , , et al. (1997) To randomize or not to randomize: Methodological pitfalls of the RCT design in psychosocial intervention studies. European Journal of Cancer Care, 6: 222–230.(2005) A review of blinding in randomized controlled trials found results inconsistent and questionable. Journal of Clinical Epidemiology, 58: 1220–1226>., and (1998) Relative contribution of specific and nonspecific treatment effects: Meta-analysis of placebo-controlled behavior therapy research. Psychological Bulletin, 103 (3): 313–323.and ([Page 215] (2005) Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health, 27 (3): 281–291.2005) ‘You decide doctor’: What do patient preference arms in clinical trials really mean? Journal of Epidemiology and Community Health, 59: 914–915.and (1992) A randomized controlled trial of dietary fat reduction: The retention of subjects and characteristics of drop outs. Journal of Clinical Epidemiology, 45 (1): 31–38., and (1993) Designing medical and educational intervention studies. Diabetes Care, 16: 509–518.(2004) Assessing the strength and integrity of an intervention. Oncology Nursing Forum, 31 (4): 833–837., , and (2010) Implementation fidelity in community-based interventions. Research in Nursing and Health, 33(2): 164–173., , , , and (1998) Choosing between randomised and non-randomised studies: A systematic review. Health Technology Assessment, 2 (13)., , , , and (2002) Nursing intervention studies: A descriptive analysis of issues important to clinicians. Research in Nursing and Health, 25: 317–327.(2007) Assessing treatment fidelity in pilot studies to assist in designing clinical trials: An illustration from a nurse practitioner community-based intervention for pain. Advances in Nursing Science, 30 (1): E72–E84.and (2009) Intervention research in highly unstable environments. Research in Nursing and Health, 32: 110–121., , , , , and (2003) An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA. Bonn: IZA.and (2007) Common and specific factors in cardiac rehabilitation: Independent and interactive effects of the working alliance and self-efficacy. Health Psychology, 26 (6): 684–692.and (2003) Overcoming subject recruitment challenges: Strategies for successful collaboration with novice research agencies. Applied Nursing Research, 16 (1): 46–52., , and (2012) Characteristics of clinical trials registered in ClinicalTrials.gov, 2007–2010. Journal of the American Medical Association, 307 (17): 1838–1847., , , , and (1995) Hawthorne effects: implications for prehospital research. Annals of Emergency Medicine, 26 (5): 590–594., and ([Page 216] , , , , , and (2000) Framework for design and evaluation of complex interventions to improved health. British Medical Journal, 321: 694–696.2007) Designing and evaluating complex interventions to improve health care. British Medical Journal, 334: 455–459., , , , , et al. (2005) Adapting the randomized consent (Zelen) design for trials of behavioural interventions for chronic disease: Feasibility study. Journal of Health Sciences Research and Policy, 10 (4): 220–225., , , and (1994) Power analysis of cutoff-based randomized clinical trials. Evaluation Review, 18 (2): 141–152., and (2007) A conceptual framework for implementation fidelity. Implementation Science, 2: 40–48., , , , and (2007) Are RCT’s the gold standard? BioSocieties, 2: 11–20.(2010) The limitations of randomized controlled trials in predicting effectiveness. Journal of Evaluation in Clinical Practice, 16: 260–266.and (1974) Two dimensions of tests: Psychometric and edumetric. American Psychologist, 29: 512–518.(1983) Bias in treatment assignment in controlled clinical trials. The New England Journal of Medicine, 309: 1358–1361., , and (1984) Applying results of randomised trials to clinical practice: Impact of losses before randomisation. British Medical Journal (Clinical research ed.), 289 (6454): 1281.and (2007) Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. American Journal of Evaluation, 28 (3): 239–255.(2009) Breast cancer survivorship program: Testing for cross-cultural relevance. Cancer Nursing, 32 (3): 236–245., , and (2005) Strategies for retaining study participants in behavioural intervention trials: Retention experiences of the NIH behaviour change consortium. The Society of Behavioural Medicine, 29(Suppl): 55–65., , Goldman , , , and (1988) Statistical Power Analysis for the Social Sciences ((2nded.). Hillsdale, NJ: Lawrence Erlbaum.1988) The effect of study design on gain in evaluation of new treatments in medicine and surgery. Drug Information Journal, 22 (3): 343–352., and (2004) Commentary: Beyond randomised versus observational studies. The Lancet, 363 (9422): 1660–1661.and (2000) Randomised, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342 (25): 1887–1892., and ([Page 217] , and (2004) Improving enrollment in cancer clinical trials. Oncology Nursing Forum, 31 (3): 610–614.1993) A quasi-sampling theory of the generalization of causal relationships. New Directions for Program Evaluation, 57: 39–81.(2006) Describing what is special about the role of experiments in contemporary educational research: Putting the ‘gold standard’ rhetoric into perspective. Journal of Multi-Disciplinary Evaluation, 6: 1–7.(2007) Randomized experiments in education: Assessing the objections to doing them. Economics of Innovation and New Technology, 16 (5): 331–355.(1979) Quasi-experimentation: Design and Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.and (2010) Contemporary thinking about causation in evaluation: A dialogue with Tom Cook and Michael Scriven. American Journal of Evaluation, 31 (1): 105–117., , and (1991) Accrual to cancer clinical trials: Directions from the research literature. Social Science and Medicine, 33 (5): 569–577.(2003) Challenges of recruitment and retention in multisite clinical research. Cancer Nursing, 26 (5): 376–386., , , , , and (2003) The acceptability of treatment of depression among African-American, Hispanic and White primary care patients. Medical Care, 41(4): 479–489., , , , , et al. (2006) Predictors of pretreatment attrition from smoking cessation among pre- and postmenopausal, weight-concerned women. Eating Behaviours, 7: 243–251., , , and (2002) Partial randomization design in a support group intervention study. Western Journal of Nursing Research, 24: 406–421.(2008) Developing and evaluating complex interventions: The New Medical Research Council guidance. British Medical Journal, 337: a1655., , , , and (1991) Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. Journal of Counsulting and Clinical Psychology, 59: 20–26.and (1991) Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1 (2): 81–91., , , , , et al. (2000) Determinants of attrition from cessation treatment in smokers with a history of major depressive disorder. Psychology of Addictive Behaviours, 14 (2): 134–142., and (2009) The delayed-start study design. New England Journal of Medicine, 361: 1304–1306.([Page 218] , , , , and (2009) Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4: 50–64.2003) Ethnicity and recruitment rates in clinical research studies. Applied Nursing Research, 16 (3): 189–195.(2006) The RCTs only doctrine: Brakes on the acquisition of knowledge? Journal of MultiDisciplinary Evaluation, 6: ii–v.(2005) Patients’ perceptions of pharmacological and cognitive-behavioral treatments for anxiety disorders. Behavior Therapy, 36: 139–145.and (2007) The effectiveness of long term psychotherapy: Methodological research issues. Psychotherapy Research, 17 (1): 59–65., , and (1998) Empirical comparison of the results of randomized controlled trials and case-control studies in evaluating the effectiveness of screening mammography. Journal of Clinical Epidemiology, 51 (2): 81–91., and (2003) Scale Development: Theory and Applications ((2nded.). Thousand Oaks, CA: Sage.1973) Outcome effects of receiving a preferred, randomly assigned, or nonpreferred therapy. Journal of Consulting and Clinical Psychology, 41: 104–107.and (1986) Assessing the responsiveness of functional scales to clinical change: An analogy to diagnostic test performance. Journal of Chronic Disease, 39 (11): 897–906.and (2001) Recruitment and retention of community-dwelling, aging women in nursing studies. Nursing Research, 50 (6): 369–373.(2008) Therapist effects on outcome and alliance in inpatient psychotherapy. Journal of Clinical Psychology, 64 (3): 344–354., , , and (2004) Determining causality in program evaluation and applied research: Should experimental evidence be the gold standard? Journal of MultiDisciplinary Evaluation, 3: 60–77.and (2000) Design and Analysis of Cluster Randomization Trials in Health Research. London: Arnold Publishers, pp. 81–82.and (2004) Pitfalls of and controversies in cluster randomization trials. American Journal of Public Health, 94 (3): 416–422.and (2007) The merits of breaking the matches: A cautionary tale. Statistics in Medicine, 26 (9): 2036–2051., and (2005) A new approach to eliciting patients’ preferences for palliative day care: The choice experiment method. Journal of Pain and Symptom Management, 29: 435–445., , and ([Page 219] , , , and (2001) Promoting intervention fidelity: Conceptual issues, methods, and preliminary results from the early alliance prevention trial. American Journal of Preventative Medicine, 20 (1S): 38–47.2006) Assessment of bias in methodology for randomized controlled trials published on implant dentistry. Journal of Prosthodontics, 15 (4): 257–263., , and (2006) Reporting attrition in randomized controlled trials. British Medical Journal, 332 (22): 969–971., and (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41: 327–350.and (2001) Can quality improvement programs for depression in primary care address patient preferences for treatment? Medical Care, 39 (9): 934–944., , , and (2001) Effectiveness of chemotherapy for advanced lung cancer in the elderly: Instrumental variable and property analysis. Journal of Clinical Oncology, 19 (4): 1064–1070., , , , and (2005) Changing the behavior of healthcare professionals: The use of theory in promoting the uptake of research findings. Journal of Clinical Epidemiology, 58: 107–112., , , and (2000) Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly, 15 (2): 123–149.and (2007) Dose-response effects of cognitive-behavioral insomnia therapy: A randomized clinical trial. Sleep, 30 (2): 203–212., , , and (1995) Placebo effect and randomized clinical trials. Theoretical Medicine, 16: 171–182.and (2004) Lessons for cluster randomized trials in the twenty-first century: A systematic review of trials in primary care. Clinical Trials, 1: 80–90., , , and (2005) Why modelling a complex intervention is an important precursor to trial design: Lessons from studying an intervention to reduce falls-related injuries in older people. Journal of Health Services Research and Policy, 10 (3): 133–142., , , , and (2006) Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research, 16 (2): 144–160., , and (2000) Attitudes towards and participation in randomized clinical trials in oncology: A review of the literature. Annals of Oncology, 11: 939–945.([Page 220] , , , , and (2005) Effect of a clinical trial alert system on physician participation in trial recruitment. JAMA Internal Medicine, 165: 2272–2277.2009) Beyond information: Exploring patients’ preferences. Journal of the American Medical Association, 302 (2): 195–197.and (2000) Responsiveness in quality-of-life assessment: Nomenclature, determinants, and clinical applications. Medical Care, 38 (9, Suppl II): 91–94.(2005) A comprehensive method for the translation and cross-cultural validation of health status questionnaires. Evaluation and the Health Professions, 28 (2): 212–232., and (2005) Does the non-randomized controlled study have a place in the systematic review? A pilot study. Criminal Behaviour and Mental Health, 15 (2): 111–120.and (2010) Biological, clinical, and ethical advances of placebo effects. The Lancet, 375: 686–695., , and (1999) Capturing the patients’ view of change as a clinical outcome measure. Journal of the American Medical Association, 282: 1157–1162., , et al. (1993) Transition questions to assess outcomes in rheumatoid arthritis. British Journal of Rheumatology, 32: 807–811., , , and (2005) Implementation Research: A Synthesis of the Literature. Tampa: University of South Florida., , , and (1986) Analysis of data from multiclinic trials. Controlled Clinical Trials, 7 (4): 267–275.(2010) Effects of participant preferences in unblinded randomized controlled trials. Journal of Empirical Research on Human Research Ethics, 5: 81–93.and (2009) Clinical intervention research in nursing. International Journal of Nursing Studies, 46: 557–568.(2005) Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behaviour Therapy, 36 (1): 3–13., and (2007) Improving informed decision-making for patients with knee pain. The Journal of Rheumatology, 34 (9): 1894–1898., , and (2010) Lies, damned lies, and medical science. The Atlantic, November.(2007) The physician–patient working alliance. Patient Education and Counseling, 66: 29–36., , , , , and (1998) Random versus volunteer selection for a community-based study. The Journals of Gerontology Series A: Biological Sciences and Medical Sciences, 53 (1): M39–M46., , and ([Page 221] , , , , and (2011) Major ingredients of fidelity: A review and scientific guide to improving quality of intervention research implementation. Clinical Psychology Review, 31: 79–88.1978) Statistics and ethics in surgery and anesthesia. Science, 78: 684–689., and (2007) Ethnicity and preferences for depression treatment. General Hospital Psychiatry, 29: 182–191., , , and (2005) Practical clinical trials for translating research to practice: Design and measurement recommendations. Medical Care, 43 (6): 551–559., , , and (2007) When are randomized trials unnecessary? Picking signal from noise. British Medical Journal, 334: 349–351., , and (2003) Nonexperimental versus experimental estimates of earning impacts. The Annals of the American Academy, 589: 63–93., and (2004) On the need for a new psychotherapy research paradigm: Comment on Westen, Novotny, and Thompson-Brenner (2004). Psychological Bulletin, 130 (4): 669–673.and (1993) Evaluating interventions with differential attrition: The importance of nonresponse mechanisms and use of follow up data. Journal of Applied Psychology, 78 (1): 119–128.and (2006) Randomized controlled trials do not reflect reality: Real world analyses are critical for treatment guidelines. The Journal of Thoracic and Cardio Surgery, 132 (1): 5–7., , , and (2000) The role of theory in evidence-based health promotion practice. Health Education Research, 15: 125–129.(2000) Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Family Practice, 17 (1): S11–S16., , and (2006) Preferences for mechanical ventilation among survivors of prolonged mechanical ventilation and tracheostomy. American Journal of Critical Care, 15: 65–77., , , , , and (2006) Depression treatment preferences in older primary care patients. The Gerontologist, 4: 14–22., , , , , , , and (2002) Introduction: The philosophy of evidence-based medicine. In and (eds), Users’ Guides to the Medical Literature: A Manual for Evidence-based Clinical Practice. Chicago: AMA Press, pp. 3–12., , , , , , , , , and ([Page 222] , , , , , and (2003) Instruments to measure acceptability of information and acquisition of knowledge in patients with heart failure. The European Journal of Heart Failure, 5: 783–791.2004) A healthy dose of criticism for randomized trials: Comment on Western Novotny, and Thompson-Brenner. Psychological Bulletin, 130 (4): 674–676.(2001) Patterns of patient enrollment in randomized controlled trials. Journal of Clinical Epidemiology, 54: 877–883.and (2005) A systematic review of trial methodology, using the placebo groups of randomized controlled trials in pediatric postoperative pain. International Association for the Study of Pain, 116: 146–158.and (2005) A causal modelling approach to the development of theory-based behaviour change for programmes for trial evaluation. Health Education Research, 20 (6): 676–687., , , , , and (2008) Statistical significance versus clinical significance. Seminars in Cardiothoracic and Vascular Anesthesia, 12 (1): 5–6.and (2003) The national longitudinal study of adolescent health: Research design. Carolina Population Center, University of North Carolina at Chapel Hill. Retrieved from: www.cpc.unc.edu/projects/addhealth/faqs/aboutdata/weight1.pdf (accessed 3 September 2008)., , , , and (1998) Attrition model. American Journal of Evaluation, 19 (3): 293–305.(2001) Recruitment of frail older people to research: Lessons learnt through experience. Journal of Advanced Nursing, 36 (5): 643–651.and (2009) Treatment definition in complex rehabilitation interventions. Neuropsychological Rehabilitation, 19 (6): 824–840.(2010) Systematic evaluation of information fidelity of complex intervention in health and social care. Implementation Science, 5: 67–75.(2004) Complex interventions: How ‘out of control’ can a randomized controlled trial be? British Medical Journal, 328: 1561–1563., and (2002) Predictors of willingness to consider medication and psychosocial treatment for panic disorder in primary care patients. General Hospital Psychiatry, 24: 316–321., , , , and (2001) Conducting health research with vulnerable women: Issues and strategies. Canadian Journal of Nursing Research, 33 (3): 81–86.(1996) Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments? Psychological Methods, 1 (2): 154–169.and (2003) Predicting treatment choice for patients with pelvic organ prolapse. The American College of Obstetrics and Gynecology, 101: 1279., , , , and ([Page 223] (2008) Considerations in determining sample size for pilot studies. Research in Nursing and Health, 31: 180–191.1998) The randomized controlled trial: A critique. Nurse Researcher, 6 (1): 19–32.(1998) Patient choice in diabetes education curriculum. Diabetes Care, 21: 896–901., , , , and (1998) Pretreatment attrition in a comparative treatment outcome study on panic disorder. American Journal of Psychiatry, 155: 43–47., , , , , et al. (1999) What is meant by intention to treat analysis? Survey of published randomised controlled trials. British Medical Journal, 319 (7211): 670–674.and (2007) Comparative effectiveness of health interventions: Strategies to change policy and practice. Report from ECRI Institute’s 15th Annual Conference, pp. 1–8.(2002) Using the Zelen design in randomized controlled trials: Debates and controversies. Journal of Advanced Nursing, 38 (2): 200–207.(2005) Another look at observational studies in rehabilitation research: Going beyond the holy grail of the randomized controlled trial. Archives of Physical Medicine and Rehabilitation, 86 (2): S8–S15., , , and (1987) Complexity and contradiction in clinical trial research. The American Journal of Medicine, 82: 498–510.(2006) Patient preference and randomized controlled trials in mental health research. British Journal of Psychiatry, 188: 303–304.and (Beurskens, A.J.H.M., , , et al. (2004) An alternative trial design to overcome validity and recruitment problems in primary care research. Family Practice, 21 (2): 213–218., ,2001) Therapists, therapist variables, and cognitive-behavioral therapy outcome in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69: 747–755., , , , and (2007) Design and analysis of stepped wedge cluster randomized trials. Contemporary Clinical Trials, 28: 182–191.and (2004) Fool’s gold: Examining the use of gold standards in the production of research evidence. British Journal of Occupational Therapy, 67 (2): 89–94.(2013) Strategies to recruit minority persons: A systematic review. Journal of Immigrant and Minority Health, 5 (11): 1784–1790.and (1997) Quantification of the placebo response in ulcerative colitis. Gastroenterology, 112: 1854–1858., , , and (2008) Regression discontinuity designs: A guide to practice. Journal of Econometrics, 142 (2): 615–635.and ([Page 224]Institute of Medicine (2001) Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press.2005) Why most published research findings are false. PLoS Medicine, 2 (8): 696–701.(2001) Comparison of evidence of treatment effects in randomized and nonrandomized studies. Journal of the American Medical Association, 286 (7): 821–830., , , , , et al. (2004) Cognitive behavior therapy and pharmacotherapy for insomnia: A randomized controlled trial and direct comparison. Archives of Internal Medicine, 164: 1888–1986., , and (2003) The role of choice in health education intervention trials: A review and case study. Social Science and Medicine, 56: 1581–1594., , , , , and (2000) Reasons for accepting or declining to participate in randomized clinical trials for cancer therapy. British Journal of Cancer, 82 (11): 1783–1788.and (2004) Interpretation of observational studies. Heart, 90: 956–960., , and (2002) Statistical power in randomized intervention studies with noncompliance. Psychological Methods, 7 (2): 178–193.(2008) Cluster randomized trials with treatment noncompliance. Psychological Methods, 13 (1): 1–18., , , and (2003) Cognitive psychology and self-reports: Models and methods. Quality of Life Research, 12: 219–227.(BRUM Steering Committee (2005) Recruitment of ethnic minority patients to a cardiac rehabilitation trial: The Birmingham Rehabilitation Uptake Maximisation (BRUM) study. BMC Medical Research Methodology, 5 :18., , , , , and the2003) The alliance as mediator of expectancy effects in short-term individual therapy. Journal of Clinical and Consulting Psychology, 71 (4): 672–679., , and (1999) The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282: 1054–1060., , and (1996) Effect of exercise intensity on glucose and insulin metabolism in obese individuals and obese NIDDM patients. Diabetes Care, 19 (4): 341–349., , , , , , and (2001) The double-blind, randomized, placebo-controlled trial: Gold standard or golden calf? Journal of Clinical Epidemiology, 54: 541–549.(2010). Placebos without deception: A randomized controlled trial in irritable bowel syndrome. Public Library of Science One, 5 (12): 1–7., , , , , et al. ([Page 225] , , , and (2009) A new ‘mechanistic-practical’ framework for designing and interpreting randomized trials. Journal of Clinical Epidemiology, 62: 479–484.2003) Causal inference from randomized trials in social epidemiology. Social Science and Medicine, 57: 2397–2409., and (2010) Trial and error: How to avoid commonly encountered limitations of published clinical trials. Journal of the American College of Cardiology, 55: 415–427.and (1980) Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis, 13: 259–273.(2003) Research Design in Psychology. Boston: Allyn and Bacon.(2010) Fidelity of implementation: Development and testing of a measure. Implementation Science, 5: 99–109., , , and (2005) Retention of minority participants in clinical research studies. Western Journal of Nursing Research, 27 (3): 292–306., and (2009) Fidelity to theory in physical activity intervention research. Western Journal of Nursing Research, 31 (3): 289–311., , and (2007) How can one choose what is unknown? A study on the image of nurses by high school students. Acta Paulista de Enfermagem, 20 (2): 125–130.and (2012) Screening mammography in women less than age 50 years. Current Opinion in Obstetrics and Gynecology, 24 (1): 38–43.(1999) Stratified randomization for clinical trials. Journal of Clinical Epidemiology, 52 (1): 19–26., , , and (2006) Optimal matches of patient preferences for information, decision-making and interpersonal behavior: Evidence, models, and interventions. Patient Education and Counseling, 61: 319–341.and (2006) Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research, 16 (2): 161–172., and (2005) Impact of participant and physician intervention preferences on randomized trials: A systematic review. Journal of the American Medical Association, 293: 1089–1099., , , , , , and (2001) Validity and completeness of registration of surgically treated malignant gynaecological diseases in the Danish National Hospital Registry. Journal of Epidemiology and Biostatistics, 6 (5): 387–392., and (1999) Randomised controlled trial of exercise for low back pain: Clinical outcomes, costs, and preferences. British Medical Journal, 319 (7205): 279–283., , , , , and ([Page 226] , , , , , and (1990) Crack and cocaine abusers in outpatient psychotherapy. In and (eds), Psychotherapy and Counseling in the Treatment of Drug Abuse. National Institute on Drug Abuse (NIDA) Research Monograph Series 104. Washington, DC: US Department of Health and Human Services, pp. 24–35.2003) Palliative chemotherapy or best supportive care? A prospective study explaining patients’ treatment preference and choice. British Journal of Cancer, 89: 2219–2226., , , , , et al. (2004) Intervention mapping: A protocol for applying health psychology theory to prevention programmes. Journal of Health Psychology, 9 (1): 85–98., , , and (2009) Some thoughts on the hazards of sloppy science when designing and testing multicomponent interventions, including the Kitchen Skin phenomenon. Research in Nursing and Health, 32: 1–3.(1997) A meta-analysis of the alcoholism treatment outcome literature: Selection with randomized and quasi-experiments (Doctoral dissertation, University of Memphis).(1990) Random assignment in clinical trials: Issues in planning (infant health and development program). Journal of Clinical Epidemiology, 43 (11): 1157–1167.and (2009) The double jeopardy of clustered measurement and cluster randomization. British Medical Journal, 339: 503–505., , , , and (1998) The unpredictability paradox: Review of empirical comparisons of randomised and non-randomised clinical trials. British Medical Journal, 317: 1185–1190.and (2010) Treatment preferences, engagement, and clinical improvement in pharmacotherapy versus psychotherapy for depression. Behavior Research and Therapy, 48: 799–804., and (1986) Evaluating the econometric evaluations of training programs. American Economic Review, 76: 604–620.(2004) Reasons underpinning patients’ preferences for various angina treatments. Health Expectations, 6: 246–256., , , , et al. (2004) Design and analysis of pilot studies: Recommendations for good practice. Journal of Evaluation in Clinical Practice, 10 (2): 307–312., and (2004) The intervention selection bias: An under recognized confound in intervention research. Psychological Bulletin, 130 (2): 289–303., and ([Page 227] , , , and (2004) Those confounded vitamins: What can we learn from the differences between observational versus randomized trial evidence? The Lancet, 363: 1924–1927.2003) Barriers to conducting randomized controlled trials: Lessons learnt from the Horsechestnut and Venous Leg Ulcer Trial (HAVLUT). Contemporary Nurse, 15: 37–47.(2004) Evaluation of a physician-focused educational intervention on Medicaid children with asthma. The Annals of Pharmacotherapy, 38 (6): 961–966., and (2006) Attrition in a multi-component smoking cessation study for females. Tobacco Induced Disease, 3 (2): 59–71., , , , , and (2003) Evidence-based psychosocial treatment practices in schizophrenia: Lessons from the Patient Outcomes Research Team (PORT) Project. Journal of the American Academy of Psychoanalysis and Dynamic Psychiatry, 31 (1): 141–154.and (1993) Reducing attrition bias with an instrumental variable in a regression model: Results from a panel of rheumatoid arthritis patients. Statistics in Medicine, 12 (11): 1005–1018., and (2011) The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research, 45 (5): 626–629., and (2004) Does establishing fidelity of treatment help in understanding treatment efficacy? Comment on Bellg et al. (2004). Health Psychology, 23 (5): 452–456.and (2006) The effect of preferences for print versus telephone interventions on compliance and attrition in a randomized controlled physical activity trial. Psychology of Sport and Exercise, 7: 453–462., , and (2007) The relation of patients’ treatment preferences to outcome in a randomized clinical trial. Behavior Therapy, 38: 209–217., , , , and (2000) Longitudinal construct validity: Establishment of clinical meaning in patient evaluative instruments. Medical Care, 38 (9, Suppl II): 84–90.(1999) Impact of study quality on outcome in placebo-controlled trials of homeopathy. Journal of Clinical Epidemiology, 52 (7): 631–636., , , , and (2004) Randomized controlled trials of socially complex nursing interventions: Creating bias and unreliability? Journal of Advanced Nursing, 45 (1): 84–94.(2002) Maximizing retention in community-based clinical trials. Journal of Nursing Scholarship, 34: 47–53., and ([Page 228] and (2008) Theory-based health behavior change: Developing, testing, and applying theories for evidence-based interventions. Applied Psychology: An International Review, 57 (4): 698–716.1990) Design Sensitivity: Statistical Power for Experimental Research. Thousand Oaks, CA: Sage.(1993) The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48: 1181–1209.and (2002) Statistical Analysis with Missing Data (and (2nded.). Hoboken, NJ: John Wiley and Sons.2009) Uncertainty in the translation of preclinical experiments to clinical trials: Why do most Phase III clinical trials fail? Current Gene Therapy, 9 (5): 368–374.and (1997) The psychotherapist matters: Comparison of outcomes across twenty two therapists and seven patient samples. Clinical Psychology: Science and Practice, 4: 53–65., , , and (2009) Effects of evidence-based clinical practice guidelines on quality of care: A systematic review. Quality and Safety in Health Care, 18: 385–392., and (2007) Therapist effects in outpatient psychotherapy: A three-level growth curve approach. Journal of Counseling Psychology, 54 (1): 32–39., , , and (2004) Locating and retaining research participants for follow-up studies. Research in Nursing and Health, 27: 63–68., , , , and (2005) Impact of referral source and study applicants’ preference for randomly assigned service on research enrollment, service engagement, and evaluative outcomes. American Journal of Psychiatry, 162: 781–787., , , , and (2009) Current directions in mediation analysis. Current Directions in Psychological Science, 18 (1): 16–20.and (2000) A systematic review of comparisons of effect sizes derived from randomized and non-randomised studies. Health Technology Assessment, 4 (34): 1–154., , , , and (2009) Explaining pragmatic trials to pragmatic policymakers. Journal of Clinical Epidemiology, 62: 476–478.(2001) Differences in treatment preferences between persons who enroll and do not enroll in clinical trials. Annals of the Royal College of Physicians and Surgeons, 34 (5): 292–296., , , and ([Page 229] (2004) Are we missing anything? Pursuing research on attrition. Canadian Journal of Nursing Research, 36 (3): 82–98.1999) The placebo effect. Survey of Ophthalmology, 44 (1): 31–44.(2000) Who will adhere? Key issues in the study and prediction of adherence in randomized controlled trials. Controlled Clinical Trials, 21: 195S–199S., , and (2007) The Hawthorne Effect: A randomized controlled trial. BMC Medical Research Methodology, 7: 30., , , , and (1997) No consent means not treating the patient with respect. British Medical Journal, 314:1076.(1997) Optimal design in psychological research. Psychological Methods, 2 (1): 3–19.(2006) What influences recruitment to randomized controlled trials? A review of trials funded by two UK funding agencies. Trials, 7: 9., , , , , et al. (2009) Do implementation issues influence the effectiveness of medications? The case of nicotine replacement therapy and bupropion in UK Stop Smoking Services. BMC Public Health, 9: 28–36.and (1998) Journal of Consulting and Clinical Psychology, 66 (4): 697–701., , , , and (1999) Interpreting the evidence: Choosing between randomised and non-randomised studies. British Medical Journal, 19: 312–315., , , , and (2007) Missing Data: A Gentle Introduction. New York: Guilford Publications., , and (2011a) When is informed consent required in cluster randomized trials in health research? Trials, 12: 202., , , , , , , , , and (2011b) Who is the research subject in cluster randomized trials in health research? Trials, 12: 183., , , , , , , , , , and (2006) Addressing program fidelity using onsite observations and program provider descriptions of program delivery. Evaluation Review, 30 (6): 714–740., and (2002) Let’s face facts: Common factors are more potent than specific therapy ingredients. Clinical Psychology and Scientific Practice, 9: 21–25.and (1995) Validity of psychological assessment. American Psychologist, 50 (9): 741–749.([Page 230]Methodology Committee of the Patient-Centered Outcomes Research Institute (PCORI) (2012) Methodological standards and patient-centeredness in comparative effectiveness research. The PCORI perspective. Journal of the American Medical Association, 307 (15): 1636–1640.1977) Gastric freezing: An example of the evaluation of medical therapy by randomized clinical trials. In , and (eds), Costs, Risks, and Benefits of Surgery. New York: Oxford University Press, pp. 198–211.(2002) The importance of implementation fidelity. 442 UCB. Retrieved from: http://incrediblyears.comwww.incredibleyears.com/Library/items/fidelity-importance.pdf(2005) Patients’ preference and randomization: New paradigm of evidence-based clinical research. World Journal of Surgery, 29: 596–600., and (1989) How study design affects outcomes in comparisons of therapy. II: Surgical. Statistics in Medicine, 8: 455–466., and (2011) Exploring treatment preferences facilitated recruitment to randomized controlled trials. Journal of Clinical Epidemiology, 64: 1127–1136., , , , and (2004) An exploration of participants’ treatment preferences in a partial RCT. Canadian Journal of Nursing Research, 36 (3): 100–114.(2005) Randomised trial of a brief physiotherapy intervention compared with usual physiotherapy for neck pain patients: Outcomes and patients’ preferences. British Medical Journal, 330: 75–80., , , , , , and (2010) When cost meets efficiency: rethinking ways to sample a rare population (pp. 136–146). In and (eds), When Research goes off the Rails. New York: Guilford Press.and (2000) Factors differentiating dropouts from completers in a longitudinal, multicenter clinical trial. Nursing Research, 49 (2): 109–116., and (2003) Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24 (3): 315–340., , and (2004) Design and analysis of group randomized trials: A review of recent methodological developments. American Journal of Public Health, 94 (3): 423–432., and (1994) Variations in therapist effectiveness in the treatment of patients with substance use disorders: An empirical review. Addiction, 89: 679–688.and (2008) Beyond the randomized clinical trial: The role of effectiveness studies in evaluating cardiovascular therapies. Circulation, 118: 1294–1303., and ([Page 231] , , and (2006) Using cognitive interviews to develop surveys in diverse populations. Medical Care, 44: S21–S30.National Institute of Neurological Disorders and Stroke [NINDS] (1995) Tissue plasminogen activator for acute ischemic stroke. The New England Journal of Medicine, 333: 1581–1587.National Institutes of Health, National Cancer Institute (2005) Theory at a Glance: A Guide for Health Promotion Practice (2nded.). Retrieved from: http://www.cancer.gov/cancertopics/cancerlibrary/theory.pdf2011) Toward a new methodological paradigm for testing theories of health behavior and health behavior change. Patient Education and Counseling, 82 (3): 468–474.and (2007) Conceptual and design essentials for evaluating mechanisms of change: Alcoholism. Clinical and Experimental Research, 31 (S3): 4S–12S.(2008) Biostatistics: The Bare Essentials (and (3rded.). Shelton, CT: PMPH US.2005) Challenges in using randomized studies in systematic reviews of treatment interventions. Annals of Internal Medicine, 142 (12): 1112–1119.and (North West Uro-Oncology Group (2002) A preliminary report on patient-preference study to compare treatment options in early prostate cancer. British Journal of Urology International, 90: 253–256.1994) Psychometric Theory (and (3rded.). New York: McGraw-Hill.2004) Telephone reminders are effective in recruiting nonresponding patients to randomized controlled trials. Journal of Clinical Epidemiology, 57: 773–776.and (1999) Few patients with prostate cancer are willing to be randomized to treatment. British Medical Journal, 318 (197): 1556.and (2003) Using random allocation to evaluate social interventions: The recent U.K. examples. The Annals of American Academy of Political and Social Science, 589: 170–189., , , , and (2003) Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology and Psychotherapy, 10: 361–373., , and (2008) Who is at risk for dropout from group cognitive-behavior therapy for insomnia? Journal of Psychosomatic Research, 64: 419–425., and (2011) Challenges and dilemmas in implementing random assignment in educational research. American Journal of Evaluation, 32 (1): 29–49., and (2005) A hybrid qualitative method for pre-testing questionnaires: The example of a questionnaire to caregivers of Alzheimer’s disease patients. Research in Nursing and Health, 28: 419–430., and ([Page 232] , , , , , , and (2006) Stated preferences of patients with cancer for health-related quality-of-life (HRQOL) domains during treatment. Quality of Life Research, 15: 273–283.1992) Impact of random assignment on study outcome: An empirical examination. Controlled Clinical Trials, 13: 50–61.(2009) Why we will remain pragmatists: Four problems with the impractical mechanistic framework and a better solution. Journal of Clinical Epidemiology, 62: 485–488., , , , and (2006) A fidelity measure for integrated management of depression in primary care. Medical Care, 44 (11): 1030–1037., , , , , and (2009) Randomization for clinical research: An easy-to-use spreadsheet method. Research in Nursing and Health, 32: 561–566., , , and (2008) The use of theory in health behavior research from 2000–2005: A systematic review. Annals of Behavioral Medicine, 35: 358–362., , , and (2005) Outcome of psychological treatments of pathological gambling: A review and analysis. Addiction, 100 (10): 1412–1422., , , and (1997) Realistic Evaluation. London: Sage.and (2004) Assessing the accuracy of administrative data in health information systems. Medical Care, 42 (11): 1066–1072., , , and (2005) Assessing the implementation of the chronic care model in quality improvement collaboratives. Health Services Research, 40 (4): 978–996., , , , , et al. (2002) Analysis of randomized controlled trials. Epidemiologic Reviews, 24 (1): 26–38., , and (2002) Limitations of randomized clinical trials in chronic diseases: Explanations and recommendations. Advances, 18 (2): 14–21.(2003) Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88 (5): 879–903., , and (2007) Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30 (4): 459–467., and (2003) Randomized controlled trials in psychiatry. Part 1: Methodology and critical evaluation. Australian and New Zealand Journal of Psychiatry, 37: 257–264., , and ([Page 233]Preference Collaborative Review Group (2009) Patients’ preferences within randomized trials: Systematic review and patient level meta-analysis. British Medical Journal Online. Retrieved 3 February 2009.2001) Empirical evaluation and justification of methodologies in psychological science. Psychological Bulletin, 127 (6): 759–772.and (Project MATCH Research Group (1998) Therapist effects in three treatments for alcohol problems. Psychotherapy Research, 8: 455–474.2001) Planning strategies for the avoidance of pitfalls in intervention research. Journal of Advanced Nursing, 35 (4): 514–520.and (1999) Placebos, drug effects, and study design: A clinician’s guide. American Journal of Psychiatry, 156 (6): 829–836.(2002) Hierarchical Linear Models: Applications and Data Analysis Methods (and (2nded.). Thousand Oaks, CA: Sage.2009) Patients’ depression treatment preferences and initiation, adherence, and outcome: A randomized primary care study. Psychiatric Services, 60: 337–343., , , and (2010) Rethinking pragmatic randomized controlled trials: Introducing the ‘cohort multiple randomized controlled trial’ design. British Medical Journal, 340: 963–967., , and (2004) Mass mailing and telephone contact were effective in recruiting veterans into an antibiotic treatment randomized clinical trial. Journal of Clinical Epidemiology, 57: 1063–1070., and (1996) Minimizing participant attrition in panel studies through the use of effective retention and tracking strategies: Review and recommendations. Evaluation and Program Planning, 19(1): 1–25., , , , and (2004) A five phase model for clinical-outcome research. Journal of Communication Disorder, 37: 401–411.(2005) Lay public’s understanding of equipoise and randomisation in randomised controlled trials. Health Technology Assessment, 9 (8)., , , , , , and (2000) Rigor at the expense of relevance equals rigidity: Where to next in the study of medical care utilization? Health Services Research, 34 (6): 1307–1314.(1999) Randomized play-the-winner clinical trials: Review and recommendations. Controlled Clinical Trials, 20: 328–342.(1999) Barriers to participation in randomized controlled trials: A systematic review. Journal of Clinical Epidemiology, 52 (12): 1143–1156., , , , and (2004) Evaluation: A Systematic Approach (, and (7thed.). Thousand Oaks, CA: Sage.[Page 234] (2005) External validity of randomized controlled trials: ‘To whom do the results of this study apply?’ The Lancet, 365: 82–93.1989) A two-stage trial design for testing treatment, self selection and treatment preference effects. Statistics in Medicine, 8(4): 477–485.(2009) Can the use of behavioural intervention studies support change in professional practice behaviours? A literature review. Evidence Based Midwifery, 7 (2): 54–59.and (2007) Commentary: Measuring the success of blinding in RCTs: Don’t, must, can’t or needn’t? International Journal of Epidemiology, 36: 664–665.(1982) Randomized versus historical controls for clinical trials. The American Journal of Medicine, 72: 233–240., and (1983) Sensitivity and specificity of clinical trials: Randomized v. historical controls. Archives of Internal Medicine, 143: 753–755., and (1998) Commitment and communication: Keys to minimizing attrition in multisite longitudinal organizational studies. Nursing Research, 47 (2): 123–125., , and (2005) Developing a process-evaluation plan for assessing health promotion program implementation: A how to guide. Health Promotion Practice, 6 (2): 134–147., and (2005) Insomnia in men treated with radical prostatectomy for prostate cancer. Psycho-Oncology, 14: 147–156., , , , and (2003) The importance of patient preferences in treatment decisions: Challenges for doctors. British Medical Journal, 327: 542–545.and (2008) Average causal effects from nonrandomized studies: A practical guide and simulation example. Psychological Methods, 13 (4): 279–313.and (2006) Randomized consent designs in randomized controlled trials: systematic literature search, Contemporary Clinical Trials, 27 (4): 320–332., , , and (2010) Detecting and connecting the lies that data tell. Perspectives on Psychological Science, 5 (3): 233–242.(2008) Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33(1); 62–87.(2002a) Generation of allocation sequences in randomised trials: Chance, not choice. The Lancet, 359: 515–519.and (2002b) Allocation concealment in randomised trials: Defending against deciphering. The Lancet, 359: 614–618.and (1995) Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Journal of the American Medical Association, 273(5): 408–412., , and ([Page 235] and (1995) Who says yes? Identifying selection biases in a psychosocial intervention study of multiple sclerosis. Social Science and Medicine, 40 (3): 359–370.1997) The control group dilemma in clinical research: Applications for psychosocial and behavioural medicine trials. Psychosomatic Medicine, 59: 362–371., , and (2004) Measuring patient treatment preferences in end-of-life care research: Applications for advance care planning interventions and response shift research. Journal of Palliative Medicine, 7: 233–245., , and (for the Seniors Medication Assessment Research Trial (SMART) Group (2002) Recruiting family physicians and patients for a clinical trial: Lessons learned. Family Practice, 19 (1): 99–104., , , , , et al.2010) How to define and measure concordance between patients’ preferences and medical treatments: A systematic review of approaches and recommendations for standardization. Patient Education and Counseling, 78: 12–23.and (2010) Campbell and Rubin: A primer and comparison of their approaches to causal inference in field settings. Psychological Methods, 15 (1): 3–17.(1996) Random versus nonrandom assignment in controlled experiments: Do you get the same answer? Journal of Consulting and Clinical Psychology, 64 (6): 1290–1305.and (1998) A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes. Psychological Methods, 3 (1): 3–22., , , and (2000) The effects of psychological therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin, 126 (4): 512–529., , and (2002) Experimental and Quasi-experimental Design for Generalized Causal Inference. Boston: Houghton-Mifflin., and (2011) A randomized experiment comparing random and cutoff-based assignment. Psychological Methods, 16: 179–191., , , and (1982) Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychological Bulletin, 92: 581–604.and (2002) Going quasi: The premature disclosure effect in a randomized clinical trial. Journal of Behavioural Medicine, 25 (6): 605–621., , , , , and (Parkinson Study Group. (1999) The placebo response in Parkinson’s disease. Clinical Neuropharmacology, 22 (4): 207–212., , , , and the2000) Enhancing adherence in randomized controlled clinical trials. Controlled Clinical Trials, 21: S226–S232., and ([Page 236] (2006) Random assignment: A systematic review. In and (eds), Strengthening Research Methodology: Psychological Measurement and Evaluation. Washington, DC: American Psychological Association Books, pp. 125–141.1998) Evaluating Nursing Interventions: A Theory-driven Approach. Thousand Oaks, CA: Sage.and (2011) Design, Evaluation, and Translation of Nursing Interventions. Ames, IA: Wiley-Blackwell.and (2007) Alternative Methods for Clinical Research (NP 05075). Final report submitted to the National Institute of Nursing Research., , , and (2006) Eliciting patient treatment preferences: a strategy to integrate evidence-based and patient-centred care, Worldviews on Evidence-Based Nursing, 3(3): 116–123., and (2009a) Influence of treatment preferences on validity: A review. Canadian Journal of Nursing Research, 41 (4): 52–67., , and (2009b) Assessment of preferences for treatment: Validation of a measure. Research in Nursing and Health, 32: 419–431., , , and (2010) Cultural adaptation and translation of measures: An integrated method. Research in Nursing and Health, 33: 133–143., , , and (2004) Medicare upcoding and hospital ownership, Journal of Health Economics, 23: 369–389.and (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22 (11): 1359–1366.., and (2007) Preference Measurement in Economic Analysis. Guidebook. Menlo Park, CA: VA Palo Alto, Health Economic Resource Center., and (2006) Specification and misspecification of theoretical foundations and logic models for health communication campaigns. Health Communication, 20 (2): 149–157.(1980) The Benefits of Psychotherapy. Baltimore, MD: Johns Hopkins University Press., and (2006) Beyond randomized controlled trials: A critical comparison of trials with non-randomized studies. Hepatology, 44: 1075–1082., and (The PLoS Medicine, et al. (2010) Comparative effectiveness research: Challenges for medical journals. Trials, 7 (4): e1000269., , , ,2010) Revisiting traditional survey methodology to recruit and surevy lesbian, gay and bisexual older adults (pp.211–218). In and (eds), When Research goes off the Rails. New York: Guilford Press.and ([Page 237] , , , , and (2007) Monitoring treatment fidelity in a randomized controlled trial of a complex intervention. Journal of Advanced Nursing, 60 (3): 343–352.Big Lung Trial Steering Committee (2005) Recruitment of patients with lung cancer into a randomised clinical trial: Experience at two centres. Thorax, 55: 463–465., , , and , on behalf of the2006) Counselor confounds in evaluations of vocational rehabilitation methods in substance dependency treatment. Evaluation Review, 30 (2): 139–170., and (Van Lin, E.N.J.T., , , et al. (2007) Doctors’ and patients’ preferences for participation and treatment in curative prostate cancer radiotherapy. Journal of Clinical Oncology, 25 (21): 3096–3100., ,Stanford Encyclopedia of Philosophy (2008) Counterfactual theories of causation.2007) Establishing fidelity of the independent variable in nursing clinical trials. Nursing Research, 56 (1): 54–62., and (2004) Research ethics, informed consent, and participant recruitment. Clinical Nurse Specialist, 18 (2): 88–97.(2004) Why do breast cancer patients decline entry into randomized trials and how do they feel about their decision later: A prospective, longitudinal, in-depth interview study. Patient Education and Counseling, 52: 341–348.and (2004) The placebo effect: Dissolving the expectancy versus conditioning debate. Psychological Bulletin, 130 (2): 324–340.and (2003) Are samples in randomized clinical trials of psychotherapy representative of community outpatients? A new methodology and initial findings. Journal of Consulting and Clinical Psychology, 71 (6): 963–972., , and (2005) Can the randomized controlled trial literature generalize to nonrandomized patients? Journal of Consulting and Clinical Psychology, 73 (1): 127–135., , and (2010) Randomized trials, statistics, and clinical inference. Journal of the American College of Cardiology, 55 (5): 428–431.and Pocock. S.J. (2002) The 2 ‘Es’ of research: Efficacy and effectiveness trials. Canadian Journal of Psychiatry, 47 (6): 552–556.(2008) Health Measurement Scales: A Practical Guide to Their Development and Use (and (4thed.). Oxford: Oxford University Press.2010) When Research Goes Off the Rails: Why it Happens and What You Can Do About it. New York: The Guilford Press.and (eds) (2011) Preferences. Journal of Clinical Psychology: In Session, 67 (2): 155–165., and ([Page 238] , and (2005) Using patient characteristics and attitudinal data to identify depression treatment preference groups: A latent-class model. Depression and Anxiety, 21: 47–54.1989) The medical outcomes study: An application of methods for monitoring the results of medical care. Journal of the American Medical Association, 262 (7): 925–930., , , , and (2006) The acceptability and preference for the psychological treatment of PTSD. Behaviour Research and Therapy, 44: 1643–1656., and (2003) Research to improve the simple randomized clinical trial. General Hospital Psychiatry, 25: 115–123., , and (2004) What influences participants’ treatment preference and can it influence outcome? Results from a primary care-based randomized trial for shoulder pain. British Journal of General Practice, 54: 93–96., , , and (2008) Publishing pilot intervention studies. Research in Nursing and Health, 31(1): 1–3.(2007) Who participates in diabetes self-management interventions? Issues of recruitment and retainment. The Diabetes Educator, 33 (3): 465–474., , , and (2009) A pragmatic – explanatory continuum indicator summary (PRECIS): A tool to help trial designers. Journal of Clinical Epidemiology, 62: 464–475., , , , , , , , , and (2000) Realistic evaluation: An overview. Paper presented at the Founding Conference of the Danish Evaluation Society.(2004) The need for pragmatic experimentation in educational research. Economics of Innovation and New Technology, 16 (5): 323–330.and (1996) Patient preference in randomized trials: Threat or opportunity? Journal of Health Science Research and Policy, 1 (4): 194–197., and (2004) Implementing randomized field trials in education: Report of a workshop. Committee on Research in Education, National Research Council. Washington, DC. National Academy Press.and (Southern European Psycho-Oncology Study (SEPOS) Group (2005) Physician–patient communication among southern European cancer physicians: The influence of psychosocial orientation and burnout. Psycho-Oncology, 14: 661–670., , , , and the2003) Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. Journal of the American Medical Association, 290 (12): 1624–1632., and ([Page 239]US Government Accountability Office (2009) Program evaluation: A variety of rigorous methods can help identify effective interventions. Retrieved from: http://www.gao.gov/products/GAO-10-302007) The effects of attrition on baseline comparability in randomized experiments in education: A meta analysis. Psychological Methods, 12 (3): 268–282.and (2008) Development and evaluation of a theory-based physical activity guidebook for breast cancer survivors. Health Education and Behavior, 35 (2): 174–189., , , and (2004) Acceptability and feasibility of an interactive computer-tailored fat intake intervention in Belgium. Health Promotion International, 19 (4): 463–470., and (2008) Understanding pregnant women’s decision making concerning prenatal screening. Health Psychology, 27 (4): 430–437., , , , , and (2010) How to integrate individual patient values and preferences in clinical practice guidelines? A research protocol. Implementation Science, 5: 10–18., , , , , , and (2009) Are we drawing the right conclusions from randomized placebo-controlled trials? A post-hoc analysis of data from a randomized controlled trial. BioMed Central Medical Research Methodology, 9: 41–47., , and (2001) General practice research: Problems and solutions in participant recruitment retention. Australian Family Physician, 30 (4): 399–406., , and (2004) Evidence based public health: Moving beyond randomized trials. American Journal of Public Health, 94 (3): 400–406., and (2005) Systematic review to determine whether participation in a trial influences outcome. British Medical Journal, 330: 1175–1181., , , , and (2003) Correlates of women’s preferences for treatment of heavy menstrual bleeding. Patient Education Counseling, 49: 125–132., , , , and (2009) Evidence-based treatment and therapist drift, Behavioural Research and Therapy, 47: 119–127.(1993) Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61 (4): 620–630., , and (2005) Estimating variability in outcomes attributable to therapists: A naturalistic study of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73 (5): 914–923.and ([Page 240] , and (2004) Using randomized controlled trials (RCTS) to test service interventions: Issues of standardisation, selection and generalisability. Nurse Researcher, 11 (3): 28–42.2011) Ethical issues posed by cluster randomized trials in health research. Trials, 12: 100., , , , , , , , , , and (2001) Multisite randomized controlled trials in health services research: Scientific challenges and operational issues. Medical Care, 39 (6): 627–634., , , , , and (2002a) Multisite randomized controlled trials in health services research: Scientific challenges and operational issues. Medical Care, 39: 627–634., , , , , and (2002b) Issues in conducting randomized controlled trials of health services research interventions in nonacademic practice settings: The case of retail pharmacies. Health Services Research, 37(4): 1067–1077., , , , , et al. (2012) On the proper use of the crossover design in clinical trials: part 18 of a series on evaluation of scientific publications. Deutsches Arzteblatt International, 109(15): 276–281.and (1995) Intention to treat: Once randomized, always analyzed. Clinical Aphasiology, 23: 57–64.(2010) Campbell’s and Rubin’s perspectives on causal inference. Psychological Methods, 15 (1): 18–37.and (2004) The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130 (4): 631–663., and (2000) Comparison of outcomes for clients in randomized versus open trials of treatment for alcohol use disorders. Journal of Studies on Alcohol, 6(5): 720–732., and (1993) A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice, 24 (2): 190–195.and (2002) How Close is Close Enough? Testing Non-experimental Estimates of Impact against Experimental Estimates of Impact with Education Test Scores as Outcomes. Madison, WI: Institute for Research on Poverty.and (1999) Generalizing from a controlled trial: The effects of patient preference versus randomization on the outcome of impatient versus outpatient chronic pain management. Pain, 83: 57–65., , , and ([Page 241] and (2001) The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6 (4): 413–429.2003) Are samples in randomized controlled trials of psychotherapy representative of community outpatients? A new methodology and initial findings. Journal of Consulting and Clinical Psychology, 71 (6): 963–972., , and (1993) Personality traits and early discontinuation from clinical trials in anxious patients. Journal of Clinical Psychopharmacology, 13 (3): 194–197., , , , and (2008) Empirical evidence of bias in treatment effect sizes in controlled trials with different interventions and outcomes: Meta-epidemiological study. British Medical Journal, 336: 601., , , , , , , , and (2002) Participation of African Americans in a smoking cessation trial: A quantitative and qualitative study. Journal of the National Medical Association, 94 (7): 609–618., , , , and (2002) What evidence in evidence-based medicine? Philosophy of Science, 69 (S3): S316–S330.(2012) Bad copy. Nature, 485: 1298–300.(2006) Pretreatment attrition and childhood social phobia. Anxiety Disorders, 20: 1133–1147., , , , and (2001) Posttraumatic concerns: A patient-centered approach to outcome assessment after traumatic physical injury. Medical Care, 39 (4): 327–339., , et al. (1979) A new design for randomized clinical trials. The New England Journal of Medicine, 300: 1242–1245.(2003) Treatment choice for PTSD. Behaviour Research and Therapy, 41: 879–886., , and (2009) What kind of randomized trials do we need? Canadian Medical Association Journal, 180 (10): 998–1000.and (