Advancing Implementation Science through Measure Development and Evaluation

Sub-award funding has been awarded to UW investigator Bryan Weiner by the NIH National Institute of Mental Health via Group Health Cooperative (now Kaiser Permanente Washington Health Research Institute) for "Advancing Implementation Science Through Measure Development and Evaluation".

Abstract:

Poor quality and impractical Measurement has impeded the study of Implementation barriers, facilitators and strategies necessary for promoting widespread delivery of evidence-based care and improving mental health. Measures derive research and policy/practice insights that guide subsequent decision making. However, for Measures to advance Science and inform practice they need to be both psychometrically strong and pragmatic. No studies have evaluated the psychometric properties of a comprehensive array of Measures delineated in highly-cited, foundational Implementation Science frameworks such as the Consolidated Framework for Implementation Research (CFIR) or the Implementation Outcomes Framework (IOF). Moreover, no research has attempted to advance pragmatic Implementation Science Measures, and the only existing pragmatic criteria was generated in the absence of stakeholder involvement.

Our long-term goal is to develop a comprehensive battery of reliable, valid, and pragmatic Measures that researchers and stakeholders could use to advance the Science and practice of Implementation. The overarching objective of this application is to put forth a measurement-focused research agenda for Implementation Science (i.e., which constructs possess psychometrically strong and pragmatic Measures, which require further Development) as well as Measures and methods to accomplish this work.

Three aims guide this proposal. One, establish a stakeholder-driven operationalization of pragmatic Measures and develop reliable, valid rating criteria for assessing the construct. Our approach to generating "pragmatic" rating criteria employs mixed methods (literature review, qualitative interviews, and Q-sort and Delphi methods) to engage stakeholders. Two, develop reliable, valid, and pragmatic Measures of accept- ability, appropriateness, and feasibility. We have vetted an effective Development procedure to establish their reliability; substantive, discriminant content, structural, known-groups, and predictive validity; structural invariance; and sensitivity to change. Our approach to developing and testing new Measures involves a series of laboratory studies to assess psychometric properties under controlled conditions and field studies to gauge psychometric and pragmatic performance under "real world" conditions. Three, identify CFIR and IOF-linked Measures that demonstrate both psychometric and pragmatic strength. Our approach to assessing over 420 existing Measures of 46 constructs builds upon our preliminary studies and accesses an established task force. The pragmatic Measure rating criteria will be integrated into our established psychometric rating criteria rendering psychometric and pragmatic Measure profiles. We predict that few existing Measures are psycho- metrically sound and pragmatic, but that rigorous, replicable Development and testing procedures will generate Measures that possess these qualities. Once pragmatic, psychometrically strong Measures are identified, the field can generate robust scientific knowledge about which Implementation strategies work best, when, and for whom so that stakeholders can use evidence-based approaches for implementing evidence-based practices.

Dr. Weiner will focus on these aims of the overall project: Predictive Validity Assessment: Using a list of NCI Community Oncology Research Program (NCORP) physicians provided by NCI/DCP and data from the American Medical Association (AMA) Physician Masterfile, we will draw a random sample of 600 NCORP physicians, stratified by NCORP organization and oncology specialty (hematology, medical, and radiation). We will survey physicians to obtain data on their perceptions of acceptability, appropriateness, and feasibility of an innovative clinical practice embedded in a Phase II/III clinical trial. We will obtain data on physicians’ adoption and use of the clinical trial from the NCORP organizations’ administrators. We will obtain data on physicians’ demographics, medical specialty, and years in practice from the AMA Physician Masterfile, which includes data for more than 1.4 million physicians, residents, and medical students in the US.

We will test our hypothesis by testing a structural equation model in Mplus 7 (Muthén & Muthén, 2012) using the three latent factors from the 3-factor CFA as exogenous predictors, and adoption and use as the endogenous outcomes. Evidence-Based Assessment (EBA) of implementation-related constructs: In collaboration with the Principal Investigator, we will modify the evidence-based assessment criteria by adding specifiers to the predictive validity criterion to clarify which implementation outcome measures (acceptability, feasibility, appropriateness, adoption, penetration, cost, sustainment, or fidelity) the focal construct has been demonstrated to predict. We will refine the EBA rating criteria based on a pilot test and work with the Principal Investigator to recruit, train, and supervise raters to begin rating packets in Year 2 (2016-2017) and complete rating packets in Year 3 (2017-2018).

Principal Investigator: Cara C. Lewis (Kaiser Permanente Washington Health Research Institute; affiliate assistant professor, UW Department of Psychiatry and Behavioral Sciences)
Sponsor Award Number: 5R01MH106510-04