How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations.

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

The impact of recreational cannabis laws on cannabis use disorder during "treat and release" visits to hospital emergency departments in four U.S. states, 2017-2020.

States' legalization of cannabis influences cannabis use and may increase cannabis use disorder (CUD)-a problematic pattern of use leading to significant impairment. Few studies have examined the influence of recreational cannabis legalization on CUD in the emergency department (ED). We used four years of data from the State Emergency Department Databases (SEDD) (2017-2020) from three states (CO, MD, OR) and three years of SEDD from Rhode Island (2017-2019) to examine the relationship between the recreational legalization of cannabis and CUD among "treat and release" ED visits. During the study period, CO and OR were legal for recreational cannabis while it was illegal in MD and RI. We examined the proportion of ED visits for CUD and used multivariate logistic regression to examine the association between recreational legalization and CUD diagnosis. The sample had 17,434,655 ED visits (56.2 % female). The proportion of ED visits for CUD was 0.63 %. Annual rates ranged from 0.67 % (2017) to 0.59 % (2019) and state-level rates were 0.39 % (CO), 0.35 % (OR), 1.03 % (MD), and 0.79 % (RI). Compared to ED visits in legal states, a higher proportion of ED visits in non-legal states were from women (56.8 % versus 55.7 %) and Blacks (40.9 % versus 5.9 %). Compared to states where recreational cannabis was illegal, legalizing cannabis for recreational use was associated with nearly a 50 % decrease in the adjusted odds of CUD (AOR = 0.49, 95 % CI 0.47, 0.52). In summary, CUD rates among "treat and release" ED visits were significantly lower in legalized states than in non-legal states.

A comparison of the practice patterns of emergency department teams that include physicians, nurse practitioners, or physician assistants.

BACKGROUND: Physicians see most emergency department (ED) patients, but, recently, nurse practitioners (NPs) and physician assistants (PAs) have provided an increasing amount of ED care. PURPOSE: Compare NP and PA teams' practice patterns to physician teams in EDs. METHODS: Using 12 years of data from the National Hospital Ambulatory Medical Care Survey (2009-2020), we used multivariate regression analysis to separately examine the associations between the ED practice patterns (i.e., number of diagnostic services, number of procedures, waiting time, boarding time, length of visit, and hospital admission) of patients seen by NP or PA teams compared with physician teams. DISCUSSION: Patient visits to NP and PA teams received fewer diagnostic services and procedures, had shorter visits, and were less likely to be hospitalized. CONCLUSION: If the additional diagnostic services, procedures, and hospital admission provided by physician teams were unnecessary for the patients studied, NP and PA team care could be more efficient.

Differences in the Patterns of Care Between Emergency Department Visits with and Without a Physician.

BACKGROUND: A variety of clinicians practice in emergency departments (EDs). Although most ED patients prefer seeing physicians, a subset sees no physician. OBJECTIVES: We sought to determine the factors that predict when an ED patient is seen by at least one physician and compared the practice patterns of patient visits seen by at least one physician compared with those seen by no physician. METHODS: We used 11 years of cross-sectional data from the National Hospital Ambulatory Medical Care Survey and focused on the sample of ED patient visits seen by at least one physician and those seen by no physician. We used bivariate statistics to compare characteristics between samples and used multivariate logistic regression analysis to identify the factors that predicted being seen by a physician. Finally, we compared the practice patterns of patient visits seen by at least one physician compared with those seen by no physician. RESULTS: Approximately 10% of the sample was not seen by any physician. Patients seen by at least one physician had, on average, 0.8 more diagnostic services ordered/provided and 0.1 more procedures provided compared with patients who were not seen by any physician. Patients seen by at least one physician had longer visits by 29.4 min, on average, and had increased odds of being hospitalized (adjusted odds ratio 3.9, 95% confidence interval 2.9-5.2). CONCLUSIONS: A variety of patient and hospital characteristics influenced whether ED patients were seen by physicians. Diagnostic services, procedures, visit length, and hospital admission differed by physician presence. Findings have implications for ED practice and future research.

Do Teams Improve the Quality of Ambulatory Care?

Using data from the National Ambulatory Medical Care Survey, we examined team composition in office-based practices and compared their relative quality of care. We found that, compared with physician-only teams, patients seen by physician and nurse practitioner/nurse midwife teams and those seen by physician and nurse teams were more likely to receive statins for hyperlipidemia and blood pressure screening, respectively. We also found that patients seen by physician and physician assistant teams were less likely to receive recommended care for all 4 quality indicators, and patients seen by any interprofessional team were less likely to receive recommended depression treatment than physician-only teams.

Special Issue Editors' Essay.

BACKGROUND: Impact evaluations draw their data from two sources, namely, surveys conducted for the evaluation or administrative data collected for other purposes. Both types of data have been used to estimate program impacts. This is an introductory essay to a Special Issue entitled "Do the Estimated Effects of Social Programs Depend on the Source of Data Used to Measure Them? Survey Data Versus Administrative Data." In addition to this essay, the Special Issue contains six articles, which appear in Volume 42, Issue 5-6 (October-December 2018) and in this issue (Volume 43, Issue 5 (October 2019)) of Evaluation Review. OBJECTIVE: To describe and summarize each of the six papers and draw lessons from them. The papers investigate the relative strengths and weaknesses of survey and administrative data for estimating the impacts of policy interventions. RESULTS: This essay first describes a simple model of the mechanisms that can cause impacts estimated with survey data to differ from those estimated with administrative data. It then describes and summarizes each of the papers appearing in this Special Issue and uses the model described to interpret the findings when it is applicable. The final section draws general lessons from the papers. CONCLUSIONS: The decision on whether to use survey or administrative data to estimate program impacts can be highly consequential because the estimates can differ considerably. All the papers in this Special Issue point to the importance of using both survey data and administrative data whenever possible.

A Comparison of Nurse Practitioners, Physician Assistants, and Primary Care Physicians' Patterns of Practice and Quality of Care in Health Centers.

BACKGROUND: Under the Affordable Care Act, the number and capacity of community health centers (HCs) is growing. Although the majority of HC care is provided by primary care physicians (PCMDs), a growing proportion is delivered by nurse practitioners (NPs) and physician assistants (PAs); yet, little is known about how these clinicians' care compares in this setting. OBJECTIVES: To compare the quality of care and practice patterns of NPs, PAs, and PCMDs in HCs. RESEARCH DESIGN: Using 5 years of data (2006-2010) from the HC subsample of the National Ambulatory Medical Care Survey and multivariate regression analysis, we estimated the impact of receiving NP-delivered or PA-delivered care versus PCMD-delivered care. We used design-based and model-based inference and weighted all estimates. SUBJECTS: Primary analyses included 23,704 patient visits to 1139 practitioners-a sample representing approximately 30 million patient visits to HCs in the United States. MEASURES: We examined 9 patient-level outcomes: 3 quality indicators, 4 service utilization measures, and 2 referral pattern measures. RESULTS: On 7 of the 9 outcomes studied, no statistically significant differences were detected in NP or PA care compared with PCMD care. On the remaining outcomes, visits to NPs were more likely to receive recommended smoking cessation counseling and more health education/counseling services than visits to PCMDs (P≤0.05). Visits to PAs also received more health education/counseling services than visits to PCMDs (P≤0.01; design-based model only). CONCLUSIONS: Across the outcomes studied, results suggest that NP and PA care were largely comparable to PCMD care in HCs.

Does the Regulatory Environment Affect Nurse Practitioners' Patterns of Practice or Quality of Care in Health Centers?

OBJECTIVE: To examine the impact of state-granted nurse practitioner (NP) independence on patient-level quality, service utilization, and referrals. DATA SOURCES/STUDY SETTING: The National Ambulatory Medical Care Survey's community health center (HC) subsample (2006-2011). Primary analyses included approximately 6,500 patient visits to 350 NPs in 220 HCs. STUDY DESIGN: Propensity score matching and multivariate regression analysis were used to estimate the impact of state-granted NP independence on each outcome, separately. Estimates were adjusted for sampling weights and NAMCS's complex design. DATA COLLECTION/EXTRACTION METHODS: Every "NP-patient visit unit" was isolated using practitioner and patient visit codes and, using geographic identifiers, assigned to its state-year and that state-year's level of NP independence based on scope of practice policies. Nine outcomes were specified using ICD-9 codes, standardized drug classification codes, and NAMCS survey items. PRINCIPAL FINDINGS: After matching, no statistically significant differences in quality were detected by states' independence status, although NP visits in states with prescriptive independence received more educational services (aIRR 1.66; 95 percent CI 1.09-2.53; p = .02) and medications (aIRR 1.26; 95 percent CI 1.04-1.53; p = .02), and NP visits in states with practice independence had a higher odds of receiving physician referrals (AOR 1.88; 95 percent CI 1.10-3.22; p = .02) than those in restricted states. CONCLUSIONS: Findings do not support a quality-scope of practice relationship.

Do estimated impacts on earnings depend on the source of the data used to measure them? Evidence from previous social experiments.

BACKGROUND: Impact evaluations draw their data from two sources, namely, surveys conducted for the evaluation or administrative data collected for other purposes. Both types of data have been used in impact evaluations of social programs. OBJECTIVE: This study analyzes the causes of differences in impact estimates when survey data and administrative data are used to evaluate earnings impacts in social experiments and discusses the differences observed in eight evaluations of social experiments that used both survey and administrative data. RESULTS: There are important trade-offs between the two data sources. Administrative data are less expensive but may not cover all income and may not cover the time period desired, while surveys can be designed to avoid these problems. We note that errors can be due to nonresponse or reporting, and errors can be balanced between the treatment and the control groups or unbalanced. We find that earnings are usually higher in survey data than in administrative data due to differences in coverage and likely overreporting of overtime hours and pay in survey data. Evaluations using survey data usually find greater impacts, sometimes much greater. CONCLUSIONS: The much lower cost of administrative data make their use attractive, but they are still subject to underreporting and other problems. We recommend further evaluations using both types of data with investigative audits to better understand the sources and magnitudes of errors in both survey and administrative data so that appropriate corrections to the data can be made.

Flaws in evaluations of social programs: illustrations from randomized controlled trials.

BACKGROUND: This article describes eight flaws that occur in impact evaluations. METHOD: The eight flaws are grouped into four categories on how they affect impact estimates: statistical imprecision; biases; failure of impact estimates to measure effects of the planned treatment; and flaws that result from weakening an evaluation design. Each flaw is illustrated with examples from social experiments. Although these illustrations are from randomized controlled trials (RCTs), they can occur in any type of evaluation; we use RCTs to illustrate because people sometimes assume that RCTs might be immune to such problems. A summary table lists the flaws, indicates circumstances under which they occur, notes their potential seriousness, and suggests approaches for minimizing them. RESULTS: Some of the flaws result in minor hurdles, while others cause evaluations to fail-that is, the evaluation is unable to provide a valid test of the hypothesis of interest. The flaws that appear to occur most frequently are response bias resulting from attrition, failure to adequately implement the treatment as designed, and too small a sample to detect impacts. The third of these can result from insufficient marketing, too small an initial target group, disinterest on the part of the target group in participating (if the treatment is voluntary), or attrition. CONCLUSION: To a considerable degree, the flaws we discuss can be minimized. For instance, implementation failures and too small a sample can usually be avoided with sufficient planning, and response bias can often be mitigated-for example, through increased follow-up efforts in conducting surveys.