2020 Rossi Award Lecture: The Evolving Art of Program Evaluation.

Evaluation of public programs has undergone many changes over the past four decades since Peter Rossi coined his "Iron Law" of program evaluation: "The expected value of any net impact assessment of any large-scale social program is zero." While that assessment may be somewhat overstated, the essence still holds. The failures far outnumber the successes, and the estimated favorable effects are rarely sizeable. Despite this grim assessment, much can be learned from "failed" experiments, and from ones that are successful in only some sites or subgroups. Advances in study design, statistical models, data, and how inferences are drawn from estimates have substantially improved our analyses and will continue to do so. However, the most actual learning about "what works" (and why, when, and where) is likely to come from gathering more detailed and comprehensive data on how the intervention was implemented and attempting to link that data to estimated impacts. Researchers need detailed data on the target population served, the content of the intervention, and the process by which it is delivered to participating service providers and individuals. Two examples presented here illustrate how researchers drew useful broader lessons from impact estimates for a set of related programs. Rossi posited three reasons most interventions fail-wrong question, wrong intervention, poor implementation. Speeding the accumulation of wisdom about how social programs can best help vulnerable populations will require that researchers work closely with program funders, developers, operators, and participants to gather and interpret these detailed data about program implementation.

Patient experience during a large primary care practice transformation initiative.

OBJECTIVES: To determine how the multipayer Comprehensive Primary Care (CPC) initiative that transformed primary care delivery affected patient experience of Medicare fee-for-service beneficiaries. The study examines whether patient experience changed during the 4-year initiative, whether ratings of CPC practices changed relative to ratings of comparison practices, and areas in which practices still have an opportunity to improve patient experience. STUDY DESIGN: Prospective study using 2 cross-sectional samples of more than 25,000 Medicare fee-for-service beneficiaries attributed to 490 CPC practices and more than 8000 beneficiaries attributed to 736 comparison practices. METHODS: We analyzed patient experience 8 to 12 months and 45 to 48 months after CPC began, measured using 5 domains of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group survey with Patient-Centered Medical Home items, version 2.0. A regression-adjusted analysis compared differences in the proportion of beneficiaries giving the best responses (and, as a sensitivity test, mean responses) to survey questions over time and between CPC and comparison practices. RESULTS: Patient ratings of care over time were generally comparable for CPC and comparison practices. CPC had favorable effects on measures of follow-up care after hospitalizations and emergency department visits. CONCLUSIONS: Practice transformation did not alter patient experience. The lack of favorable findings raises questions about how future efforts in primary care can succeed in improving patient experience.

The Comprehensive Primary Care Initiative: Effects On Spending, Quality, Patients, And Physicians.

The Comprehensive Primary Care Initiative (CPC), a health care delivery model developed by the Centers for Medicare and Medicaid Services (CMS), tested whether multipayer support of 502 primary care practices across the country would improve primary care delivery, improve care quality, or reduce spending. We evaluated the initiative's effects on care delivery and outcomes for fee-for-service Medicare beneficiaries attributed to initiative practices, relative to those attributed to matched comparison practices. CPC practices reported improvements in primary care delivery, including care management for high-risk patients, enhanced access, and improved coordination of care transitions. The initiative slowed growth in emergency department visits by 2 percent in CPC practices, relative to comparison practices. However, it did not reduce Medicare spending enough to cover care management fees or appreciably improve physician or beneficiary experience or practice performance on a limited set of Medicare claims-based quality measures. As CMS and other payers increasingly use alternative payment models that reward quality and value, CPC provides important lessons about supporting practices in transforming care.

Care coordination for children with special needs in Medicaid: lessons from Medicare.

OBJECTIVES: To provide actionable recommendations for improving care coordination programs for children with special healthcare needs (CSHCN) in Medicaid managed care. STUDY DESIGN: Literature review and interviews with stakeholders and policy experts to adapt lessons learned from Medicare care coordination programs for CSHCN in Medicaid managed care. METHODS: We reviewed syntheses of research on Medicare care coordination programs to identify lessons learned from successful programs. We adapted findings from Medicare to CSHCN in Medicaid based on an environmental scan and discussions with experts. The scan focused on Medicaid financing and eligibility for care coordination and how these intersect with Medicaid managed care. The expert discussions included pediatricians, Medicaid policy experts, Medicaid medical directors, and a former managed care executive, all experienced in care coordination for CSHCN. RESULTS: We found 6 elements that are consistently associated with improved outcomes from Medicare care coordination programs and relevant to CSHCN in Medicaid: 1) identifying and targeting high-risk patients, 2) clearly articulating what outcomes programs are likely to improve, 3) encouraging active engagement between care coordinators and primary care providers, 4) requiring some in-person contact between care coordinators and patients, 5) facilitating information sharing among providers, and 6) supplementing care coordinators' expertise with that of other clinical experts. CONCLUSIONS: States and Medicaid managed care organizations have many options for designing effective care coordination programs for CSHCN. Their choices should account for the diversity of conditions among CSHCN, families' capacity to coordinate care, and social determinants of health.

Development and Validation of the Modified Patient-Centered Medical Home Assessment for the Comprehensive Primary Care Initiative.

OBJECTIVE: To describe the modified Patient-Centered Medical Home Assessment (M-PCMH-A) survey module developed to track primary care practices' care delivery approaches over time, assess whether its underlying factor structure is reliable, and produce factor scores that provide a more reliable summary measure of the practice's care delivery than would a simple average of question responses. DATA SOURCES/STUDY SETTING: Survey data collected from diverse practices participating in the Comprehensive Primary Care (CPC) initiative in 2012 (n = 497) and 2014 (n = 493) and matched comparison practices in 2014 (n = 423). STUDY DESIGN: Confirmatory factor analysis. DATA COLLECTION: Thirty-eight questions organized in six domains: Access and Continuity of Care, Planned Care for Chronic Conditions and Preventive Care, Risk-Stratified Care Management, Patient and Caregiver Engagement, Coordination of Care across the Medical Neighborhood, and Continuous Data-Driven Improvement. PRINCIPAL FINDINGS: Confirmatory factor analysis suggested using seven factors (splitting one domain into two), reassigning two questions to different domain factors, and removing one question, resulting in high reliability, construct validity, and stability in all but one factor. The seven factors together formed a single higher-order factor summary measure. Factor scores guard against potential biases from equal weighting. CONCLUSIONS: The M-PCMH-A can validly and reliably track primary care delivery across practices and over time using factors representing seven key components of care as well as an overall score. Researchers should calculate factor loadings for their specific data if possible, but average scores may be suitable if they cannot use factor analysis due to resource or sample constraints.

Adapting Evaluations of Alternative Payment Models to a Changing Environment.

OBJECTIVE: To identify the most robust methods for evaluating alternative payment models (APMs) in the emerging health care delivery system environment. STUDY DESIGN (APPROACH): We assess the impact of widespread testing of alternative payment models on the ability to find credible comparison groups. We consider the applicability of factorial research designs for assessing the effects of these models. PRINCIPAL FINDINGS: The widespread adoption of alternative payment models could effectively eliminate the possibility of comparing APM results with a "pure" control or comparison group unaffected by other interventions. In this new environment, factorial experiments have distinct advantages over the single-model experimental or quasi-experimental designs that have been the mainstay of recent tests of Medicare payment and delivery models. CONCLUSIONS: The best prospects for producing definitive evidence of the effects of payment incentives for APMs include fractional factorial experiments that systematically vary requirements and payment provisions within a payment model.

Patient experience midway through a large primary care practice transformation initiative.

OBJECTIVES: To determine how the multipayer Comprehensive Primary Care (CPC) initiative that transforms primary care delivery affects the patient experience of Medicare fee-for-service beneficiaries. The study examines how experience changed between the first and second years of CPC, how ratings of CPC practices have changed relative to ratings of comparison practices, and areas in which practices still have opportunities to improve patient experience. STUDY DESIGN: Prospective study using 2 serial cross-sectional samples of more than 25,000 Medicare fee-for-service beneficiaries attributed to 496 CPC practices and nearly 9000 beneficiaries attributed to 792 comparison practices. METHODS: We analyzed patient experience 8 to 12 months and 21 to 24 months after CPC began, measured using 6 domains of the Consumer Assessment of Healthcare Providers and Systems Clinician and Group 12-Month Survey with Patient-Centered Medical Home supplemental items. We compared changes over time in patients giving the best responses between CPC and comparison practices using a regression-adjusted difference-in-differences analysis. RESULTS: Patient ratings of care over time were generally comparable for CPC and comparison practices, with slightly more favorable differences-generally of small magnitude-for CPC practices than expected by chance. There were small, statistically significant, favorable effects for 2 of 6 composite measures measured using both the proportion giving the best responses and mean responses: getting timely appointments, care, and information; providers support patients in taking care of their own health; and providers discuss medication decisions. There was an additional small favorable effect on the proportion of patients giving the best response in getting timely appointments, care, and information; there was no effect on the mean. CONCLUSIONS: During the first 2 years of CPC, CPC practices showed slightly better year-to-year patient experience ratings for selected items, indicating that transformation did not negatively affect patient experience and improved some aspects slightly. Patient ratings for the 2 groups were generally comparable, and both faced substantial room for improvement.

Testing the Replicability of a Successful Care Management Program: Results from a Randomized Trial and Likely Explanations for Why Impacts Did Not Replicate.

OBJECTIVES: To test whether a care management program could replicate its success in an earlier trial and determine likely explanations for why it did not. DATA SOURCES/SETTING: Medicare claims and nurse contact data for Medicare fee-for-service beneficiaries with chronic illnesses enrolled in the trial in eastern Pennsylvania (N = 483). STUDY DESIGN: A randomized trial with half of enrollees receiving intensive care management services and half receiving usual care. We developed and tested hypotheses for why impacts declined. DATA EXTRACTION: All outcomes and covariates were derived from claims and the nurse contact data. PRINCIPAL FINDINGS: From 2010 to 2014, the program did not reduce hospitalizations or generate Medicare savings to offset program fees that averaged $260 per beneficiary per month. These estimates are statistically different (p < .05) from the large reductions in hospitalizations and spending in the first trial (2002-2010). The treatment-control differences in the second trial disappeared because the control group's risk-adjusted hospitalization rate improved, not because the treatment group's outcomes worsened. CONCLUSION: Even if demonstrated in a randomized trial, successful results from one test may not replicate in other settings or time periods. Assessing whether gaps in care that the original program filled exist in other settings can help identify where earlier success is likely to replicate.

Staffing patterns of primary care practices in the comprehensive primary care initiative.

PURPOSE: Despite growing calls for team-based care, the current staff composition of primary care practices is unknown. We describe staffing patterns for primary care practices in the Centers for Medicare and Medicaid Services (CMS) Comprehensive Primary Care (CPC) initiative. METHODS: We undertook a descriptive analysis of CPC initiative practices' baseline staffing using data from initial applications and a practice survey. CMS selected 502 primary care practices (from 987 applicants) in 7 regions based on their health information technology, number of patients covered by participating payers, and other factors; 496 practices were included in this analysis. RESULTS: Consistent with the national distribution, most of the CPC initiative practices included in this study were small: 44% reported 2 or fewer full-time equivalent (FTE) physicians; 27% reported more than 4. Nearly all reported administrative staff (98%) and medical assistants (89%). Fifty-three percent reported having nurse practitioners or physician assistants; 47%, licensed practical or vocational nurses; 36%, registered nurses; and 24%, care managers/coordinators-all of these positions are more common in larger practices. Other clinical staff were reported infrequently regardless of practice size. Compared with other CPC initiative practices, designated patient-centered medical homes were more likely to have care managers/coordinators but otherwise had similar staff types. Larger practices had fewer FTE staff per physician. CONCLUSIONS: At baseline, most CPC initiative practices used traditional staffing models and did not report having dedicated staff who may be integral to new primary care models, such as care coordinators, health educators, behavioral health specialists, and pharmacists. Without such staff and payment for their services, practices are unlikely to deliver comprehensive, coordinated, and accessible care to patients at a sustainable cost.

Six features of Medicare coordinated care demonstration programs that cut hospital admissions of high-risk patients.

As policy makers seek to slow the growth in Medicare spending, they have appropriately focused attention on beneficiaries with multiple chronic conditions. Many care coordination and disease management programs designed to improve beneficiaries' care and reduce their need for hospitalizations have been tested, but few have been successful. This study, however, found that four of eleven programs that were part of the Medicare Coordinated Care Demonstration reduced hospitalizations by 8-33 percent among enrollees who had a high risk of near-term hospitalization. The six approaches practiced by care coordinators in at least three of the four programs were as follows: supplementing telephone calls to patients with frequent in-person meetings; occasionally meeting in person with providers; acting as a communications hub for providers; delivering evidence-based education to patients; providing strong medication management; and providing timely and comprehensive transitional care after hospitalizations. When care management fees were included, the programs were essentially cost-neutral, but none of these programs generated net savings to Medicare. Our results suggest that incorporating these approaches into medical homes, accountable care organizations, and other policy initiatives could reduce hospitalizations and improve patients' lives. However, the approaches would save money only if care coordination fees were modest and organizations found cost-effective ways to deliver the interventions.

How changes in Washington University's Medicare coordinated care demonstration pilot ultimately achieved savings.

As one of the initial fifteen participants in the Medicare Coordinated Care Demonstration, the Washington University School of Medicine in St. Louis was not able to demonstrate any reduction in hospitalizations or Medicare spending for the patients it served. In fact, the Washington University program increased total Medicare spending by 12 percent. But after a redesign, the results changed. The program stopped conducting care management of most of its patients via telephone from a remote site in California and, instead, served all patients through frequent phone and occasional in-person contact from local care managers in St. Louis. Care management efforts were focused especially on patients deemed at greatest risk of hospitalization, and stronger hospital transition planning and medication reconciliation were provided, among other changes. After that point, the program reduced hospitalizations by 12 percent and monthly Medicare spending by $217 per enrollee-more than offsetting the program's monthly $151 care management fee. The results underscore findings from the overall Medicare Coordinated Care Demonstration that suggest that programs with more in-person contacts were more likely than others to build trusting relationships with patients and providers, improve patient adherence to care plans, and address additional needs and barriers that entirely telephonic contacts had been unable to identify. The results also indicate that programs can be more effective by focusing on the highest-risk patients, for whom the largest savings resulted.

The research design and methodological issues for the Cash and Counseling Evaluation.

OBJECTIVE: To provide an overview of the design, research questions, data sources, and methods used to evaluate the Cash and Counseling Demonstration and resolution of analytic concerns that arose. The methodology was designed to provide statistically rigorous estimates while presenting the findings in a manner easily accessible to a broad, non-technical audience. STUDY SETTING: Eligible Medicaid beneficiaries in Arkansas, Florida, and New Jersey who volunteered to participate in the demonstration were randomly assigned to receive an allowance and direct their own Medicaid supportive services as Cash and Counseling consumers (the treatment group) or to rely on Medicaid services as usual (the control group). The demonstration included elderly and non-elderly adults in all three states and children in Florida. Both age groups in Arkansas and New Jersey and the elderly adults group in Florida primarily included individuals with physical disabilities. In Florida, the children and non-elderly adults primarily included individuals with developmental disabilities. The intervention was conducted from 1999 through 2003. DATA SOURCES: Data included baseline and 9-month follow-up surveys of consumers, surveys of the primary informal caregiver and the primary paid worker for sample members, program data, interviews with program staff, and Medicaid and Medicare claims data. METHODS: Descriptive data analyses were conducted on program participation, program implementation, and the experiences of hired workers. Program impacts on consumers, caregivers, and costs were estimated using an intent-to-treat-approach, comparing the regression-adjusted means of outcomes for the full treatment and control groups. A broad set of control variables from the baseline interview and prior Medicaid claims data controlled for possible preexisting differences. Ordinal scale responses were converted to binary outcome indicators for high and for low values for ease of presentation and interpretation of effects. Two-tailed statistical tests of the estimated effects were conducted at the .05 level. Separate estimates were provided for each state and for each age group. Sensitivity tests were conducted of the robustness of estimates to outliers (for continuous outcome measures) and to proxy use. PRINCIPAL FINDINGS/CONCLUSION: The experimental design, high survey response rates, and available sample sizes led to valid, unbiased estimates of program impacts, with adequate power to detect moderate-size impacts on most outcomes for the key age subgroups examined. For certain survey-based outcome measures related to satisfaction with paid care, the sample had to be restricted to those who received care and those without proxy respondents who were also hired workers. Sensitivity tests suggest that these necessary restrictions were unlikely to have led to over statement of favorable program effects on these outcome measures. The high proportions of sample members with proxy respondents reflect the frailty of the sample members. Similar rates for treatment and control groups cases with proxy respondents suggest the high use of proxy respondents has not biased the estimated program effects on survey measures.

How does Cash and Counseling affect costs?

OBJECTIVE: To test the effect of a consumer-directed model (Cash and Counseling) of Medicaid personal care services (PCS) or home- and community-based waiver services (HCBS) on the cost of Medicaid services. DATA SOURCES/STUDY SETTING: Medicaid claims data were collected for all enrollees in the Cash and Counseling demonstration. Demonstration enrollees included those eligible for PCS (in Arkansas), those assessed to receive such services (in New Jersey), and recipients of Medicaid HCBS (in Florida). Enrollment occurred from December 1998 through April 2001. The follow-up period covered up to 24 months after enrollment. STUDY DESIGN: Demonstration volunteers were randomly assigned to have the option to participate in Cash and Counseling (the treatment group), or to receive Medicaid services as usual from an agency (the control group). Ordinary least squares regressions were used to estimate the effect of the program on costs for Medicaid PCS/waiver services and other Medicaid services, while controlling for consumers' preenrollment characteristics and preenrollment Medicaid spending. Models were estimated separately for nonelderly and elderly adults in each state and for children in Florida. DATA EXTRACTION METHODS: Each state supplied claims data for demonstration enrollees. PRINCIPAL FINDINGS: Largely because the program increased consumers' ability to get the authorized amount of paid care, expenditures for personal care/waiver services were higher for the treatment group than for the control group in each state and age group, except among the elderly in Florida. Higher costs for personal care/waiver services were partially offset by savings in other Medicaid services, particularly those related to long-term care. During year 1, total Medicaid costs were generally higher for the treatment group than for the control group, with treatment-control cost differences ranging from 1 percent (and statistically insignificant) for the elderly in Florida to 17 percent for the elderly in Arkansas. In year 2, these cost differences were generally greater than in year 1. Only in Arkansas did the treatment-control difference in total cost shrink over time-to less than 5 percent (and statistically insignificant) in year 2. CONCLUSIONS: Medicaid costs were generally higher under Cash and Counseling because those in the traditional system did not get the services they were entitled to. Compared with the treatment group, (1) control group members were less likely to receive any services at all (despite being authorized for them), and (2) service recipients received a lower proportion of the amount of care that was authorized. In addition, a flaw in Florida's reassessment procedures led to treatment group members receiving more generous benefit amounts than control group members. To keep total Medicaid costs per recipient at the level incurred under the traditional system, consumer-directed programs need to be carefully designed and closely monitored.