Fairness in Predicting Cancer Mortality Across Racial Subgroups.

Importance: Machine learning has potential to transform cancer care by helping clinicians prioritize patients for serious illness conversations. However, models need to be evaluated for unequal performance across racial groups (ie, racial bias) so that existing racial disparities are not exacerbated. Objective: To evaluate whether racial bias exists in a predictive machine learning model that identifies 180-day cancer mortality risk among patients with solid malignant tumors. Design, Setting, and Participants: In this cohort study, a machine learning model to predict cancer mortality for patients aged 21 years or older diagnosed with cancer between January 2016 and December 2021 was developed with a random forest algorithm using retrospective data from the Mount Sinai Health System cancer registry, Social Security Death Index, and electronic health records up to the date when databases were accessed for cohort extraction (February 2022). Exposure: Race category. Main Outcomes and Measures: The primary outcomes were model discriminatory performance (area under the receiver operating characteristic curve [AUROC], F1 score) among each race category (Asian, Black, Native American, White, and other or unknown) and fairness metrics (equal opportunity, equalized odds, and disparate impact) among each pairwise comparison of race categories. True-positive rate ratios represented equal opportunity; both true-positive and false-positive rate ratios, equalized odds; and the percentage of predictive positive rate ratios, disparate impact. All metrics were estimated as a proportion or ratio, with variability captured through 95% CIs. The prespecified criterion for the model's clinical use was a threshold of at least 80% for fairness metrics across different racial groups to ensure the model's prediction would not be biased against any specific race. Results: The test validation dataset included 43 274 patients with balanced demographics. Mean (SD) age was 64.09 (14.26) years, with 49.6% older than 65 years. A total of 53.3% were female; 9.5%, Asian; 18.9%, Black; 0.1%, Native American; 52.2%, White; and 19.2%, other or unknown race; 0.1% had missing race data. A total of 88.9% of patients were alive, and 11.1% were dead. The AUROCs, F1 scores, and fairness metrics maintained reasonable concordance among the racial subgroups: the AUROCs ranged from 0.75 (95% CI, 0.72-0.78) for Asian patients and 0.75 (95% CI, 0.73-0.77) for Black patients to 0.77 (95% CI, 0.75-0.79) for patients with other or unknown race; F1 scores, from 0.32 (95% CI, 0.32-0.33) for White patients to 0.40 (95% CI, 0.39-0.42) for Black patients; equal opportunity ratios, from 0.96 (95% CI, 0.95-0.98) for Black patients compared with White patients to 1.02 (95% CI, 1.00-1.04) for Black patients compared with patients with other or unknown race; equalized odds ratios, from 0.87 (95% CI, 0.85-0.92) for Black patients compared with White patients to 1.16 (1.10-1.21) for Black patients compared with patients with other or unknown race; and disparate impact ratios, from 0.86 (95% CI, 0.82-0.89) for Black patients compared with White patients to 1.17 (95% CI, 1.12-1.22) for Black patients compared with patients with other or unknown race. Conclusions and Relevance: In this cohort study, the lack of significant variation in performance or fairness metrics indicated an absence of racial bias, suggesting that the model fairly identified cancer mortality risk across racial groups. It remains essential to consistently review the model's application in clinical settings to ensure equitable patient care.

Malnutrition risk assessment using a machine learning-based screening tool: A multicentre retrospective cohort.

BACKGROUND: Malnutrition is associated with increased morbidity, mortality, and healthcare costs. Early detection is important for timely intervention. This paper assesses the ability of a machine learning screening tool (MUST-Plus) implemented in registered dietitian (RD) workflow to identify malnourished patients early in the hospital stay and to improve the diagnosis and documentation rate of malnutrition. METHODS: This retrospective cohort study was conducted in a large, urban health system in New York City comprising six hospitals serving a diverse patient population. The study included all patients aged ≥ 18 years, who were not admitted for COVID-19 and had a length of stay of ≤ 30 days. RESULTS: Of the 7736 hospitalisations that met the inclusion criteria, 1947 (25.2%) were identified as being malnourished by MUST-Plus-assisted RD evaluations. The lag between admission and diagnosis improved with MUST-Plus implementation. The usability of the tool output by RDs exceeded 90%, showing good acceptance by users. When compared pre-/post-implementation, the rate of both diagnoses and documentation of malnutrition showed improvement. CONCLUSION: MUST-Plus, a machine learning-based screening tool, shows great promise as a malnutrition screening tool for hospitalised patients when used in conjunction with adequate RD staffing and training about the tool. It performed well across multiple measures and settings. Other health systems can use their electronic health record data to develop, test and implement similar machine learning-based processes to improve malnutrition screening and facilitate timely intervention.

Implementing a Machine Learning Screening Tool for Malnutrition: Insights From Qualitative Research Applicable to Other Machine Learning-Based Clinical Decision Support Systems.

BACKGROUND: Machine learning (ML)-based clinical decision support systems (CDSS) are popular in clinical practice settings but are often criticized for being limited in usability, interpretability, and effectiveness. Evaluating the implementation of ML-based CDSS is critical to ensure CDSS is acceptable and useful to clinicians and helps them deliver high-quality health care. Malnutrition is a common and underdiagnosed condition among hospital patients, which can have serious adverse impacts. Early identification and treatment of malnutrition are important. OBJECTIVE: This study aims to evaluate the implementation of an ML tool, Malnutrition Universal Screening Tool (MUST)-Plus, that predicts hospital patients at high risk for malnutrition and identify best implementation practices applicable to this and other ML-based CDSS. METHODS: We conducted a qualitative postimplementation evaluation using in-depth interviews with registered dietitians (RDs) who use MUST-Plus output in their everyday work. After coding the data, we mapped emergent themes onto select domains of the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework. RESULTS: We interviewed 17 of the 24 RDs approached (71%), representing 37% of those who use MUST-Plus output. Several themes emerged: (1) enhancements to the tool were made to improve accuracy and usability; (2) MUST-Plus helped identify patients that would not otherwise be seen; perceived usefulness was highest in the original site; (3) perceived accuracy varied by respondent and site; (4) RDs valued autonomy in prioritizing patients; (5) depth of tool understanding varied by hospital and level; (6) MUST-Plus was integrated into workflows and electronic health records; and (7) RDs expressed a desire to eventually have 1 automated screener. CONCLUSIONS: Our findings suggest that continuous involvement of stakeholders at new sites given staff turnover is vital to ensure buy-in. Qualitative research can help identify the potential bias of ML tools and should be widely used to ensure health equity. Ongoing collaboration among CDSS developers, data scientists, and clinical providers may help refine CDSS for optimal use and improve the acceptability of CDSS in the clinical context.

Factors associated with mammography use: A side-by-side comparison of results from two national surveys.

BACKGROUND: Mammography use is affected by multiple factors that may change as public health interventions are implemented. We examined two nationally representative, population-based surveys to seek consensus and identify inconsistencies in factors associated with mammography use in the entirety of the US population, and by black and white subgroups. METHODS: Self-reported mammography use in the past year was extracted for 12 639 and 169 116 women aged 40-74 years from the 2016 National Health Interview Survey (NHIS) and the 2016 Behavioral Risk Factor Surveillance System (BRFSS), respectively. We applied a random forest algorithm to identify the risk factors of mammography use and used a subset of them in multivariable survey logistic regressions to examine their associations with mammography use, reporting predictive margins and effect sizes. RESULTS: The weighted prevalence of past year mammography use was comparable across surveys: 54.31% overall, 54.50% in white, and 61.57% in black in NHIS and 53.24% overall, 56.97% in white, and 62.11% in black in BRFSS. Overall, mammography use was positively associated with black race, older age, higher income, and having health insurance, while negatively associated with having three or more children at home and residing in the Western region of the US. Overweight and moderate obesity were significantly associated with increased mammography use among black women (NHIS), while severe obesity was significantly associated with decreased mammography use among white women (BRFSS). CONCLUSION: We found higher mammography use among black women than white women, a change in the historical trend. We also identified high parity as a risk factor for mammography use, which suggests a potential subpopulation to target with interventions aimed at increasing mammography use.

Perioperative comparative effectiveness research.

The goal of comparative effectiveness research (CER) is to improve effectiveness, efficacy and efficiency in health care. While CER seems to present a major opportunity to introduce accountability into health care by identifying and promoting best practices in medicine, many issues surrounding CER remain poorly understood by clinicians and researchers, including what study designs are most appropriate for such research and what analytic tools are most helpful. The goal of this review is therefore to provide background and definitions of what constitutes CER and to discuss the various study designs and their strengths and weaknesses in achieving the stated goals of CER, while relating them to examples relevant to perioperative research. We provide a brief outline of the types of analytic methods particularly useful for CER and connect the reader to references for their practice. Finally, we assess the role of CER in perioperative research and provide some thoughts on future paths.

Demographics, outcomes, and risk factors for adverse events associated with primary and revision total hip arthroplasties in the United States.

We conducted a study to analyze nationally representative data on patient and health care system characteristics and in-hospital outcomes associated with primary and revision total hip arthroplasties in the United States. Between 1990 and 2004, there were an estimated 2,748,187 hospital discharges after total hip arthroplasty. The risk factors we identified for procedure-related complications and in-hospital mortality included revision procedures, increased age, and male sex. Compared with smaller hospital capacity (number of beds), large hospital capacity was associated with a decreased odds ratio for complications but an increased risk for in-hospital mortality. Additional studies are warranted to determine causal relationships.

Evaluation of the universal newborn hearing screening and intervention program.

During the last 20 years, the number of infants evaluated for permanent hearing loss at birth has increased dramatically with universal newborn hearing screening and intervention (UNHSI) programs operating in all US states and many territories. One of the most urgent challenges of UNHSI programs involves loss to follow-up among families whose infants screen positive for hearing loss. We surveyed 55 state and territorial UNHSI programs and conducted site visits with 8 state programs to evaluate progress in reaching program goals and to identify barriers to successful follow-up. We conclude that programs have made great strides in screening infants for hearing loss, but barriers to linking families of infants who do not pass the screening to further follow-up remain. We identified 4 areas in which there were barriers to follow-up (lack of service-system capacity, lack of provider knowledge, challenges to families in obtaining services, and information gaps), as well as successful strategies used by some states to address barriers within each of these areas. We also identified 5 key areas for future program improvements: (1) improving data systems to support surveillance and follow-up activities; (2) ensuring that all infants have a medical home; (3) building capacity beyond identified providers; (4) developing family support services; and (5) promoting the importance of early detection.

National trends in anterior cervical fusion procedures.

STUDY DESIGN: Population-based database analysis. OBJECTIVE: To analyze trends in patient- and healthcare-system-related characteristics, utilization and outcomes associated with anterior cervical spine fusions. SUMMARY OF BACKGROUND DATA: Anterior cervical decompression and spine fusion (ACDF) is one of the most commonly performed surgical procedures of the spine. However, few data analyzing trends in patient- and healthcare-system-related characteristics, utilization and outcomes exist. METHODS: Data from 1990 to 2004 collected in the National Hospital Discharge Survey were accessed. ACDF procedures were identified. Five-year periods of interest (POI) were created for temporal analysis and changes in the prevalence and utilization of this procedure as well as in patient- and healthcare-system-related variables were examined. The changes in the occurrence of procedure-related complications were evaluated. RESULTS: An estimated total of 771,932 discharges after ACDF were identified. Temporally, an almost 8-fold increase in total prevalence was accompanied by a similar increase in utilization (23/100.000 civilians/POI to 157/100.000/civilians/POI). The highest increase in utilization was observed in those > or =65 years (28-fold). Average age increased from 47.2 years to 50.5 years over time. Length of hospital stay decreased from 5.17 days to 2.38 days. Overall procedure-related complication rates decreased from 4.6% to 3.03%. The prevalence of hypertension, diabetes mellitus, hypercholesterolemia, obesity, pulmonary, and coronary artery increased over time among patients undergoing ACDF. CONCLUSION: Despite limitations inherent to secondary analysis of large databases, we identified a number of significant changes in the utilization, demographics, and outcomes associated with ACDF, which can be used to assess the effect of changes in medical care, direct health care resources, and future research. The effect of the increased prevalence of comorbidities on medical practice remains to be evaluated. Further studies are necessary to evaluate causal relationships.

Risk factors for perioperative mortality after lower extremity arthroplasty: a population-based study of 6,901,324 patient discharges.

The goal of this study was to provide nationally representative data on characteristics of patients who died after hip and knee arthroplasty and to determine risk factors for such outcome. Using national in-patient data collected between 1990 and 2004, we identified a cumulative in-hospital mortality rate of 0.35% among an estimated 6,901,324 procedures. The strongest independent risk factors for in-hospital mortality were pulmonary embolism and cerebrovascular complications, which increased the odds for a fatal outcome by approximately 40-fold. Preoperative risk factors for in-hospital mortality were revision total hip arthroplasty, advanced age, and the presence of a number of comorbid diseases, predominantly dementia, renal, and cerebrovascular disease. Our results can be used to identify patients at risk for fatal outcome and implement interventions to reduce such risk.

Risk factors for pulmonary embolism after hip and knee arthroplasty: a population-based study.

Pulmonary embolism (PE) is a cause of death after total hip and knee arthroplasty (THA, TKA). We characterised the patient population suffering from in-hospital PE and identified perioperative risk factors associated with PE using nationally representative data. Data from the National Hospital Discharge Survey between 1990 and 2004 on patients who underwent primary or revision THA/TKA in the United States were analysed. Multivariate regression analysis was performed to determine if perioperative factors were associated with increased risk of in-hospital PE. An estimated 6,901,324 procedures were identified. The incidence of in-hospital PE was 0.36%. Factors associated with an increased risk for the diagnosis of PE included: revision THA, female gender, dementia, obesity, renal and cerebrovascular disease. An increased association with PE was found among patients with diagnosis of Adult Respiratory Distress Syndrome (ARDS), psychosis (confusion), and peripheral thrombotic events. Our findings may be useful in stratifying the individual patient's risk of PE after surgery.

Trends in bilateral total knee arthroplasties: 153,259 discharges between 1990 and 2004.

UNLABELLED: Information regarding national trends in bilateral TKAs is needed for a rational allocation of resources, policy making, and research. Therefore, we analyzed data from the National Hospital Discharge Survey to elucidate temporal changes in the demographics, comorbidity profiles, hospital stay, and in-hospital complications of patients undergoing bilateral TKAs in the United States. We created three 5-year periods: 1990-1994, 1995-1999, 2000-2004. Procedure, healthcare system, and patient-related variables were analyzed for an estimated 153,259 discharges. Use of bilateral TKAs more than doubled for the entire civilian population and almost tripled among the female population, with the steepest increase seen during the last two study periods. A decline of nearly 50% in the use of bilateral TKAs in patients 85 years and older was seen between the second and third study periods. The prevalence of coronary artery disease and pulmonary disease increased from the first to the second study periods but decreased from the second to the third. The changes in the variables studied may reflect a recently acquired reluctance to perform bilateral TKAs in elderly patients with cardiopulmonary comorbidities. Additional studies are necessary to identify other causal relationships and define the impact of these changes on various aspects of the healthcare system. LEVEL OF EVIDENCE: Level II, prognostic study. See the Guidelines for Authors for a complete description of levels of evidence.

Trends in mortality, complications, and demographics for primary hip arthroplasty in the United States.

To determine trends in characteristics of total hip arthroplasty (THA) in the United States, the National Hospital Discharge Survey (NHDS) was analyzed from 1990 to 2004 for trends in in-hospital mortality and complications, length of hospital stay, demographics, and comorbidities. The number of THAs performed increased by 158%, whereas mortality rates remained low and slightly decreased (from 0.32% to 0.29%). Prevalence of procedure-related complications decreased over time, and length of stay decreased from an average of 8.7 days to 4.5 days. These improvements occurred despite an increase in comorbidities in patients. An increase in both the proportion of discharges to long- and short-term care facilities and in the proportion of procedures performed in smaller hospitals was noted. Multiple temporal changes in outcomes and demographics for THA were found. These changes have implications for clinical care and allocation of health resources.

Trends in demographics, comorbidity profiles, in-hospital complications and mortality associated with primary knee arthroplasty.

We analyzed the National Hospital Discharge Survey to elucidate temporal changes in the demographics, comorbidities, hospital stay, in-hospital complications, and mortality of patients undergoing primary total knee arthroplasties (TKAs) in the United States. Three 5-year periods were created (1990-1994, 1995-1999, and 2000-2004), and temporal changes were analyzed. The number of TKAs performed increased by 125% for the 3 periods. The increasing proportion of younger patients was accompanied by a concomitant decrease of Medicare-insured patients. Length of stay decreased from 8.44 to 4.18 days. An increase in the proportion of discharges to long-term and short-term care facilities and in procedures performed in small hospitals was noted. Although the prevalence of procedure-related complications decreased over time, comorbidities increased. Despite a decrease in mortality from the first to the second study period (0.50% vs 0.21%), a slight increase was noticed more recently (0.28%). We identified significant changes in most variables studied.

In-hospital complications and mortality of unilateral, bilateral, and revision TKA: based on an estimate of 4,159,661 discharges.

UNLABELLED: Patients undergoing bilateral total knee arthroplasty (BTKA) may have higher complication rates and mortality than those undergoing a unilateral procedure (UTKA). To evaluate this hypothesis, we analyzed nationally representative data collected for the National Hospital Discharge Survey on discharges after BTKA, UTKA, and revision TKA (RTKA) between 1990 and 2004. The demographics, comorbidities, in-hospital stay, complications, and mortality of each procedure were compared. An estimate of 4,159,661 discharges (153,259 BTKAs; 3,672,247 UTKAs; 334,155 RTKAs) were included. Patients undergoing BTKA were younger (1.5 years) and had a lower prevalence of comorbidities for hypertension (versus UTKA), diabetes, pulmonary disease, and coronary artery disease (versus UTKA and RTKA). The length of hospitalization was 5.8 days for BTKA, 5.3 for UTKA, and 5.4 for RTKA. Despite similar length of hospitalization, the prevalence of procedure-related complications was higher for BTKA (12.2%) compared with UTKA (8.2%) and RTKA (8.7%). In-hospital mortality was highest for patients undergoing BTKA (BTKA, 0.5%; UTKA, 0.3%; RTKA, 0.3%). Patients undergoing BTKA had a 1.6 times higher rate of procedure-related complications and mortality compared with those undergoing UTKA. Outcomes for patients undergoing RTKA for most variables were similar to those for UTKA. BTKA, advanced age, and male gender were independent risk factors for complications and mortality after TKA. LEVEL OF EVIDENCE: Level III, therapeutic study. See the Guidelines for Authors for a complete description of levels of evidence.

Best practices in implementing lifestyle interventions in the WISEWOMAN program: adaptable strategies for public health programs.

PURPOSE: Describe best practices for implementing a variety of lifestyle interventions targeting cardiovascular disease risk factors. APPROACH: A mixed-methods approach was used to collect and analyze data. The study was guided by the RE-AIM framework. SETTING: Selected Well-Integrated Screening and Intervention for Women Across the Nation (WISEWOMAN) projects funded by the Centers for Disease Control and Prevention. PARTICIPANTS: Five of the 15 currently operating WISEWOMAN projects were selected for study. Selection was based on availability of quantitative performance data, which were used to identify two high-performing and one low-performing sites within each project. METHOD: Qualitative data collection included a review of program materials; telephone interviews with federal, project, and local staff, and site visits. Site visits involved interviews with staff observations of the lifestyle intervention, and discussions with focus groups of participants. Analysis involved writing site reports, developing theme tables, identifying practices of interest, and applying an algorithm to identify best practices. RESULTS: Eighty-seven best practices were identified. We present a subset of 31 practices applicable to other public health programs and for which differences in how high- and low-performing sites used the practices were identified. DISCUSSION: Many of the best practices identified are applicable to broader audiences. Practitioners interested in strategies to recruit, engage, and retain participants and to facilitate behavior change can learn from these practices.

Tools for developing, implementing, and evaluating state policy.

Policy can improve health by initiating changes in physical, economic, and social environments. In contrast to interventions focused on individual people, policies have the potential to affect health across populations. For this reason, the Division for Heart Disease and Stroke Prevention of the Centers for Disease Control and Prevention (CDC) advises states funded under the Heart Disease and Stroke Prevention Program to engage in activities supporting the development and maintenance of policies that can help reduce the burden of cardiovascular disease. Currently, the Division for Heart Disease and Stroke Prevention funds programs in 33 states and the District of Columbia to promote cardiovascular health. One goal of these programs is to build states' capacity to develop, implement, track, and sustain population-based interventions that address heart disease and stroke. Because of the critical role of policy in these activities, CDC provides guidance in developing, implementing, and evaluating policy. In 2004, the division contracted with Mathematica Policy Research, Inc, to conduct the Heart Disease and Stroke Prevention Policy Project, which included development of an online database of state heart disease and stroke prevention policies and a mapping application to show which states have these policies. We discuss the method for developing the database, mapping application, and other tools to assist states in developing, implementing, and evaluating heart disease and stroke prevention policies. We also highlight lessons learned in developing these tools and ways that states can use the tools in their policy and program planning.

Trends in lung surgery: United States 1988 to 2002.

BACKGROUND: Reports on the temporal evolution in lung resection are limited. To elucidate temporal changes in the demographics of lung resections, we analyzed nationally representative data that were collected for the National Hospital Discharge Survey from 1988 to 2002. METHODS: Data collected between 1988 and 2002 were analyzed. Patients with International Classification of Diseases, ninth revision, clinical modification, procedure codes for lung resection were included in the sample. Three 5-year time periods were created (1988 to 1992, 1993 to 1997, and 1998 to 2002) to simplify the temporal analysis. Changes in the prevalence of procedures, age, gender, race, length of care, mortality, disposition status, and distribution by hospital size were evaluated. Trends in procedure-related complications were analyzed. RESULTS: Between 1988 and 2002, a total of 512,758 lung resections were performed. Comparing the earliest to the most recent time period, we found increases in the average age (61.1 years [range, 1 to 89 years] vs 63.2 years [range, 1 to 91 years], respectively), in the proportion of patients who were female (40.1% vs 49.6%, respectively), and in the proportion of Medicare/Medicaid patients (43.8% vs 49%/4.7% vs 6.7%, respectively). Decreases in the average length of stay (12.9 days [range, 1 to 358 days] vs 9.1 days [range, 1 to 175 days], respectively) and in the proportion of patients discharged to their primary residence (86% vs 79.5%, respectively) were seen. The proportion of patients who had undergone lobectomies compared to other types of lung resection increased. Mortality rates were 5% vs 5.4%, respectively, while the frequency of complications decreased. CONCLUSION: We identified temporal changes in lung resection surgery that may help in the construction of health-care policies to address the changing needs of and financial burdens on the health-care system.

Do race, gender, and source of payment impact on anesthetic technique for inguinal hernia repair?

STUDY OBJECTIVE: To evaluate the potential differences in the type of anesthesia provided to patients of different race, gender, and source of payment undergoing inguinal hernia repair (IHR). DESIGN: Retrospective cohort study. SETTING: Ambulatory surgical centers/National Survey of Ambulatory Surgery. PATIENTS: 5810 patients older than 14 years who underwent IHR in an ambulatory surgical center. INTERVENTIONS: Inguinal hernia repair under different types of anesthesia. MEASUREMENTS: The association of race, gender, and source of payment with different types of anesthesia for IHR as determined by multivariate regression analysis. RESULTS: Significant discrepancies in the use of various anesthetics between patients of different race, gender, and source of payment were found. Patients identified as black and those of other minority groups were significantly more likely to receive general anesthesia compared with those identified as white (odds ratio [OR] 2.76, confidence interval [CI] 1.96-3.88 and OR 1.66, CI 1.14-2.42, respectively). Those identified as black were less likely to receive epidural anesthesia compared with their white counterparts (OR 0.36, CI 0.14-0.95). Women were less likely than men to undergo IHR with epidural anesthesia (OR 0.5, 95% CI 0.3-0.85). CONCLUSION: Significant discrepancies in the use of various anesthetics for IHR between patients of different race, gender, and insurance status were found. Despite limitations inherent to secondary data analysis, the findings raise the possibility that nonmedical factors may influence anesthetic management.

Identifying best practices for WISEWOMAN programs using a mixed-methods evaluation.

INTRODUCTION: Recommendations on best practices typically are drawn from unique settings; these practices are challenging to implement in programs already in operation. We describe an evaluation that identifies best practices in implementing lifestyle interventions in the Center for Disease Control and Prevention's WISEWOMAN program and discuss our lessons learned in using the approach. METHODS: We used a mixed-methods evaluation that integrated quantitative and qualitative inquiry. Five state or tribal WISEWOMAN projects were included in the study. The projects were selected on the basis of availability of quantitative program performance data, which were used to identify two high-performing and one low-performing site within each project. We collected qualitative data through interviews, observation, and focus groups so we could understand the practices and strategies used to select and implement the interventions. Data were analyzed in a multistep process that included summarization, identification of themes and practices of interest, and application of an algorithm. RESULTS: Pilot testing data collection methods allowed for critical revisions. Conducting preliminary interviews allowed for more in-depth interviews while on site. Observing the lifestyle intervention being administered was key to understanding the program. Conducting focus groups with participants helped to validate information from other sources and offered a more complete picture of the program. CONCLUSION: Using a mixed-methods evaluation minimized the weaknesses inherent in each method and improved the completeness and quality of data collected. A mixed-methods evaluation permits triangulation of data and is a promising strategy for identifying best practices.