Implementing parallel spreadsheet models for health policy decisions: The impact of unintentional errors on model projections.

BACKGROUND: Spreadsheet software is increasingly used to implement systems science models informing health policy decisions, both in academia and in practice where technical capacity may be limited. However, spreadsheet models are prone to unintentional errors that may not always be identified using standard error-checking techniques. Our objective was to illustrate, through a methodologic case study analysis, the impact of unintentional errors on model projections by implementing parallel model versions. METHODS: We leveraged a real-world need to revise an existing spreadsheet model designed to inform HIV policy. We developed three parallel versions of a previously validated spreadsheet-based model; versions differed by the spreadsheet cell-referencing approach (named single cells; column/row references; named matrices). For each version, we implemented three model revisions (re-entry into care; guideline-concordant treatment initiation; immediate treatment initiation). After standard error-checking, we identified unintentional errors by comparing model output across the three versions. Concordant model output across all versions was considered error-free. We calculated the impact of unintentional errors as the percentage difference in model projections between model versions with and without unintentional errors, using +/-5% difference to define a material error. RESULTS: We identified 58 original and 4,331 propagated unintentional errors across all model versions and revisions. Over 40% (24/58) of original unintentional errors occurred in the column/row reference model version; most (23/24) were due to incorrect cell references. Overall, >20% of model spreadsheet cells had material unintentional errors. When examining error impact along the HIV care continuum, the percentage difference between versions with and without unintentional errors ranged from +3% to +16% (named single cells), +26% to +76% (column/row reference), and 0% (named matrices). CONCLUSIONS: Standard error-checking techniques may not identify all errors in spreadsheet-based models. Comparing parallel model versions can aid in identifying unintentional errors and promoting reliable model projections, particularly when resources are limited.

Evaluating child welfare policies with decision-analytic simulation models.

The objective was to demonstrate decision-analytic modeling in support of Child Welfare policymakers considering implementing evidence-based interventions. Outcomes included permanency (e.g., adoptions) and stability (e.g., foster placement changes). Analyses of a randomized trial of KEEP-a foster parenting intervention-and NSCAW-1 estimated placement change rates and KEEP's effects. A microsimulation model generalized these findings to other Child Welfare systems. The model projected that KEEP could increase permanency and stability, identifying strategies targeting higher-risk children and geographical regions that achieve benefits efficiently. Decision-analytic models enable planners to gauge the value of potential implementations.

A simulation model investigating the impact of tumor volume doubling time and mammographic tumor detectability on screening outcomes in women aged 40-49 years.

BACKGROUND: Compared with women aged 50-69 years, the lower sensitivity of mammographic screening in women aged 40-49 years is largely attributed to the lower mammographic tumor detectability and faster tumor growth in the younger women. METHODS: We used a Monte Carlo simulation model of breast cancer screening by age to estimate the median tumor size detectable on a mammogram and the mean tumor volume doubling time. The estimates were calculated by calibrating the predicted breast cancer incidence rates to the actual rates from the Surveillance, Epidemiology, and End Results (SEER) database and the predicted distributions of screen-detected tumor sizes to the actual distributions obtained from the Breast Cancer Surveillance Consortium (BCSC). The calibrated parameters were used to estimate the relative impact of lower mammographic tumor detectability vs faster tumor volume doubling time on the poorer screening outcomes in younger women compared with older women. Mammography screening outcomes included sensitivity, mean tumor size at detection, lifetime gained, and breast cancer mortality. In addition, the relationship between screening sensitivity and breast cancer mortality was investigated as a function of tumor volume doubling time, mammographic tumor detectability, and screening interval. RESULTS: Lowered mammographic tumor detectability accounted for 79% and faster tumor volume doubling time accounted for 21% of the poorer sensitivity of mammography screening in younger women compared with older women. The relative contributions were similar when the impact of screening was evaluated in terms of mean tumor size at detection, lifetime gained, and breast cancer mortality. Screening sensitivity and breast cancer mortality reduction attributable to screening were almost linearly related when comparing annual or biennial screening with no screening. However, when comparing annual with biennial screening, the greatest reduction in breast cancer mortality attributable to screening did not correspond to the greatest gain in screening sensitivity and was more strongly affected by the mammographic tumor detectability than tumor volume doubling time. CONCLUSION: The age-specific differences in mammographic tumor detection contribute more than age-specific differences in tumor growth rates to the lowered performance of mammography screening in younger women.