Methods for Building and Validating Equations for Physiology-Based Mathematical Models: Glucose Metabolism and Type 2 Diabetes in the Archimedes Model.

OBJECTIVES: Describe steps for deriving and validating equations for physiology processes for use in mathematical models. Illustrate the steps using glucose metabolism and Type 2 diabetes in the Archimedes model. METHODS AND RESULTS: The steps are as follows: identify relevant variables, describe their relationships, identify data sources that relate the variables, correct for biases in data sources, use curve fitting algorithms to estimate equations, validate the accuracy of curve fitting against empirical data, perform partially and fully independent external validations, examine any discrepancies to determine causes and make corrections, and periodically update and revalidate equations as necessary. Specific methods depend on the available data. Specific data sources and methods are illustrated for equations that represent the cause of Type 2 diabetes and its effect on fasting plasma glucose in the Archimedes model. Methods for validating the equations are illustrated. Applications enabled by including physiological equations in healthcare models are discussed. CONCLUSIONS: The methods can be used to derive equations that represent the relationships between physiological variables and the causes of diseases and that validate well against empirical data.

Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report.

The evaluation of the cost and health implications of agreeing to cover a new health technology is best accomplished using a model that mathematically combines inputs from various sources, together with assumptions about how these fit together and what might happen in reality. This need to make assumptions, the complexity of the resulting framework, the technical knowledge required, as well as funding by interested parties have led many decision makers to distrust the results of models. To assist stakeholders reviewing a model's report, questions pertaining to the credibility of a model were developed. Because credibility is insufficient, questions regarding relevance of the model results were also created. The questions are formulated such that they are readily answered and they are supplemented by helper questions that provide additional detail. Some responses indicate strongly that a model should not be used for decision making: these trigger a "fatal flaw" indicator. It is hoped that the use of this questionnaire, along with the three others in the series, will help disseminate what to look for in comparative effectiveness evidence, improve practices by researchers supplying these data, and ultimately facilitate their use by health care decision makers.

The 'Global Outcomes Score': a quality measure, based on health outcomes, that compares current care to a target level of care.

The quality of health care is measured today using performance measures that calculate the percentage of people whose health conditions are managed according to specified processes or who meet specified treatment goals. This approach has several limitations. For instance, each measure looks at a particular process, risk factor, or biomarker one by one, and each uses sharp thresholds for defining "success" versus "failure." We describe a new measure of quality called the Global Outcomes Score (GO Score), which represents the proportion of adverse outcomes expected to be prevented in a population under current levels of care compared to a target level of care, such as 100 percent performance on certain clinical guidelines. We illustrate the use of the GO Score to measure blood pressure and cholesterol care in a longitudinal study of people at risk of atherosclerotic diseases, or hardening of the arteries. In that population the baseline GO Score was 40 percent, which indicates that the care being delivered was 40 percent as effective in preventing myocardial infarctions and strokes as our target level of care. The GO Score can be used to assess the potential effectiveness of different interventions such as prevention activities, tests, and treatments.

A simulation shows limited savings from meeting quality targets under the Medicare Shared Savings Program.

The Medicare Shared Savings Program, created under the Affordable Care Act, will reward participating accountable care organizations that succeed in lowering health care costs while improving performance. Depending on how the organizations perform on several quality measures, they will "share savings" in Medicare Part A and B payments-that is, they will receive bonus payments for lowering costs. We used a simulation model to analyze the effects of the Shared Savings Program quality measures and performance targets on Medicare costs in a simulated population of patients ages 65-75 with type 2 diabetes. We found that a ten-percentage-point improvement in performance on diabetes quality measures would reduce Medicare costs only by up to about 1 percent. After the costs of performance improvement, such as additional tests or visits, are accounted for, the savings would decrease or become cost increases. To achieve greater savings, accountable care organizations will have to lower costs by other means, such as through improved use of information technology and care coordination.

Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--7.

Trust and confidence are critical to the success of health care models. There are two main methods for achieving this: transparency (people can see how the model is built) and validation (how well the model reproduces reality). This report describes recommendations for achieving transparency and validation developed by a taskforce appointed by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making. Recommendations were developed iteratively by the authors. A nontechnical description--including model type, intended applications, funding sources, structure, intended uses, inputs, outputs, other components that determine function, and their relationships, data sources, validation methods, results, and limitations--should be made available to anyone. Technical documentation, written in sufficient detail to enable a reader with necessary expertise to evaluate the model and potentially reproduce it, should be made available openly or under agreements that protect intellectual property, at the discretion of the modelers. Validation involves face validity (wherein experts evaluate model structure, data sources, assumptions, and results), verification or internal validity (check accuracy of coding), cross validity (comparison of results with other models analyzing the same problem), external validity (comparing model results with real-world results), and predictive validity (comparing model results with prospectively observed events). The last two are the strongest form of validation. Each section of this article contains a number of recommendations that were iterated among the authors, as well as among the wider modeling taskforce, jointly set up by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7.

Trust and confidence are critical to the success of health care models. There are two main methods for achieving this: transparency (people can see how the model is built) and validation (how well it reproduces reality). This report describes recommendations for achieving transparency and validation, developed by a task force appointed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM). Recommendations were developed iteratively by the authors. A nontechnical description should be made available to anyone-including model type and intended applications; funding sources; structure; inputs, outputs, other components that determine function, and their relationships; data sources; validation methods and results; and limitations. Technical documentation, written in sufficient detail to enable a reader with necessary expertise to evaluate the model and potentially reproduce it, should be made available openly or under agreements that protect intellectual property, at the discretion of the modelers. Validation involves face validity (wherein experts evaluate model structure, data sources, assumptions, and results), verification or internal validity (check accuracy of coding), cross validity (comparison of results with other models analyzing same problem), external validity (comparing model results to real-world results), and predictive validity (comparing model results with prospectively observed events). The last two are the strongest form of validation. Each section of this paper contains a number of recommendations that were iterated among the authors, as well as the wider modeling task force jointly set up by the International Society for Pharmacoeconomics and Outcomes Research and the Society for Medical Decision Making.

Individualized guidelines: the potential for increasing quality and reducing costs.

BACKGROUND: Current guidelines focus on a particular risk factor and specify criteria for categorizing persons into a small number of treatment groups. OBJECTIVE: To compare current guidelines with individualized guidelines (that use readily available characteristics from each person to calculate the risk reduction expected from treatment and to identify persons for treatment in ranked order of decreasing expected benefit), in the context of blood pressure management. DESIGN: Analysis of person-specific, longitudinal data. SETTING: The ARIC (Atherosclerosis Risk in Communities) Study. PARTICIPANTS: Persons aged 45 to 64 years without preexisting cardiovascular disease who currently do not receive antihypertensive treatment. INTERVENTION: Treatment according to the criteria of the Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC 7 guidelines); individualized guidelines, or treatment in decreasing order of expected benefit; and random care, or treatment of persons selected at random. MEASUREMENTS: Number of myocardial infarctions (MIs) and strokes and medical costs. RESULTS: Compared with treating people according to random care, individualized guidelines could prevent the same number of MIs and strokes as JNC 7 guidelines at savings that are 67% greater than using JNC 7 guidelines, or it could prevent 43% more MIs and strokes for the same cost as treatment according to JNC 7 guidelines. The superiority of individualized guidelines was not sensitive to a wide range of assumptions about costs, treatment effectiveness, level of risk for cardiovascular disease in the population, or effects on workflow. The degree of superiority was sensitive to the accuracy of the method used to rank patients and to its span (the proportion of the population for whom all of the outcomes of interest can be calculated). LIMITATIONS: Specific results apply to the effects of blood pressure management on MI and stroke in the ARIC Study population. The methods for calculating individual benefits require quantitative evidence about the relationships among risk factors, long-term outcomes, and treatment effects. CONCLUSION: Use of individualized guidelines can help to increase the quality and reduce the cost of care.

Effect of smoking cessation advice on cardiovascular disease.

Performance measures and guidelines encourage physicians to advise smokers to quit. The effect of these efforts on the morbidity, mortality, and cost of cardiovascular disease is not known. This article analyzes the effects of offering smoking cessation advice in the US population. The Archimedes model is used to simulate several clinical trials in which basic advice and medication advice are offered and to calculate the rates of myocardial infarctions, congestive heart disease deaths, strokes, life years, quality-adjusted life years (QALYs), costs, and cost/ QALY. The simulated population is a representative sample of the US population drawn from the Third National Health and Nutrition Survey conducted just before the performance measures and guidelines were introduced. The results show that offering basic advice and medication advice can prevent about 13% and 19% of myocardial infarctions and strokes, respectively. The 30-year cost/QALY is approximately $3000 less than the base-case assumptions and less than $10 000 under pessimistic assumptions.

The potential effects of HEDIS performance measures on the quality of care.

We analyzed the potential effects of different levels of performance on eight Health Care Employer Data and Information Set (HEDIS) measures for cardiovascular disease and diabetes during 1995-2005. The measures targeted 3.3 million (25 percent) heart attacks. Improvements in performance to those achieved by the median plan in 2005 imply prevention of 1.9 million myocardial infarctions (MIs, 15 percent), 0.8 million strokes (8 percent), and 0.1 million cases of end-stage renal disease (17 percent). If performance had been 100 percent, 1.4 million more MIs would have been prevented. Control of blood pressure has the largest potential effect on quality at the national level.

Diabetes Risk Calculator: a simple tool for detecting undiagnosed diabetes and pre-diabetes.

OBJECTIVE: The objective of this study was to develop a simple tool for the U.S. population to calculate the probability that an individual has either undiagnosed diabetes or pre-diabetes. RESEARCH DESIGN AND METHODS: We used data from the Third National Health and Nutrition Examination Survey (NHANES) and two methods (logistic regression and classification tree analysis) to build two models. We selected the classification tree model on the basis of its equivalent accuracy but greater ease of use. RESULTS: The resulting tool, called the Diabetes Risk Calculator, includes questions on age, waist circumference, gestational diabetes, height, race/ethnicity, hypertension, family history, and exercise. Each terminal node specifies an individual's probability of pre-diabetes or of undiagnosed diabetes. Terminal nodes can also be used categorically to designate an individual as having a high risk for 1) undiagnosed diabetes or pre-diabetes, 2) pre-diabetes, or 3) neither undiagnosed diabetes or pre-diabetes. With these classifications, the sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic area for detecting undiagnosed diabetes are 88%, 75%, 14%, 99.3%, and 0.85, respectively. For pre-diabetes or undiagnosed diabetes, the results are 75%, 65%, 49%, 85%, and 0.75, respectively. We validated the tool using v-fold cross-validation and performed an independent validation against NHANES 1999-2004 data. CONCLUSIONS: The Diabetes Risk Calculator is the only currently available noninvasive screening tool designed and validated to detect both pre-diabetes and undiagnosed diabetes in the U.S. population.

Linking electronic medical records to large-scale simulation models: can we put rapid learning on turbo?

One method for rapid learning is to use data from electronic medical records (EMRs) to help build and validate large-scale, physiology-based simulation models. These models can than be used to help answer questions that cannot be addressed directly from the EMR data. Their potential uses include analyses of physiological pathways; simulation and design of clinical trials; and analyses of clinical management tools such as guidelines, performance measures, priority setting, and cost-effectiveness. Linking the models to EMR data also facilitates tailoring analyses to specific populations. The models' power and accuracy can be improved by linkage to comprehensive, person-specific, longitudinal data from EMRs.

Accuracy versus transparency in pharmacoeconomic modelling: finding the right balance.

As modellers push to make their models more accurate, the ability of others to understand the models can decrease, causing the models to lose transparency. When this type of conflict between accuracy and transparency occurs, the question arises, "Where do we want to operate on that spectrum?" This paper argues that in such cases we should give absolute priority to accuracy: push for whatever degree of accuracy is needed to answer the question being asked, try to maximise transparency within that constraint, and find other ways to replace what we wanted to get from transparency. There are several reasons. The fundamental purpose of a model is to help us get the right answer to a question and, by any measure, the expected value of a model is proportional to its accuracy. Ironically, we use transparency as a way to judge accuracy. But transparency is not a very powerful or useful way to do this. It rarely enables us to actually replicate the model's results and, even if we could, replication would not tell us the model's accuracy. Transparency rarely provides even face validity; from the content expert's perspective, the simplifications that modellers have to make usually raise more questions than they answer. Transparency does enable modellers to alert users to weaknesses in their models, but that can be achieved simply by listing the model's limitations and does not get us any closer to real accuracy. Sensitivity analysis tests the importance of uncertainty about the variables in a model, but does not tell us about the variables that were omitted or the structure of the model. What people really want to know is whether a model actually works. Transparency by itself can't answer this; only demonstrations that the model accurately calculates or predicts real events can. Rigorous simulations of clinical trials are a good place to start. This is the type of empirical validation we need to provide if the potential of mathematical models in pharmacoeconomics is to be fully achieved.

Clinical outcomes and cost-effectiveness of strategies for managing people at high risk for diabetes.

BACKGROUND: Lifestyle modification can forestall diabetes in high-risk people, but the long-term cost-effectiveness is uncertain. OBJECTIVE: To estimate the effects of the lifestyle modification program used in the Diabetes Prevention Program (DPP) on health and economic outcomes. DESIGN: Cost-effectiveness analysis using the Archimedes model. DATA SOURCES: Published basic and epidemiologic studies, clinical trials, and Kaiser Permanente administrative data. TARGET POPULATION: Adults at high risk for diabetes (body mass index >24 kg/m2, fasting plasma glucose level of 5.2725 to 6.9375 mmol/L [95 to 125 mg/dL], 2-hour glucose tolerance test result of 7.77 to 11.0445 mmol/L [140 to 199 mg/dL]). TIME HORIZON: 5 to 30 years. PERSPECTIVE: Patient, health plan, and societal. INTERVENTIONS: No prevention, DPP's lifestyle modification program, lifestyle modification begun after a person develops diabetes, and metformin. MEASUREMENTS: Diagnosis and complications of diabetes. RESULTS OF BASE-CASE ANALYSIS: Compared with no prevention program, the DPP lifestyle program would reduce a high-risk person's 30-year chances of getting diabetes from about 72% to 61%, the chances of a serious complication from about 38% to 30%, and the chances of dying of a complication of diabetes from about 13.5% to 11.2%. Metformin would deliver about one third the long-term health benefits achievable by immediate lifestyle modification. Compared with not implementing any prevention program, the expected 30-year cost/quality-adjusted life-year (QALY) of the DPP lifestyle intervention from the health plan's perspective would be about 143,000 dollars. From a societal perspective, the cost/QALY of the lifestyle intervention compared with doing nothing would be about 62,600 dollars. Either using metformin or delaying the lifestyle intervention until after a person develops diabetes would be more cost-effective, costing about 35,400 dollars or 24,500 dollars per QALY gained, respectively, compared with no program. Compared with delaying the lifestyle program until after diabetes is diagnosed, the marginal cost-effectiveness of beginning the DPP lifestyle program immediately would be about 201,800 dollars. RESULTS OF SENSITIVITY ANALYSIS: Variability and uncertainty deriving from the structure of the model were tested by comparing the model's results with the results of real clinical trials of diabetes and its complications. The most critical element of uncertainty is the effectiveness of the lifestyle program, as expressed by the 95% CI of the DPP study. The most important potentially controllable factor is the cost of the lifestyle program. Compared with no program, lifestyle modification for high-risk people can be made cost-saving over 30 years if the annual cost of the intervention can be reduced to about 100 dollars. LIMITATIONS: Results depend on the accuracy of the model. CONCLUSIONS: Lifestyle modification is likely to have important effects on the morbidity and mortality of diabetes and should be recommended to all high-risk people. The program used in the DPP study may be too expensive for health plans or a national program to implement. Less expensive methods are needed to achieve the degree of weight loss seen in the DPP.

Evidence-based medicine: a unified approach.

Behind the wide acceptance of the idea of "evidence-based medicine" are two curious facts: There are two very different approaches to applying evidence to medicine, and the most commonly cited definition applies to only one of them. This paper describes the problem that we are asking "evidence" to solve and the different methods by which evidence can be used to help solve that problem, and recommends a unified approach.

Archimedes: a trial-validated model of diabetes.

OBJECTIVE: To build a mathematical model of the anatomy, pathophysiology, tests, treatments, and outcomes pertaining to diabetes that could be applied to a wide variety of clinical and administrative problems and that could be validated. RESEARCH DESIGN AND METHODS: We used an object-oriented approach, differential equations, and a construct we call "features." The level of detail and realism was determined by what clinicians considered important, by the need to distinguish clinically relevant variables, and by the level of detail used in the conduct of clinical trials. RESULTS: The model includes the pertinent organ systems, more than 50 continuously interacting biological variables, and the major symptoms, tests, treatments, and outcomes. The level of detail corresponds to that found in general medical textbooks, patient charts, clinical practice guidelines, and designs of clinical trials. The model is continuous in time and represents biological variables continuously. As demonstrated in a companion article, the equations can simulate a variety of clinical trials and reproduce their results with good accuracy. CONCLUSIONS: It is possible to build a mathematical model that replicates the pathophysiology of diabetes at a high level of biological and clinical detail and that can be tested by simulating clinical trials.

Validation of the archimedes diabetes model.

OBJECTIVE: To validate the Archimedes model of diabetes and its complications for a variety of populations, organ systems, treatments, and outcomes. RESEARCH DESIGN AND METHODS: We simulated a variety of randomized controlled trials by repeating in the model the steps taken for the real trials and comparing the results calculated by the model with the results of the trial. Eighteen trials were chosen by an independent advisory committee. Half the trials had been used to help build the model ("internal" or "dependent" validations); the other half had not. Those trials comprise "external" or "independent" validations. RESULTS: A total of 74 validation exercises were conducted involving different treatments and outcomes in the 18 trials. For 71 of the 74 exercises there were no statistically significant differences between the results calculated by the model and the results observed in the trial. Considering only the trials that were never used to help build the model-the independent or external validations-the correlation was r = 0.99. Including all of the exercises, the correlation between the outcomes calculated by the model and the outcomes seen in the trials was r = 0.99. When the absolute differences in outcomes between the control and treatment groups were compared, the correlation coefficient was r = 0.97. CONCLUSIONS: The Archimedes diabetes model is a realistic representation of the anatomy, pathophysiology, treatments, and outcomes pertinent to diabetes and its complications for applications that involve the populations, treatments, outcomes, and health care settings spanned by the trials.