Sensitivity and specificity of the RAND/UCLA Appropriateness Method to identify the overuse and underuse of coronary revascularization and hysterectomy.

There is no empirical evidence on the sensitivity and specificity of methods to identify the possible overuse and underuse of medical procedures. To estimate the sensitivity and specificity of the RAND/UCLA Appropriateness Method. Parallel three-way replication of the RAND/UCLA Appropriateness Method for each of two procedures, coronary revascularization and hysterectomy. Maximum likelihood estimates of the sensitivity and specificity of the method for each procedure. These values were then used to re-calculate past estimates of overuse and underuse, correcting for the error rate in the appropriateness method. The sensitivity of detecting overuse of coronary revascularization was 68% (95% confidence interval 60-76%) and the specificity was 99% (98-100%). The corresponding values for hysterectomy were 89% (85-94%) and 86% (83-89%). The sensitivity and specificity of detecting the underuse of coronary revascularization were 94% (92-95%) and 97% (96-98%), respectively. Past applications of the appropriateness method have overestimated the prevalence of the overuse of hysterectomy, underestimated the prevalence of the overuse of the coronary revascularization, and provided true estimates of the underuse of revascularization. The sensitivity and specificity of the RAND/UCLA Appropriateness Method vary according to the procedure assessed and appear to estimate the underuse of procedures more accurately than their overuse.

The Vulnerable Elders Survey: a tool for identifying vulnerable older people in the community.

OBJECTIVES: To develop a simple method for identifying community-dwelling vulnerable older people, defined as persons age 65 and older at increased risk of death or functional decline. To assess whether self-reported diagnoses and conditions add predictive ability to a function-based survey. DESIGN: Analysis of longitudinal survey data. SETTING: A nationally representative community-based survey. PARTICIPANTS: Six thousand two hundred five Medicare beneficiaries age 65 and older. MEASUREMENTS: Bivariate and multivariate analyses of the Medicare Current Beneficiary Survey; development and comparison of scoring systems that use age, function, and self-reported diagnoses to predict future death and functional decline. RESULTS: A multivariate model using function, self-rated health, and age to predict death or functional decline was only slightly improved when self-reported diagnoses and conditions were included as predictors and was significantly better than a model using age plus self-reported diagnoses alone. These analyses provide the basis for a 13-item function-based scoring system that considers age, self-rated health, limitation in physical function, and functional disabilities. A score of >or=3 targeted 32% of this nationally representative sample as vulnerable. This targeted group had 4.2 times the risk of death or functional decline over a 2-year period compared with those with scores <3. The receiver operating characteristics curve had an area of.78. An alternative scoring system that included self-reported diagnoses did not substantially improve predictive ability when compared with a function-based scoring system. CONCLUSIONS: A function-based targeting system effectively and efficiently identifies older people at risk of functional decline and death. Self-reported diagnoses and conditions, when added to the system, do not enhance predictive ability. The function-based targeting system relies on self-report and is easily transported across care settings.

Development of subscales from the symptoms/problems and effects of kidney disease scales of the kidney disease quality of life instrument.

BACKGROUND: The Kidney Disease Quality of Life Instrument (KDQOL) was developed to provide clinicians with a comprehensive assessment of the important domains of health-related quality of life (HRQOL) for patients with end-stage renal disease who are undergoing hemodialysis. OBJECTIVE: The purpose of this study was to develop subscales from the 55 items comprising the Symptoms/Problems and Effects of Kidney Disease scales of the KDQOL and to measure the internal consistency reliability of these subscales. METHODS: The 55 items from the Symptoms/Problems and Effects of Kidney Disease scales were arranged into substantively meaningful clusters using an affinity mapping procedure. The resulting subscales were assessed for internal consistency reliability using data from a sample of 165 individuals with kidney disease who had completed the KDQOL. RESULTS: Eleven multi-item subscales were identified: pain, psychological dependency, cognitive functioning, social functioning, dialysis-related symptoms, cardiopulmonary symptoms, sleep, energy, cramps, diet, and appetite. Four items (clotting or other problems with access site, high blood pressure, numbness in hands or feet, and blurred vision) were not included in any of these subscales. Internal consistency reliability estimates for the 11 subscales ranged from 0.66 to 0.92. These subscales correlated with the scales from the 36-Item Short-Form Health Survey as hypothesized (ie, corresponding pain, energy, and social functioning scales had the highest correlations). In addition, several subscales were significantly associated, as hypothesized, with other variables such as the number of disability days. CONCLUSIONS: The results of this study further support the reliability and validity of the KDQOL. The 11 subscales identified yield more detailed information on the HRQOL of patients with kidney disease and provide a basis for specific improvements in the quality of care delivered to these patients.

Selecting target conditions for quality of care improvement in vulnerable older adults.

OBJECTIVE: To identify a set of geriatric conditions as optimal targets for quality improvement to be used in a quality measurement system for vulnerable older adults. DESIGN: Discussion and two rounds of ranking of conditions by a panel of geriatric clinical experts informed by literature reviews. METHODS: A list of 78 conditions common among vulnerable older people was reduced to 35 on the basis of their (1) prevalence, (2) impact on health and quality of life, (3) effectiveness of interventions in improving mortality and quality of life, (4) disparity in the quality of care across providers and geographic areas, and (5) feasibility of obtaining the data needed to test compliance with quality indicators. A panel of 12 experts in geriatric care discussed and then ranked the 35 conditions on the basis of the same five criteria. We then selected 21 conditions, based on panelists' iterative rankings. Using available national data, we compiled information about prevalence of the selected conditions for community-dwelling older people and older nursing home residents and estimated the proportion of inpatient and outpatient care attributable to the selected conditions. RESULTS: The 21 conditions selected as targets for quality improvement among vulnerable older adults include (in rank order): pharmacologic management; depression; dementia; heart failure; stroke (and atrial fibrillation); hospitalization and surgery; falls and mobility disorders; diabetes mellitus; end-of-life care; ischemic heart disease; hypertension; pressure ulcers; osteoporosis; urinary incontinence; pain management; preventive services; hearing impairment; pneumonia and influenza; vision impairment; malnutrition; and osteoarthritis. The selected conditions had mean rank scores from 1.2 to 3.8, and those excluded from 4.6 to 6.9, on a scale from 1 (highest ranking) to 7 (lowest ranking). Prevalence of the selected conditions ranges from 10 to 50% among community-dwelling older adults and from 25 to 80% in nursing home residents for the six most common selected conditions. The 21 target conditions account for at least 43% of all acute hospital discharges and 33% of physician office visits among persons 65 years of age and older. Actual figures must be higher because several of the selected conditions (e.g., end-of-life care) are not recorded as diagnoses. CONCLUSIONS: Twenty-one conditions were selected as targets for quality improvement in vulnerable older people for use in a quality measurement system. The 21 geriatric conditions selected are highly prevalent in this group and likely account for more than half of the care provided to this group in hospital and ambulatory settings.

The reproducibility of a method to identify the overuse and underuse of medical procedures.

BACKGROUND: To assess the overuse and underuse of medical procedures, various methods have been developed, but their reproducibility has not been evaluated. This study estimates the reproducibility of one commonly used method. METHODS: We performed a parallel, three-way replication of the RAND-University of California at Los Angeles appropriateness method as applied to two medical procedures, coronary revascularization and hysterectomy. Three nine-member multidisciplinary panels of experts were composed for each procedure by stratified random sampling from a list of experts nominated by the relevant specialty societies. Each panel independently rated the same set of clinical scenarios in terms of the appropriateness of the relevant procedure on a risk-benefit scale ranging from 1 to 9. Final ratings were used to classify the procedure in each scenario as necessary or not necessary (to evaluate underuse) and inappropriate or not inappropriate (to evaluate overuse). Reproducibility was measured by overall agreement and by the kappa statistic. The criteria for underuse and overuse derived from these ratings were then applied to real populations of patients who had undergone coronary revascularization or hysterectomy. RESULTS: The rates of agreement among the three coronary-revascularization panels were 95, 94, and 96 percent for inappropriate-use scenarios and 93, 92, and 92 percent for necessary-use scenarios. Agreement among the three hysterectomy panels was 88, 70, and 74 percent for inappropriate-use scenarios. Scenarios involving necessary use of hysterectomy were not assessed. The three-way kappa statistic to detect overuse was 0.52 for coronary revascularization and 0.51 for hysterectomy. The three-way kappa statistic to detect underuse of coronary revascularization was 0.83. Application of individual panels' criteria to real populations of patients resulted in a 100 percent variation in the proportion of cases classified as inappropriate and a 20 percent variation in the proportion of cases classified as necessary. CONCLUSIONS: The appropriateness method is far from perfect. Appropriateness criteria may be useful in comparing levels of appropriate procedures among populations but should not by themselves be used to direct care for individual patients.

Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures.

The authors compare the appropriateness ratings and mutual influence of panelists from different specialties rating a comprehensive set of indications for six surgical procedures. Nine-member panels rated each procedure: abdominal aortic aneurysm surgery, carotid endarterectomy, cataract surgery, coronary angiography, and coronary artery bypass graft surgery/percutaneous transluminal coronary angioplasty (common panel). Panelists individually rated the appropriateness of indications at home and then discussed and re-rated the indications during a 2-day meeting. Subsequently, they rated the necessity of those indications scored by the group as appropriate. There were 45 panelists, including specialists (either performers of the procedure or members of a related specialty) and primary care providers, all drawn from nominations by their respective specialty societies. Main outcome measures included: individual panelists' mean ratings over all indications, mean change and conformity scores between rounds of ratings, and the percentage of audited actual procedures rated appropriate or necessary. Performers had the highest mean ratings, followed by physicians in related specialties, trailed by primary care providers. One fifth of all actual procedures were for indications rated appropriate by performers and less than appropriate by primary care providers. At the panel meetings, primary care providers and related specialists showed no greater tendency to be influenced by other panelists than did performers. Multispecialty panels provide more divergent viewpoints than panels composed entirely of performers. This divergence means that fewer actual procedures are deemed performed for appropriate or necessary indications.

Phase II of the AHCPR-sponsored heart failure guideline: translating practice recommendations into review criteria.

BACKGROUND: In 1992, under the sponsorship of the U.S. Agency for Health Care Policy and Research, RAND assembled an expert panel to develop the Heart Failure Clinical Practice Guideline. Phase II of the effort was intended to identify which of the guideline's recommendations the panel felt were suitable for use in retrospective utilization review and quality assessment programs and to develop review criteria, performance measures, and standards of quality for use in monitoring compliance with those recommendations. SELECTION OF RECOMMENDATIONS: Selecting guideline recommendations for translation into review criteria and ultimately into standards of care was a multistep process comprising (1) identification of 34 recommendations from the guideline, (2) rating them on the basis of importance to quality of care and feasibility of monitoring, (3) review by a subcommittee and the full guideline panel, (4) translation into review criteria, and (5) further review and input by panelists and peer and pilot reviewers. Finally, standards of care (the minimum proportion of cases expected to be in accordance with guideline recommendations) were determined to be 90%-95% for six of the final criteria and 75%-80% for the other two. CONCLUSION: Despite some reservations, physicians and other health care professionals agreed to be held accountable for following a core set of guideline recommendations for the treatment of heart failure. Substantial progress was made in identifying recommendations that panelists and reviewers were willing to endorse in utilization review activities, including adoption of improved documentation standards. The review criteria's major impact may be the knowledge that the criteria are in place and that care is being monitored based on those standards.

Measuring the necessity of medical procedures.

This is a report on the extension of the concept of the appropriateness of a procedure to the necessity, or crucial importance, of that procedure. To state that a procedure is crucial means that withholding the procedure would be deleterious to the patient's health. Appropriateness and necessity ratings for six procedures were obtained using a modified Delphi panel process developed in earlier work. Panels were composed of practicing clinicians who were recognized leaders in their fields. The panels included both performers and nonperformers of the procedure under discussion. For most procedures and panelists, necessity was related to appropriateness, but was distinct from it. The proportion of indications for which the procedure was crucial varied in clinically consistent ways both among and within procedures. However, panelists did not achieve a consensus on necessity. Further research is suggested to refine the method to promote consensus and to validate further the ratings of necessity. In conclusion, necessity ratings can be used together with appropriateness ratings to address not only the overuse of procedures, but also to indicate limited access to care through underuse of procedures.

Appropriateness of the use of cardiovascular procedures: a method and results of this application.

Resources available to provide adequate health care in western countries must compete with other priorities such as education and the environment. At the same time the allocation of health resources often does not correspond to the real needs of patients. We have developed a method that combines what is known in the literature with expert physician judgment to generate clinically valid appropriateness guidelines. The method involves a modified Delphi approach, including a detailed literature review, consultations with experts, and three rounds of panel ratings. Clinical scenarios, or indications, are rated appropriate, uncertain, or inappropriate. Appropriate means that the procedure is worth doing for the given indication if the health benefit exceeds the health risk. We have conducted panels relating to coronary procedures in four countries. Application of the indications to individual cases has demonstrated that the amount of inappropriate care is too large to be ignored. Dissemination of appropriateness results might take the form of public disclosure or as part of the physician/patient exchange to improve performance. Indications for which a procedure is frequently performed and which are rated uncertain should be considered to be the focus of controlled clinical trials.

The appropriateness of use of coronary artery bypass graft surgery in New York State.

OBJECTIVE: To determine the appropriateness of use of coronary artery bypass graft surgery in New York State. DESIGN: Retrospective randomized medical record review. SETTING: Fifteen randomly selected hospitals in New York State that provide coronary artery bypass graft surgery. PATIENTS: Random sample of 1338 patients undergoing isolated coronary artery bypass graft surgery in New York State in 1990. MAIN OUTCOME MEASURES: Percentage of patients who had bypass surgery for appropriate, inappropriate, or uncertain indications; operative (30-day) mortality; and complications. RESULTS: Nearly 91% of the bypass operations were rated appropriate; 7%, uncertain; and 2.4%, inappropriate. This low inappropriate rate differs substantially from the 14% rate found in a previous study of patients operated on in 1979, 1980, and 1982. The difference in rates was not due to more lenient criteria but to changes in practice, the most important being that the fraction of patients receiving coronary artery bypass grafts for one- and two-vessel disease fell from 51% to 24%. Individual hospital rates of inappropriateness (0% to 5%) did not vary significantly. Rates of appropriateness also did not vary by hospital location, volume, or teaching status. Operative mortality was 2.0%; 17% of patients suffered a complication. Complication rates varied significantly among hospitals (P < .01) and were higher in downstate hospitals. CONCLUSIONS: The rates of inappropriate and uncertain use of coronary artery bypass graft surgery in New York State were very low. Rates of inappropriate use did not vary significantly among hospitals, or according to region, volume of bypass operations performed, or teaching status.

The appropriateness of use of percutaneous transluminal coronary angioplasty in New York State.

OBJECTIVE: To determine the appropriateness of use of percutaneous transluminal coronary angioplasty (PTCA) in New York State. DESIGN: Retrospective randomized medical record. SETTING: Fifteen randomly selected hospitals in New York State that provide PTCA. PATIENTS: Random sample of 1306 patients undergoing PTCA in New York State in 1990. MAIN OUTCOME MEASURES: Percentage of patients who underwent PTCA for indications rated appropriate, uncertain, and inappropriate. RESULTS: The majority of patients received PTCA for chronic stable angina, unstable angina, and in the post-myocardial infarction period (up to 3 weeks). Fifty-eight percent of PTCAs were rated appropriate; 38%, uncertain; and 4%, inappropriate. The inappropriate rate varied by hospital from 1% to 9% (P = .12); the uncertain rate, from 26% to 50% (P = .02); and the combined inappropriate and uncertain rate, from 29% to 57% (P < .001). There was no difference in appropriateness when the institutions were grouped by volume (fewer than 300 procedures annually or at least 300 procedures annually), location (upstate vs downstate), or by teaching status. CONCLUSIONS: Few PTCAs were performed for inappropriate indications in New York State. However, the large number of procedures performed for indications that were rated uncertain as to their net benefit requires further study and justification at both clinical and policy levels.

The appropriateness of use of coronary angiography in New York State.

OBJECTIVE: To determine the appropriateness of use of coronary angiography in New York State. DESIGN: Retrospective randomized medical record review. SETTING: Fifteen randomly selected hospitals in New York State that provide coronary angiography. PATIENTS: Random sample of 1335 patients undergoing coronary angiography in New York State in 1990. MAIN OUTCOME MEASURES: Percentage of patients who underwent coronary angiography for appropriate, uncertain, or inappropriate indications. RESULTS: Approximately 76% of coronary angiographies were rated appropriate; 20%, uncertain; and 4%, inappropriate. Inappropriate use did not vary significantly between the elderly (ie, patients aged 65 years and older) and nonelderly, 4.7% and 3.9%, respectively. Although the rate of inappropriate use varied from 0% to 9% among hospitals, the difference was not significant. Rates of appropriateness did not vary by hospital location (upstate vs downstate), volume (fewer than 750 procedures annually or at least 750 procedures annually), teaching status, or whether revascularization was available at the hospital where angiography was performed. CONCLUSIONS: Although coronary angiography was used for few inappropriate indications in New York State, many procedures were performed for uncertain indications in which the benefit and risk were approximately equal or unknown.

Quality of ambulatory care. Epidemiology and comparison by insurance status and income.

In this report the data from medical history questionnaires, screening examinations, insurance claims, and a face-to-face physician interview were used to examine the quality of ambulatory care received for 17 chronic conditions by a general population of 5986 adults (less than or equal to 65) and children (less than or equal to 14) enrolled in the RAND Health Insurance Experiment. Subjects in six U.S. sites were randomly assigned to insurance plans that were free or that required cost sharing, or in one site to an HMO. Quality-of-care criteria--both process (what was done to patients) and outcome (what happened to them)--were developed. Overall, 81% of outcome criteria and 62% of process criteria were met. Physicians interviewed patients with selected conditions at the Experiment's end to evaluate care. They suggested that approximately 70% of patients should have their current therapy changed, but only 30% of patients would obtain more than minor improvement from such a change. Clinically meaningful plan differences in quality of care were observed only for the process criteria dealing with the need for a visit (free plan compliance 59%; cost sharing compliance 52%). Quality of care for the poor was slightly worse than for the nonpoor and persons randomized to an HMO had slightly better overall quality of care than those in the fee-for-service system. Substantial improvements in the quality of the process of care could be made, but impact on outcome may be small. Results of the analysis suggest the need for development of clinical models to test the relationship between specific process criteria and improvements in outcome.

Appropriateness of acute medical care for the elderly: an analysis of the literature.

Over the past 30 years, an explosion in health care expenditures has occurred. Prior to 1960, health care accounted for 4.4% of the U.S. Gross National Product; today it is 11%. Before rational solutions to controlling this rise can be proposed, we must determine whether the care that we are currently paying for is appropriate to the needs of the elderly. This paper analyzes the literature regarding appropriateness of acute care provided to the elderly. We identified 17 articles that explicitly cited appropriate or inappropriate care (including under-, over- and misuse) provided in hospital and ambulatory settings and for procedures, and 19 articles that presented data on the appropriateness of medication use in the elderly. Virtually every study included in this review found at least double-digit levels of inappropriate care. Perhaps as much as one-fifth to one-quarter of acute hospital services or procedures were felt to be used for equivocal or inappropriate reasons, and two-fifths to one-half of the medications studied were overused in outpatients. The few studies that examined underuse or misuse of services also documented the existence of these phenomena. This was especially true for the ambulatory care of chronic physical and mental conditions and concerned the use of low-cost technologies (visits, preventive services, some medications). Thus, we conclude that there appears to be a substantial problem in the matching of acute services to the needs of elderly patients. This mismatch occurs both in terms of overuse and underuse, at least for areas where research has been conducted.

How free care improved vision in the health insurance experiment.

We studied reasons for the improvement in the functional vision of enrollees receiving free care in the Rand Health Insurance Experiment. Among low income enrollees, 78 per cent on the free plan and 59 per cent on the cost-sharing plans had an eye examination; the proportions of those obtaining lenses were 30 per cent and 20 per cent, respectively. Visual acuity outcomes of low income vs non-poor enrollees were more adversely affected by enrollment in cost-sharing plans. Free care resulted in improved vision by increasing the frequency of eye examinations and lens purchases.

Chronic disease in a general adult population. Findings from the Rand Health Insurance Experiment.

Using questionnaire and physical screening examination data for a general population of 4,962 adults aged 18 to 61 years enrolled in the Rand Health Insurance Experiment, we calculated the prevalence of 13 chronic illnesses and assessed disease impact. Low-income men had a significantly higher prevalence of anemia, chronic airway disease and hearing impairment than their high-income counterparts, low-income women a higher prevalence of congestive heart failure, diabetes mellitus, hypertension, hearing impairment and vision impairment. Of our sample, 30% had one chronic condition and 16% had two or more. Several significant pairs or "clusters" of chronic illnesses were found. With few exceptions (diabetes, hypertension), the use of physician care in the previous year for a specific condition tended to be low. Disease impact (worry, activity restriction) was widespread but mild. Persons with angina, congestive heart failure, mild chronic joint disorders and peptic ulcer disease reported a greater impact than persons with other illnesses.

How free care reduced hypertension in the health insurance experiment.

In a controlled trial of the effects of medical insurance on spending and health status, we previously reported lower average (0.8 mm Hg) diastolic blood pressures with free care than with cost-sharing plans. We show herein that for clinically defined hypertensives, blood pressures with free care were significantly lower (1.9 mm Hg) than with cost-sharing plans, with a larger difference for low-income hypertensives than for high-income hypertensives (3.5 vs 1.1 mm Hg), but similar differences for blacks and whites. The cause of the difference was the additional contact with physicians under free care; this led to better detection and treatment of hypertensives not under care at the start of the study. Free care also led to higher compliance by hypertensives with diet and smoking recommendations and higher use of medication by those who needed it.