Sensitivity and specificity of the RAND/UCLA Appropriateness Method to identify the overuse and underuse of coronary revascularization and hysterectomy.

There is no empirical evidence on the sensitivity and specificity of methods to identify the possible overuse and underuse of medical procedures. To estimate the sensitivity and specificity of the RAND/UCLA Appropriateness Method. Parallel three-way replication of the RAND/UCLA Appropriateness Method for each of two procedures, coronary revascularization and hysterectomy. Maximum likelihood estimates of the sensitivity and specificity of the method for each procedure. These values were then used to re-calculate past estimates of overuse and underuse, correcting for the error rate in the appropriateness method. The sensitivity of detecting overuse of coronary revascularization was 68% (95% confidence interval 60-76%) and the specificity was 99% (98-100%). The corresponding values for hysterectomy were 89% (85-94%) and 86% (83-89%). The sensitivity and specificity of detecting the underuse of coronary revascularization were 94% (92-95%) and 97% (96-98%), respectively. Past applications of the appropriateness method have overestimated the prevalence of the overuse of hysterectomy, underestimated the prevalence of the overuse of the coronary revascularization, and provided true estimates of the underuse of revascularization. The sensitivity and specificity of the RAND/UCLA Appropriateness Method vary according to the procedure assessed and appear to estimate the underuse of procedures more accurately than their overuse.

Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in women with breast cancer.

BACKGROUND: There is controversy whether high-dose chemotherapy and a blood cell or bone marrow autotransplant is a better treatment than conventional-dose chemotherapy for women with local/regional or metastatic breast cancer. Subject selection and time-to-treatment biases make definitive comparison impossible. Recent results of randomized trials are contradictory. OBJECTIVE: Determine appropriateness of high-dose chemotherapy and a blood cell or bone marrow autotransplant in women with breast cancer. PANELISTS: Nine breast cancer experts from diverse geographic sites and practice settings. EVIDENCE: Boolean MEDLINE searches of 'breast cancer' and 'chemotherapy' and/or 'blood cell' or 'bone marrow transplants'. PROCESS: We used a modified Delphi-panel group judgement process. Clinical variables were permuted to define 2058 clinical settings. Each panelist rated appropriateness of high-dose therapy and an autotransplant versus conventional therapy on a 9-point ordinal scale (1: most inappropriate, 9: most appropriate). An appropriateness index was developed based on median rating and amount of disagreement. The relationship of appropriateness indices to the permuted clinical variables was considered by analysis of variance and recursive partitioning. CONCLUSIONS: In women with local/regional breast cancer autotransplants were rated: 1) appropriate in those with > or = 10 cancer-involved lymph nodes; 2) uncertain in those with 4-9 cancer-involved nodes; and 3) inappropriate in women with < or = 3 cancer-involved lymph nodes. In women with metastatic breast cancer autotransplants were rated: 1) appropriate in those with metastases to 'favorable' sites (skin, lymph node, pleura) and a complete or partial response to chemotherapy; 2) uncertain in women with metastases to 'unfavorable' sites (lung, liver, or central nervous system) and a complete response to chemotherapy or those with bone metastases and a complete or partial response or stable disease after chemotherapy; and 3) inappropriate in other settings.

Effect of variability in the interpretation of coronary angiograms on the appropriateness of use of coronary revascularization procedures.

BACKGROUND: Evidence from numerous studies of coronary angiography show differences between observers' assessments of 15% to 45%. The implication of this variation is serious: If readings are erroneous, some patients will undergo revascularization procedures unnecessarily and others will be denied an essential treatment. We evaluated the variation in interpretation of angiograms and its potential effect on appropriateness of use of revascularization procedures. METHODS AND RESULTS: Angiograms of 308 randomly selected patients previously studied for appropriateness of angiography, coronary artery bypass grafting (CABG), and percutaneous transluminal coronary angioplasty (PTCA) were interpreted by a blinded panel of 3 experienced angiographers and compared with the original interpretations. The potential effect on differences on the appropriateness of revascularization was assessed by use of the RAND criteria. Technical deficiencies were found in 52% of cases. Panel readings tended to show less significant disease (none in 16% of vessels previously read as showing significant disease), less severity of stenosis (43% lower, 6% higher), and lower extent of disease (23% less, 6% more). The classification of CABG changed from necessary/appropriate to uncertain/inappropriate for 17% to 33% of cases when individual ratings were replaced by panel readings. CONCLUSIONS: The general level of technical quality of coronary angiography is unsatisfactory. Variation in the interpretation of angiograms was substantial in all measures and tended to be higher in individual than in panel readings. The effect was to lead to a potential overestimation of appropriateness of use of CABG by 17% and of PTCA by 10%. These findings indicate the need for increased attention to the technical quality of studies and an independent second reading for angiograms before recommending revascularization.

Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in chronic myelogenous leukemia in chronic phase.

BACKGROUND: It is uncertain which people with chronic myelogenous leukemia (CML) in chronic phase should receive conventional treatment (interferon and/or chemotherapy) versus high-dose therapy and a bone marrow transplant. There are no randomized trials comparing these approaches and analyses of data from non-randomized studies are complex, contradictory without sufficient detail to allow subject-level treatment decisions. OBJECTIVE: Determine appropriateness of high-dose therapy and bone marrow transplants in persons with CML in chronic phase with specific features. Develop a treatment algorithm. PANELISTS: nine leukemia experts from diverse geographic sites and practice settings. EVIDENCE: Boolean MEDLINE searches of chronic myelogenous leukemia and chemotherapy and/or transplants. CONSENSUS PROCESS: We used a modified Delphi-panel group judgment process. Age, prognostic score, disease duration, and type of conventional therapy and response were permuted to define 90 clinical settings. Each panelist rated appropriateness of high-dose therapy and a transplant versus conventional therapy on a 9-point ordinal scale (1, most inappropriate, 9, most appropriate) considering three types of donors: (1) HLA-identical siblings; (2) alternative donors (HLA-matched related or unrelated people other than an HLA-identical sibling); and (3) autotransplants. An appropriateness index was developed based on median rating and amount of disagreement. Relationship of appropriateness indices to permuted clinical variables was considered by analysis of variance and recursive partitioning. Preference between donor types was analyzed by comparing mean appropriateness indices of similar settings and a treatment algorithm developed. CONCLUSIONS: In people with CML in chronic phase and an HLA-identical sibling donor and in those with an alternative donor (but no HLA-identical sibling), a transplant was rated appropriate in those with a < or = partial cytogenetic response to interferon and uncertain or inappropriate in all other settings. Autotransplants were rated uncertain or inappropriate in all settings. Most of the variance in appropriateness ratings between different clinical settings was accounted for by response to interferon: complete versus < or = partial response. An HLA-identical sibling donor, when available, was always preferred to an alternative donor or autotransplant. In people without an HLA-identical sibling, an alternative donor was favored over an autotransplant at higher appropriateness indices and the converse at lower appropriateness indices.

Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute myelogenous leukemia in 1st remission.

BACKGROUND: Despite considerable data, there is still controversy over which adults with acute myelogenous leukemia (AML) in 1st remission should receive high-dose therapy and a bone marrow transplant rather than conventional-dose chemotherapy. Analyses of data from randomized trials are complex, conclusions sometimes contradictory and results not sufficiently detailed to allow subject-level decisions. OBJECTIVE: To determine appropriate use of high-dose therapy and bone marrow transplants in persons with AML in 1st remission with specific features. Develop a treatment algorithm. PANELISTS: Nine leukemia experts from diverse geographic sites and practice settings. EVIDENCE: Boolean MEDLINE searches of acute myelogenous leukemia and chemotherapy and/or transplants. CONSENSUS PROCESS: We used a modified Delphi-panel group judgment process. Age, WBC, cytogenetics and FAB-type were permuted to define 72 clinical settings. Each panelist rated appropriateness of high-dose therapy and a transplant versus conventional-dose chemotherapy on a nine-point ordinal scale (1, most inappropriate, 9, most appropriate) considering 3 types of donors: (1) HLA-identical siblings; (2) alternative donors (HLA-matched related or unrelated people other than an HLA-identical sibling); and (3) autotransplants. An appropriateness index was developed based on median rating and amount of disagreement. The relationship of appropriateness indices to the permuted clinical variables was considered by analysis of variance and recursive partitioning. Preference between donor types was analyzed by comparing mean appropriateness indices of comparable settings and a treatment algorithm developed. CONCLUSIONS: In people with an HLA-identical sibling, this type of transplant was rated appropriate in those with unfavorable cytogenetics and uncertain in all other settings. In people without an HLA-identical sibling, an alternative donor transplant was rated appropriate in those < 30 years with unfavorable cytogenetics, uncertain in those > 30 years and unfavorable cytogenetics and inappropriate in all other settings. Autotransplants were rated appropriate in people with unfavorable cytogenetics and uncertain in all other settings. An HLA-identical sibling donor, when available, was always preferred to an alternative donor transplant or autotransplant. In people without an HLA-identical sibling, an autotransplant was almost always favored over an alternative donor transplant with the magnitude of preference inversely correlated with transplant appropriateness.

Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow autotransplants in newly diagnosed multiple myeloma.

There is controversy whether high-dose therapy and a bone marrow autotransplant or conventional chemotherapy is a better treatment for newly diagnosed multiple myeloma. Data from 1 comparative study and 1 randomized trial provide insufficient subject-level data to advise specific people whether to have an autotransplant. We analyzed appropriate use of high-dose therapy and bone marrow autotransplants in people with newly diagnosed, multiple myeloma using a modified Delphi-panel group judgment process. The panel consisted of 9 myeloma experts from diverse geographic sites and practice settings who reviewed Boolean MEDLINE searches of multiple myeloma and chemotherapy or autotransplants. The panel rated a metric of 64 clinical setting developed by permuting age, performance score, disease-stage and disease-related prognostic variables and response to initial therapy. Each panelist rated appropriateness of high-dose therapy and an autotransplant versus conventional-dose chemotherapy on a 9-point ordinal scale (1, most inappropriate, 9, most appropriate). An appropriateness index was developed based on median rating and amount of disagreement. Relationship of appropriateness indices to the permuted clinical variables was considered by analysis of variance and recursive partitioning. Autotransplants were rated appropriate in persons <55 years old with stage 3 disease and a complete or partial response or stable disease after initial chemotherapy, inappropriate in persons with stage 1 or 2 disease, a performance score <70% and a complete or partial response or stable disease after initial chemotherapy and uncertain in all other settings.

Delphi-panel analysis of appropriateness of high-dose therapy and bone marrow transplants in adults with acute lymphoblastic leukemia in first remission.

BACKGROUND: There is controversy over whether high-dose therapy and a bone marrow transplant is better than conventional-dose chemotherapy in adults with acute lymphoblastic leukemia (ALL) in first remission. This decision may depend on which type of donor is available: an HLA-identical sibling, an alternative donor transplant (HLA-matched related or unrelated people other than HLA-identical siblings), or autotransplant. OBJECTIVE: To determine the appropriate use of high-dose therapy and bone marrow transplants in ALL in first remission. Develop a treatment algorithm. PANELISTS: Nine leukemia experts from diverse geographic sites and practice settings. EVIDENCE: Boolean MEDLINE searches of acute lymphoblastic leukemia and chemotherapy and/or transplants. CONSENSUS PROCESS: We used a modified Delphi-panel group judgment process. Age, white blood cell (WBC) count, cytogenetics and immune type were permuted to define 48 clinical settings. Each panelist rated appropriateness of high-dose therapy and a transplant versus conventional-dose chemotherapy on a 9-point ordinal scale (1, most inappropriate; 9, most appropriate) considering three types of donors: (1) HLA-identical siblings; (2) alternative donors; and (3) autotransplants. An appropriateness index was developed based on median rating and amount of disagreement. Relationship of appropriateness indices to the permuted clinical variables was considered by analysis of variance and recursive partitioning. Preference between donor types was analyzed by comparing mean appropriateness indices of comparable settings and a treatment algorithm was developed. CONCLUSIONS: In people with an HLA-identical sibling donor, transplants were rated appropriate in those with unfavorable cytogenetics and uncertain in all other settings. An HLA-identical sibling donor was always preferred to an alternative donor or autotransplant. In people without an HLA-identical sibling but with an alternative donor, this type of transplant was rated appropriate in those with unfavorable cytogenetics. However, an autotransplant was preferred over an alternative donor transplant in all other settings where a transplant was rated uncertain. In people without an HLA-identical sibling or alternative donor, autotransplants were rated uncertain in all settings except in those with not unfavorable cytogenetics, WBC < 100 x 10(9) l(-1) and T- or pre-B-cell type where they were rated inappropriate.

The development of indications for the preoperative use of recombinant erythropoietin.

OBJECTIVE: To develop indications for the preoperative use of recombinant erythropoietin (rHuEPO) alone and in conjunction with preoperative autologous donation (PAD). DESIGN: A 2-round modified Delphi-consensus process. PARTICIPANTS: Nine physicians representing multiple clinical specialties, practice environments and geographic locations. METHOD: From evidence tables and a literature summary (MEDLINE database from January 1985 to August 1996) provided and using the RAND-UCLA appropriateness method, the physicians developed 264 indications for the preoperative use of rHuEPO by permuting 7 clinical factors (age, history of transfusion or antibody incompatibility, hemoglobin level, anemia of chronic disease, expected blood loss, presence of cardiovascular or cardiopulmonary disease and patient anxiety). These indications were rated on a 9-point appropriateness scale. Median scores and measures of agreement were determined. OUTCOME MEASURES: The significance of cost constraints or cost and blood supply constraints and the impact of each clinical factor on the ratings as judged by statistical analysis. RESULTS: Of the 264 indications, 54% were rated appropriate, 18% uncertain and 28% inappropriate. Expected blood loss had the greatest impact on the ratings (high expected blood loss had a 5.9 point more appropriate rating on the 9-point scale than low expected blood loss [p < 0.0001]). Preoperative hemoglobin level also significantly influenced the ratings (p < 0.0001). Compared with the clinical context, the ratings under the cost constraint were 1.0 less appropriate (p < 0.0001) for rHuEPO alone and 1.2 less appropriate for rHuEPO and PAD (p < 0.0001). The ratings for patients with moderate expected blood loss were significantly influenced by the cost constraint (less appropriate). CONCLUSIONS: Expected blood loss and preoperative hemoglobin level were the best indicators of rHuEPO appropriateness. Different contexts modify the appropriateness ratings of an expensive drug like rHuEPO.

The reproducibility of a method to identify the overuse and underuse of medical procedures.

BACKGROUND: To assess the overuse and underuse of medical procedures, various methods have been developed, but their reproducibility has not been evaluated. This study estimates the reproducibility of one commonly used method. METHODS: We performed a parallel, three-way replication of the RAND-University of California at Los Angeles appropriateness method as applied to two medical procedures, coronary revascularization and hysterectomy. Three nine-member multidisciplinary panels of experts were composed for each procedure by stratified random sampling from a list of experts nominated by the relevant specialty societies. Each panel independently rated the same set of clinical scenarios in terms of the appropriateness of the relevant procedure on a risk-benefit scale ranging from 1 to 9. Final ratings were used to classify the procedure in each scenario as necessary or not necessary (to evaluate underuse) and inappropriate or not inappropriate (to evaluate overuse). Reproducibility was measured by overall agreement and by the kappa statistic. The criteria for underuse and overuse derived from these ratings were then applied to real populations of patients who had undergone coronary revascularization or hysterectomy. RESULTS: The rates of agreement among the three coronary-revascularization panels were 95, 94, and 96 percent for inappropriate-use scenarios and 93, 92, and 92 percent for necessary-use scenarios. Agreement among the three hysterectomy panels was 88, 70, and 74 percent for inappropriate-use scenarios. Scenarios involving necessary use of hysterectomy were not assessed. The three-way kappa statistic to detect overuse was 0.52 for coronary revascularization and 0.51 for hysterectomy. The three-way kappa statistic to detect underuse of coronary revascularization was 0.83. Application of individual panels' criteria to real populations of patients resulted in a 100 percent variation in the proportion of cases classified as inappropriate and a 20 percent variation in the proportion of cases classified as necessary. CONCLUSIONS: The appropriateness method is far from perfect. Appropriateness criteria may be useful in comparing levels of appropriate procedures among populations but should not by themselves be used to direct care for individual patients.

Delphi-panel analysis of appropriateness of high-dose chemotherapy and blood cell or bone marrow autotransplants in diffuse large-cell lymphoma.

Although high-dose chemotherapy and a blood cell or bone marrow autotransplant are commonly used to treat people with diffuse large-cell lymphoma, there is controversy whether this is better than conventional-dose chemotherapy. Subject-selection and time-to-treatment biases preclude comparison of data from uncontrolled trials and there are few date from randomized trials. We used a Delphi-panel group judgment process to determine appropriateness of high-dose chemotherapy and a blood cell or bone marrow transplant. Results were compared to those of randomized trials. Nine lymphoma experts from diverse geographic sites and practice settings were panelists. Boolean MEDLINE searches of lymphoma and chemotherapy and an autotransplant formed the dataset. Panelists were asked to rate appropriateness of high-dose chemotherapy and an autotransplant compared to conventional-dose chemotherapy. Clinical variables were permuted to define 80 clinical settings rated by the panelists on a 9-point ordinal scale. Results were used to determine an appropriateness index reflecting the mean and distribution of ratings. The relationship of appropriateness indices to permuted clinical variables was considered by analysis of variance and recursive partitioning. In people with initial diffuse large-cell lymphoma, autotransplants were never rated appropriate. They were rated uncertain in all settings except in people never receiving chemotherapy and in those with a complete response to chemotherapy and an international prognostic index < 3, where they were rated inappropriate. In people with recurrent lymphoma, autotransplants were rated appropriate in those with a complete or partial response to chemotherapy, uncertain in those with a less than partial response and in those not receiving re-induction chemotherapy and inappropriate in people with CNS lymphoma. These conclusions agree with results of randomized trials.

Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy.

We assessed the predictive validity of an expert panel's ratings of the appropriateness of carotid endarterectomy by comparing ratings to the results of subsequent randomized clinical trials. We found the trials confirmed the ratings for 44 indications (covering almost 30% of operations performed in 1981) and refuted the ratings for none.

Adjusting cesarean delivery rates for case mix.

OBJECTIVES: (1) To describe the issues in developing a clinical predictor of cesarean delivery that could be used to adjust reported cesarean rates for case mix, and (2) to compare its performance to other, simpler predictors using clinical and statistical criteria. DATA SOURCES: Singleton births greater than 2,500 grams in Washington State in 1989 and 1990 for whom mothers and infant hospital discharge records could be matched to birth certificate data. DESIGN: Statistical analysis of retrospective merged hospital and birth certificate data, which were used to develop variables and models to predict the probability that any particular delivery would be a cesarean. PRINCIPAL FINDINGS: Merged data led to better predictor variables than those based on one source. A simple four-category hierarchical classification into births with prior cesarean, breech but no prior cesarean, first birth, and other explains 30 percent of the variance in individual cesarean rates. The full clinical model fit the data well and explained 37 percent of the variance. Multiparas without serious complications comprised 35 percent of the mothers and averaged less than 2 percent cesareans. A hospital's predicted cesarean rate depends strongly on the proportion of its births that are first births. CONCLUSION: Government and private agencies have reported cesarean rates as measures of hospital performance. Depending on data and resources available, both simple and complex measures of case mix can be used to adjust reported rates. These adjustments should not include all variables related to the rates. Proper adjustments may not alter hospital rankings greatly, but they will improve the validity and acceptability of the reports.

Measuring the clinical consistency of panelists' appropriateness ratings: the case of coronary artery bypass surgery.

OBJECTIVE: To assess the clinical consistency of expert panelists' ratings of appropriateness for coronary artery bypass surgery. DESIGN: Quantitative analysis of panelists' ratings. PARTICIPANTS: Nine physicians (three cardiothoracic surgeons, four cardiologists, and two internists) convened by RAND to establish criteria for the appropriateness of coronary artery bypass surgery. MAIN OUTCOMES MEASURES: Percentage of indication-pairs given clinically inconsistent ratings (i.e. higher rating assigned to one member of an indication-pair when rating should have been equal or lower). RESULTS: In the final round of appropriateness ratings, among 1785 pairs of indications differing only on a single clinical factor (e.g., three-vessel vs. two-vessel stenosis), 6.6% were assigned clinically inconsistent ratings by individual panelists, but only 2.7% received inconsistent ratings from the panel as a whole (using the median panel rating as the criterion). Internists on the panel provided fewer inconsistent ratings (4.6%) than either cardiologists (7.8%) or cardiothoracic surgeons (6.3%) (p < 0.001). More inconsistencies were noted when the factor distinguishing otherwise identical indications was symptom severity (inconsistency rate, 13.2%) or intensity of medical therapy (13.2%) than when it was number of stenosed vessels (3.8%) or proximal left anterior descending (PLAD) involvement (1.9%). Contrary to expectations, panelists' inconsistency rates increased between the initial and final rounds of appropriateness ratings (from 3.9 to 6.6%, p < 0.001). Panelists' mean ratings across indications were only weakly correlated with individual inconsistency rates (r = 0.18, p = ns). CONCLUSIONS: The RAND/UCLA method for assessing the appropriateness of coronary revascularization generally produces criteria that are clinically consistent. However, research is needed to understand the sources of panelists' inconsistencies and to reduce inconsistency rates further.

The appropriateness of coronary artery bypass graft surgery in academic medical centers. Working Group of the Appropriateness Project of the Academic Medical Center Consortium.

OBJECTIVE: To compare the appropriateness of use of coronary artery bypass graft (CABG) surgery in Academic Medical Center Consortium hospitals as judged 1) according to criteria developed by an expert panel, 2) according to revisions of those criteria made by cardiac surgeons from the Academic Medical Center Consortium, and 3) by review of cases by the surgeons responsible for those cases. DESIGN: Retrospective, randomized medical record review. SETTING: 12 Academic Medical Center Consortium hospitals. PATIENTS: Random sample of 1156 patients who had had isolated CABG surgery in 1990. MAIN OUTCOME MEASURES: 1) Percentage of patients with indications for which CABG surgery was classified as appropriate, Inappropriate, or of uncertain appropriateness and 2) percentage of cases in which CABG surgery was judged inappropriate or uncertain for which ratings changed after local case review. RESULTS: Data were retrieved from medical records by trained abstractors using an explicit data collection instrument. Cases in which CABG surgery was judged to be inappropriate or uncertain were individually reviewed by the responsible surgeons. According to the expert panel ratings, 83% of the CABG operations (95% CI, 81% to 85%) were necessary, 9% (CI, 8% to 10%) were appropriate, 7% (CI, 5% to 8%) were uncertain, and 1.6% (CI, 0.6% to 2.5%) were inappropriate. These rates are almost identical to those found in a previous study that was done in New York State and that used the same criteria (in that study, 91% of operations were classified as necessary or appropriate, 7% were classified as uncertain, and 2.4% were classified as inappropriate). Rates of inappropriate procedures varied from 0% to 5% among the 12 member hospitals (P = 0.02). The Academic Medical Center Consortium cardiac surgeons revised 568 (24%) of the indications used by the expert panel. However, because those revisions altered the appropriateness ratings in both directions and affected only 50 cases (4%), the net effect of the revisions was slight: The rate of inappropriate CABG surgery increased from 1.6% to 1.9%. Local review found that data collection errors had caused erroneous ratings in 12.5% of 64 cases in which surgery had been classified as inappropriate or uncertain. CONCLUSIONS: The Academic Medical Center Consortium hospitals had low rates of inappropriate and uncertain use of CABG surgery, regardless of the criteria used for assessment. Even though surgeons from the Consortium revised the appropriateness ratings extensively, their revisions had a negligible effect on the overall assessment of appropriateness. However, because of potential data collection errors, appropriateness criteria should be used for individual case audits only if supplemented by subsequent physician review.

Variations by specialty in physician ratings of the appropriateness and necessity of indications for procedures.

The authors compare the appropriateness ratings and mutual influence of panelists from different specialties rating a comprehensive set of indications for six surgical procedures. Nine-member panels rated each procedure: abdominal aortic aneurysm surgery, carotid endarterectomy, cataract surgery, coronary angiography, and coronary artery bypass graft surgery/percutaneous transluminal coronary angioplasty (common panel). Panelists individually rated the appropriateness of indications at home and then discussed and re-rated the indications during a 2-day meeting. Subsequently, they rated the necessity of those indications scored by the group as appropriate. There were 45 panelists, including specialists (either performers of the procedure or members of a related specialty) and primary care providers, all drawn from nominations by their respective specialty societies. Main outcome measures included: individual panelists' mean ratings over all indications, mean change and conformity scores between rounds of ratings, and the percentage of audited actual procedures rated appropriate or necessary. Performers had the highest mean ratings, followed by physicians in related specialties, trailed by primary care providers. One fifth of all actual procedures were for indications rated appropriate by performers and less than appropriate by primary care providers. At the panel meetings, primary care providers and related specialists showed no greater tendency to be influenced by other panelists than did performers. Multispecialty panels provide more divergent viewpoints than panels composed entirely of performers. This divergence means that fewer actual procedures are deemed performed for appropriate or necessary indications.

The appropriateness of use of cardiovascular procedures in women and men.

OBJECTIVE: To determine whether there are differences between women and men in the appropriateness of use of cardiovascular procedures. DESIGN: Retrospective chart review. SETTING: Thirty hospitals located in New York State. PATIENTS: Random sample of 3979 patients undergoing coronary angiography, percutaneous transluminal coronary angioplasty, or coronary artery bypass graft surgery in 1990. MEASURES: We evaluated two measures: (1) the percent of women and men who underwent cardiovascular procedures for appropriate, uncertain, and inappropriate indications and (2) for coronary angiography patients, the prognostic exercise stress treadmill score that predicts before the coronary angiogram the 5-year probability of death from a cardiovascular event. RESULTS: The inappropriate rate of use of cardiovascular procedures was low and not significantly different for men and women (4% vs 5% for coronary angiography; 4% vs 3% for percutaneous transluminal coronary angioplasty; and 2% vs 3% for coronary artery bypass graft surgery, respectively), and the use of these procedures for uncertain reasons also did not vary significantly by gender. There was also no significant gender difference in the predicted risk of death from a cardiovascular event for coronary angiography patients: 24% of men and 22% of women were at high risk (ie, < 75% 5-year survival rate) and 20% and 16%, respectively, were at low risk (ie, > or = 95% 5-year survival rate). CONCLUSION: Based on two indicators, the RAND appropriateness score and the Duke prognostic exercise treadmill score, we were unable to find any evidence of a difference in the clinical appropriateness of use of these three cardiovascular procedures between women and men.

Comparison of the appropriateness of coronary angiography and coronary artery bypass graft surgery between Canada and New York State.

OBJECTIVE: To compare the appropriateness of coronary angiography and coronary artery bypass graft (CABG) use between the United States and Canada. DESIGN: Retrospective randomized medical record review. SETTING: All hospitals performing coronary angiography and/or CABG surgery in two Canadian provinces (Ontario and British Columbia); in New York State, 15 randomly selected hospitals that provide coronary angiography and 15 randomly selected hospitals that provide CABG surgery. PATIENTS: All patients were randomly selected. For coronary angiography, 533 patients in Canada and 1333 patients in New York were selected; for CABG, 556 patients in Canada and 1336 patients in New York were selected. MAIN OUTCOME MEASURES: Percentage of patients in each country who had coronary angiography or CABG for necessary, appropriate, uncertain, or inappropriate indications as rated by criteria developed separately in each country and the complications of those procedures. RESULTS: For coronary angiography, 9% of Canadian cases and 10% of New York cases were rated inappropriate using Canadian criteria compared with 5% and 4%, respectively, using US criteria. For CABG, 4% of Canadian cases and 6% of New York cases were rated inappropriate by Canadian criteria compared with 3% and 2%, respectively, using US criteria. A lower proportion of procedures were performed on persons aged 75 years or older in Canada than in New York for both coronary angiography (5% vs 11%; P < .001) and CABG (6% vs 14%; P < .001). Women were also represented in lower proportions among angiography cases in Canada than in New York (28% vs 35%; P = .023). Canadian patients with left main coronary disease waited significantly longer between angiography and CABG than did New York patients (P < .0001). CONCLUSIONS: Rates of inappropriate use of cardiac procedures were low in Canada and New York, which suggests that the regionalization of cardiac procedures that characterizes both health care systems contributes to better clinical decision making. Differences in the use of cardiac procedures among the elderly in the two countries merits further comparative examination.

The effect of alternative case-mix adjustments on mortality differences between municipal and voluntary hospitals in New York City.

OBJECTIVE: This study investigated how mortality differences between groups of municipal versus voluntary hospitals are affected by case-mix adjustment methods. DATA SOURCES AND STUDY SETTING: We sampled about 10,000 random admissions from administrative data for patients hospitalized with each of six conditions in hospitals in New York City during 1984-1987. STUDY DESIGN: We developed logistic regression models adjusting for age and gender, for principal diagnosis, for "limited other diagnoses" (secondary diagnoses that were very unlikely to result from care received), for "full other diagnoses" (all secondary diagnoses irrespective of whether they might have been due to care received), for previous diagnoses, and for other variables. PRINCIPAL FINDINGS: For five of the six conditions, when the limited other diagnoses adjustment was used there was higher mortality in the municipal hospitals (p < .05), with 3.3 additional deaths/100 admissions for myocardial infarction, 1.2 for pneumonia, 8.3 for stroke, 2.8 for head trauma, and 0.8 for hip repair. However, when the full other diagnoses adjustment was used, differences remained significant only for stroke (4.3 additional deaths/100 admissions) and head trauma (1.3) (p < .05). CONCLUSIONS: Estimates of mortality differences between New York City municipal and voluntary hospitals are substantially affected by which secondary diagnoses are used in case-mix adjustment. Judgments of quality should not be based on administrative data unless models can be developed that validly capture level of sickness at admission.