A decision making approach to assessing critical appraisal skills.

Critical appraisal is an important skill for practicing evidence-based medicine. Most assessments of critical appraisal focus either on knowledge about research design or evaluate a written critique of an article. This paper presents a Bayesian framework for assessing critical appraisal through measuring how clinical decisions and judgments change with the introduction of new research evidence. In a skilled clinician-appraiser, new evidence is predicted to cause the clinician to revise beliefs in the appropriateness of a decision, in proportion to both the strength of the findings and the methodological soundness of the research which produced the findings. The framework is illustrated by two studies using different items, respondents, and designs.

Medical students' application of published evidence: randomised trial.

OBJECTIVES: To determine how medical students apply research evidence that varies in validity of methods and importance of results to a clinical decision. DESIGN: Students examined a standardised patient with a whiplash injury, decided whether to order a cervical spine radiograph, and rated their confidence in their decision. They then read one of four randomly assigned variants of a structured abstract from a study of a decision rule that argued against such a procedure in this patient. Variants factorially combined two levels of validity of methods (prospective cohort or chart review) with two levels of importance of results (high sensitivity or high specificity rule). After reading the abstract, students repeated their choice and rated their confidence. SETTING: Academic medical centre in the United States. PARTICIPANTS: 164 graduating medical students. MAIN OUTCOME MEASURES: Proportion of students in each group whose beliefs shifted or stayed the same. RESULTS: When abstracts were of low importance students were more likely to shift their beliefs in favour of radiography, which was not supported by the evidence (odds ratio 3.42, 95% confidence interval 1.10 to 10.66). Neither methodological validity nor the interaction between validity and importance influenced decision shift. Few students acquired all necessary clinical data from the patient. CONCLUSIONS: Although the students could apply concepts of diagnostic testing, greater focus is needed on appraisal of validity and application of evidence to a particular patient.