The Mastery Rubric for Evidence-Based Medicine: Institutional Validation via Multidimensional Scaling.

UNLABELLED: CONSTRUCT: In this study we describe a multidimensional scaling (MDS) exercise to validate the curricular elements composing a new Mastery Rubric (MR) for a curriculum in evidence-based medicine (EBM). This MR-EBM comprises 10 elements of knowledge, skills, and abilities (KSAs) representing our institutional learning goals of career-spanning engagement with EBM. An MR also includes developmental trajectories for each KSA, beginning with medical school coursework, including residency training, and outlining the qualifications of individuals to teach and mentor in EBM. The development was not part of the validation effort, as our curriculum is focused at a single stage (undergraduate medical students). BACKGROUND: An MR comprises the desired KSAs for an entire curriculum, together with descriptions of a learner's performance and/or capabilities as they develop from novice to proficiency of the curricular target(s). The MR construct is intended to support curriculum development or refinement by capturing the KSAs that support the articulation of concrete learning goals; it also promotes assessment that demonstrates development in the target KSAs and encourages reflection and self-directed learning throughout the learner's career. Two other MRs have been published, and this is the first one specific to teaching and learning in medicine; this is also the first one created specifically to evaluate an existing curriculum. APPROACH: To validate the dispersion of the elements of the EBM curriculum, the nine clinical instructors in the EBM two-course curriculum completed an MDS exercise, rating the similarities of the 10 curricular elements. MDS is a mathematical approach to understanding relationships among concepts/objects when these relationships are difficult to quantify. Eliciting similarity ratings biased the responses toward the null hypothesis (that the elements are not different). RESULTS: MDS results suggested that the MR represents 10 different, although related, facets of the construct "evidence-based medicine." The results support the makeup of the MR-EBM, and its use to revise our EBM curriculum so that it is more closely aligned with this MR. CONCLUSIONS: An MR is a tool, and the MR-EBM that we describe can be useful to develop or evaluate a curriculum in EBM. The MR tool is particularly compatible with the objectives of training for EBM and practice and can be applied to create or evaluate a curriculum using any topical KSA framework. The MR-EBM we describe could be adopted or adapted to represent other institutional objectives for EBM training.

Multiple choice questions can be designed or revised to challenge learners' critical thinking.

Multiple choice (MC) questions from a graduate physiology course were evaluated by cognitive-psychology (but not physiology) experts, and analyzed statistically, in order to test the independence of content expertise and cognitive complexity ratings of MC items. Integration of higher order thinking into MC exams is important, but widely known to be challenging-perhaps especially when content experts must think like novices. Expertise in the domain (content) may actually impede the creation of higher-complexity items. Three cognitive psychology experts independently rated cognitive complexity for 252 multiple-choice physiology items using a six-level cognitive complexity matrix that was synthesized from the literature. Rasch modeling estimated item difficulties. The complexity ratings and difficulty estimates were then analyzed together to determine the relative contributions (and independence) of complexity and difficulty to the likelihood of correct answers on each item. Cognitive complexity was found to be statistically independent of difficulty estimates for 88 % of items. Using the complexity matrix, modifications were identified to increase some item complexities by one level, without affecting the item's difficulty. Cognitive complexity can effectively be rated by non-content experts. The six-level complexity matrix, if applied by faculty peer groups trained in cognitive complexity and without domain-specific expertise, could lead to improvements in the complexity targeted with item writing and revision. Targeting higher order thinking with MC questions can be achieved without changing item difficulties or other test characteristics, but this may be less likely if the content expert is left to assess items within their domain of expertise.

How informed is online informed consent?

We examined participants' reading and recall of informed consent documents presented via paper or computer. Within each presentation medium, we presented the document as a continuous or paginated document to simulate common computer and paper presentation formats. Participants took slightly longer to read paginated and computer informed consent documents and recalled slightly more information from the paginated documents. We concluded that obtaining informed consent online is not substantially different than obtaining it via paper presentation. We also provide suggestions for improving informed consent--in both face-to-face and online experiments.

Individual determinants of research utilization: a systematic review.

CONTEXT: In order to design interventions that increase research use in nursing, it is necessary to have an understanding of what influences research use. OBJECTIVE: To report findings on a systematic review of studies that examine individual characteristics of nurses and how they influence the utilization of research. SEARCH STRATEGY: A survey of published articles in English that examine the influence of individual factors on the research utilization behaviour of nurses, without restriction of the study design, from selected computerized databases and hand searches. INCLUSION CRITERIA: Articles had to measure one or more individual determinants of research utilization, measure the dependent variable (research utilization), and evaluate the relationship between the dependent and independent variables. The studies also had to indicate the direction of the relationship between the independent and dependent variables, report a P-value and the statistic used, and indicate the magnitude of the relationship. RESULTS: Six categories of potential individual determinants were identified: beliefs and attitudes, involvement in research activities, information seeking, professional characteristics, education and other socio-economic factors. Research design, sampling, measurement, and statistical analysis were examined to evaluate methodological quality. Methodological problems surfaced in all of the studies and, apart from attitude to research, there was little to suggest that any potential individual determinant influences research use. CONCLUSION: Important conceptual and measurement issues with regard to research utilization could be better addressed if research in the area were undertaken longitudinally by multi-disciplinary teams of researchers.