Composite reliability of a workplace-based assessment toolbox for postgraduate medical education.

In recent years, postgraduate assessment programmes around the world have embraced workplace-based assessment (WBA) and its related tools. Despite their widespread use, results of studies on the validity and reliability of these tools have been variable. Although in many countries decisions about residents' continuation of training and certification as a specialist are based on the composite results of different WBAs collected in a portfolio, to our knowledge, the reliability of such a WBA toolbox has never been investigated. Using generalisability theory, we analysed the separate and composite reliability of three WBA tools [mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), and multisource feedback (MSF)] included in a resident portfolio. G-studies and D-studies of 12,779 WBAs from a total of 953 residents showed that a reliability coefficient of 0.80 was obtained for eight mini-CEXs, nine DOPS, and nine MSF rounds, whilst the same reliability was found for seven mini-CEXs, eight DOPS, and one MSF when combined in a portfolio. At the end of the first year of residency a portfolio with five mini-CEXs, six DOPS, and one MSF afforded reliable judgement. The results support the conclusion that several WBA tools combined in a portfolio can be a feasible and reliable method for high-stakes judgements.

An experimental electronic medical-record system with multiple views on medical narratives.

In this article we describe a prototype electronic medical-record system that has been developed specifically to facilitate searching through medical narratives, as part of a preparatory study to the development of a medical workstation. This system contains a searching structure in which several views on the medical data are combined. We describe the structure of the database and the user-interface. A second design goal was to prepare the system for conducting research with respect to the use of the medical record. We address the kind of test designs the system can support and describe the research module that was implemented. We also describe a study we conducted with the system and report part of its results. We discuss the significance of this medical-record system compared to systems in the literature, some technical and functional problems encountered, and our future plans with the system.

A closer look at cueing effects in multiple-choice questions.

This study investigates the cueing effect occurring in multiple choice questions. Two parallel tests with matching contents were administered. By means of a computer program, examinees of different training levels and professional expertise were presented the same set of 35 cases (derived from patient problems in general practice) twice. The first time the cases were linked to open-ended questions; the second time they were linked to multiple choice questions. The examinees consisted of 75 medical students from three different years of training, 25 residents in training for general practice and 25 experienced general practitioners. Across groups, total test scores reflected a difference in mean scores on both formats, and a high inter-test correlation. Within each level of expertise, differences in mean scores and high correlations were also found. The data were further explored per group of examinees. Two types of cueing effects were found: positive cueing (examinees were cued towards the correct answer) and negative cueing (examinees were cued towards an incorrect answer). These effects were found at all levels of expertise and in almost all items. However, both effects decline with increasing level of expertise. Positive cueing mainly occurs in difficult items, whereas negative cueing mainly occurs in easy items.

Development, implementation and a first evaluation of a protocol processing system (ProtoVIEW).

In this contribution the protocol information system ProtoVIEW is presented. The system provides the necessary information to physicians about diagnostic procedures and therapies. It is implemented as a stand alone system. The design criteria are discussed and the results of a first evaluation presented. It appears that interns can easily find the required information with the help of the system. The time that they need for accessing the relevant information is relatively short (about 1 min). The users expressed the opinion that the system is easy to use and does support them in the management of their patients. On the basis of this evaluation and evaluations reported elsewhere it is concluded (a) that stand-alone protocol systems can support daily patient management in a positive way and (b) that the design criteria for a protocol information system as presented in this paper are useful for prospective protocol information system developers.

Computer supported patient management.

Both a prototype medical record system and a protocol information system are presented. Both systems are used for research purposes. The philosophy behind the medical record system is that the system should provide both freedom of vocabulary and freedom of level of detail. The freedom of vocabulary is guaranteed by allowing free text entry. Freedom of level of detail is obtained by structuring the medical record in a hierarchical way and to allow the physician to enter data at the level of detail that suits him best. The protocol information system supports the physician in managing the patient. It guides the user through the protocol and allows him to obtain background information about the protocol. Both flowcharts and images are included. After each session a summary of the steps taken can be printed. A first evaluation indicates that physicians like to use the system and learn from it.