Development of an adaptive multimedia program to collect patient health data.

BACKGROUND: Direct entry of personal health survey information into a computer by patients could streamline clinical data collection, improve completeness and accuracy of health information available to providers, and provide data for tailoring health education messages. Few computerized programs, however, have been developed to adapt the interface to diverse literacy levels, language, computer skills, and health literacy levels of the broad spectrum of patient populations. METHODS: To develop such a program, we conducted cognitive response interviews with a diverse sample of 21 adult participants from two North Carolina health clinics. Participants were placed into one of three interview categories: (1) low reading skill level, low computer skills; (2) high reading skill level, low computer skills; and (3) high reading skill level, high computer skills. The "think-aloud" technique was employed to elicit feedback on a series of computer interface screens and health risk assessment questions. RESULTS: Interface-related findings showed that preference for touchscreen versus keyboard data entry was not strongly related to computer skill level but was related to question type. Respondents with limited education indicated that they would not click on a "help" or hyperlink option to get more information and that frequent reminders of directions on the screen were needed. Content-related findings showed numerous misperceptions regarding meanings of common health questions and terminology (e.g., seat belt use and intravenous drugs). CONCLUSIONS: These findings have implications for health literacy and may have an impact on the accuracy of information obtained. Study results will be incorporated into the development and testing of an innovative, adaptive multimedia program.

Scoring performance on computer-based patient simulations: beyond value of information.

As computer based clinical case simulations become increasingly popular for training and evaluating clinicians, approaches are needed to evaluate a trainee's or examinee's solution of the simulated cases. In 1997 we developed a decision analytic approach to scoring performance on computerized patient case simulations, using expected value of information (VOI) to generate a score each time the user requested clinical information from the simulation. Although this measure has many desirable characteristics, we found that the VOI was zero for the majority of information requests. We enhanced our original algorithm to measure potential decrements in expected utility that could result from using results of information requests that have zero VOI. Like the original algorithm, the new approach uses decision models, represented as influence diagrams, to represent the diagnostic problem. The process of solving computer based patient simulations involves repeated cycles of requesting and receiving these data from the simulations. Each time the user requests clinical data from the simulation, the influence diagram is evaluated to determine the expected VOI of the requested clinical datum. The VOI is non-zero only it the requested datum has the potential to change the leading diagnosis. The VOI is zero when the data item requested does not map to any node in the influence diagram or when the item maps to a node but does not change the leading diagnosis regardless of it's value. Our new algorithm generates a score for each of these situations by modeling what would happen to the expected utility of the model if the user changes the leading diagnosis based on the results. The resulting algorithm produces a non-zero score for all information requests. The score is the VOI when the VOI is non-zero It is a negative number when the VOI is zero.

Student and faculty performance in clinical simulations with access to a searchable information resource.

In this study we explore how students' use of an easily accessible and searchable database affects their performance in clinical simulations. We do this by comparing performance of students with and without database access and compare these to a sample of faculty members. The literature supports the fact that interactive information resources can augment a clinician's problem solving ability in small clinical vignettes. We have taken the INQUIRER bacteriological database, containing detailed information on 63 medically important bacteria in 33 structured fields, and incorporated it into a computer-based clinical simulation. Subjects worked through the case-based clinical simulations with some having access to the INQUIRER information resource. Performance metrics were based on correct determination of the etiologic agent in the simulation and crosstabulated with student access of the information resource; more specifically it was determined whether the student displayed the database record describing the etiologic agent. Chi-square tests show statistical significance for this relationship (chi 2 = 3.922; p = 0.048). Results support the idea that students with database access in a clinical simulation environment can perform at a higher level than their counterparts who lack access to such information, reflecting favorably on the use of information resources in training environments.

Feasibility and marketing studies of health sciences librarianship education programs.

The University of North Carolina at Chapel Hill evaluated five curricular models designed to improve education for health sciences librarianship. Three of the models enhanced existing degree and certificate programs, and two were new programs for working information professionals. Models were developed with input from experts and a Delphi study; the marketability of the models was tested through surveys of potential students and employers; and recommendations were made as a guide to implementation. The results demonstrated a demand for more specialized curricula and for retraining opportunities. Marketing data showed a strong interest from potential students in a specialized master's degree, and mid-career professionals indicated an interest in postmaster's programs that provided the ability to maintain employment. The study pointed to the opportunity for a center of excellence in health sciences information education to enable health sciences librarians to respond to their evolving roles.

Preparing tomorrow's health sciences librarians: feasibility and marketing studies.

The University of North Carolina at Chapel Hill is devising and evaluating five curricular models designed to improve education for health sciences librarianship. These models fit into a continual learning process from the initial professional preparation to lifelong learning opportunities. Three of them enhance existing degree and certificate programs in the School of Information and Library Science (SILS) with a health sciences specialization, and two are new programs for working information professionals. The approaches involve partnerships among SILS, the Health Sciences Library, and the program in Medical Informatics. The planning process will study the feasibility of the proposed programs, test the marketability of the models to potential students and employers, and make recommendations about implementation.

A comparison of hypertext and Boolean access to biomedical information.

This study explored which of two modes of access to a biomedical database better supported problem solving in bacteriology. Boolean access, which allowed subjects to frame their queries as combinations of keywords, was compared to hypertext access, which allowed subjects to navigate from one database node to another. The accessible biomedical data were identical across systems. Data were collected from 42 first year medical students, each randomized to the Boolean or hypertext system, before and after their bacteriology course. Subjects worked eight clinical case problems, first using only their personal knowledge and, subsequently, with aid from the database. Database retrievals enabled students to answer questions they could not answer based on personal knowledge only. This effect was greater when personal knowledge of bacteriology was lower. The results also suggest that hypertext was superior to Boolean access in helping subjects identify possible infectious agents in these clinical case problems.

End-user search behaviors and their relationship to search effectiveness.

One hundred sixty-one MEDLINE searches conducted by third-year medical students were analyzed and evaluated to determine which search moves were used, whether those individual moves were effective, and whether there was a relationship between specific search behaviors and the effectiveness of the search strategy as a whole. The typical search included fourteen search statements, used seven terms or "limit" commands, and resulted in the display of eleven citations. The most common moves were selection of a database, entering single-word terms and free-text term phrases, and combining sets of terms. Syntactic errors were also common. Overall, librarians judged the searches to be adequate, and students were quite satisfied with their own searches. However, librarians also identified many missed opportunities in the search strategies, including underutilization of the controlled vocabulary, subheadings, and synonyms for search concepts. No strong relationships were found between specific search behaviors and search effectiveness (as measured by the librarians' or students' evaluations). Implications of these findings for system design and user education are discussed.

Information retrieved from a database and the augmentation of personal knowledge.

OBJECTIVE: To assess the degree to which information retrieved from a biomedical database can augment personal knowledge in addressing novel problems, and how the ability to retrieve information evolves over time. DESIGN: This longitudinal study comprised three assessments of two cohorts of medical students. The first assessment occurred just before student course experience in bacteriology, the second occurred just after the course, and the third occurred five months later. At each assessment, the students were initially given a set of bacteriology problems to solve using their personal knowledge only. Each student was then reassigned a sample of problems he or she had answered incorrectly, to work again with assistance from a database containing information about bacteria and bacteriologic concepts. The initial pass through the problems generated a "personal knowledge" score; the second pass generated a "database-assisted" score for each student at each assessment. RESULTS: Over two cohorts, students' personal knowledge scores were very low (approximately 12%) at the first assessment. They rose substantially at the second assessment (approximately 48%) but decreased six months later (approximately 25%). By contrast, database-assisted scores rose linearly: from approximately 44% at the first assessment to approximately 57% at the second assessment, to approximately 75% at the third assessment. CONCLUSION: The persistent increase in database-assisted scores, even when personal knowledge had attenuated, was the most remarkable finding of this study. While some of the increase may be attributed to artifacts of the design, the pattern seems to result from the retained ability to recognize problem-relevant information in a database even when it cannot be recalled.

Information-seeking behaviors of medical students: a classification of questions asked of librarians and physicians.

To solve a problem, a person often asks questions of someone with more expertise. This paper reports on a study of the types of questions asked and how the experts are chosen. In the study, sixty-three first-year medical students responded to clinical scenarios, each describing a patient affected by a toxin and asking questions concerning the identity of the toxin and its characteristics. After answering those questions, the students were asked to imagine that they had access to a medical reference librarian and an internist specializing in toxicology. The students then generated two questions for each expert about each clinical scenario. Each question was categorized according to the type of information requested, and the frequency of each type of question was calculated. The study found that students most often asked for the identification of the toxin(s), references about the scenario, or the effects of the toxin; an explanation of the patient's symptoms; or a description of the appropriate treatment. Students were more likely to address questions on the identity of the toxin and references to the hypothetical librarian; they were more likely to ask the internist for explanations of the symptoms and descriptions of the treatment. The implications of these results for the design of information and educational systems are discussed.

Database access and problem solving in the basic sciences.

This study examined the potential contribution that access to a database of biomedical information may offer in support of problem-solving exercises when personal knowledge is inadequate. Thirty-six medical students were assessed over four occasions and three domains in the basic sciences: bacteriology, pharmacology, and toxicology. Each assessment consisted of a two-pass protocol in which students were first assessed for their personal knowledge of a domain with a short-answer problem set. Then, for a sample of problems they had missed, they were asked to use a database, INQUIRER, to respond to questions which they had been unable to address with their personal knowledge. Results indicate that for a domain in which the database is well-integrated in course activities, useful retrieval of information which augmented personal knowledge increased over three assessment occasions, even continuing to increase several months after course exposure and experience with the database. For all domains, even at assessments prior to course exposure, students were able to moderately extend their ability to solve problems through access to the INQUIRER database.