The full spectrum of biomedical informatics education at Oregon Health & Science University.

OBJECTIVES: The growing use of health information technology in operational settings, along with the maturation of the discipline of biomedical informatics, requires reorganization of educational programs in the field. The objective of this paper is to provide a context and description of the biomedical informatics education program at Oregon Health & Science University. METHODS: The details of the program are provided. RESULTS: The paper describes the overall program and its component curricula. CONCLUSIONS: OHSU has developed a program that caters to the full spectrum of those who will work in the field, allowing education tailored to their career goals and needs. The maturation of Internet technologies also allow most aspects of the program to be delivered on-line. The informatics field must step up to the challenge of educating the best workforce to achieve our goals for the optimal use of HIT.

Reducing workload in systematic review preparation using automated citation classification.

OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.

Using co-occurrence network structure to extract synonymous gene and protein names from MEDLINE abstracts.

BACKGROUND: Text-mining can assist biomedical researchers in reducing information overload by extracting useful knowledge from large collections of text. We developed a novel text-mining method based on analyzing the network structure created by symbol co-occurrences as a way to extend the capabilities of knowledge extraction. The method was applied to the task of automatic gene and protein name synonym extraction. RESULTS: Performance was measured on a test set consisting of about 50,000 abstracts from one year of MEDLINE. Synonyms retrieved from curated genomics databases were used as a gold standard. The system obtained a maximum F-score of 22.21% (23.18% precision and 21.36% recall), with high efficiency in the use of seed pairs. CONCLUSION: The method performs comparably with other studied methods, does not rely on sophisticated named-entity recognition, and requires little initial seed knowledge.

Clinical outcomes resulting from telemedicine interventions: a systematic review.

BACKGROUND: The use of telemedicine is growing, but its efficacy for achieving comparable or improved clinical outcomes has not been established in many medical specialties. The objective of this systematic review was to evaluate the efficacy of telemedicine interventions for health outcomes in two classes of application: home-based and office/hospital-based. METHODS: Data sources for the study included deports of studies from the MEDLINE, EMBASE, CINAHL, and HealthSTAR databases; searching of bibliographies of review and other articles; and consultation of printed resources as well as investigators in the field. We included studies that were relevant to at least one of the two classes of telemedicine and addressed the assessment of efficacy for clinical outcomes with data of reported results. We excluded studies where the service did not historically require face-to-face encounters (e.g., radiology or pathology diagnosis). All included articles were abstracted and graded for quality and direction of the evidence. RESULTS: A total of 25 articles met inclusion criteria and were assessed. The strongest evidence for the efficacy of telemedicine in clinical outcomes comes from home-based telemedicine in the areas of chronic disease management, hypertension, and AIDS. The value of home glucose monitoring in diabetes mellitus is conflicting. There is also reasonable evidence that telemedicine is comparable to face-to-face care in emergency medicine and is beneficial in surgical and neonatal intensive care units as well as patient transfer in neurosurgery. CONCLUSIONS: Despite the widespread use of telemedicine in virtually all major areas of health care, evidence concerning the benefits of its use exists in only a small number of them. Further randomized controlled trials must be done to determine where its use is most effective.

Implementation and evaluation of a medical informatics distance education program.

OBJECTIVE: Given the need for continuing education in medical informatics for mid-career professionals, the authors aimed to implement and evaluate distance learning courses in this area. DESIGN: The authors performed a needs assessment, content and technology planning, implementation, and student evaluation. MEASUREMENTS: The needs assessment and student evaluations were assessed using a combination of Likert scale and free-form questions. RESULTS: The needs assessment indicated much interest in a medical informatics distance learning program, with electronic medical records and outcome research the subject areas of most interest. The courses were implemented by means of streaming audio plus slides for lectures and threaded discussion boards for student interaction. Students were assessed by multiple-choice tests, a term paper, and a take-home final examination. In their course evaluations, student expressed strong satisfaction with the teaching modalities, course content, and system performance. Although not assessed experimentally, the performance of distance learning students was superior to that of on-campus students. CONCLUSION: Medical informatics education can be successfully implemented by means of distance learning technologies, with favorable student satisfaction and demonstrated learning. A graduate certificate program is now being implemented.

Telemedicine for the Medicare population: pediatric, obstetric, and clinician-indirect home interventions.

BACKGROUND: This report is a supplement to an earlier evidence report, Telemedicine for the Medicare Population, which was intended to help policymakers weigh the evidence relevant to coverage of telemedicine services under Medicare. That report focused on telemedicine programs and clinical settings that had been used with or were likely to be applied to Medicare beneficiaries. While we prepared that report, it became apparent that there are also telemedicine studies among non-Medicare beneficiaries--e.g., children and pregnant women--that could inform policymakers and provide more comprehensive evidence of the state of the science regarding telemedicine applications. In addition, the first evidence report only partially included a class of telemedicine applications (called self-monitoring/testing telemedicine) in which the beneficiary used a home computer or modern-driven telephone system to either report information or access information and support from Internet resources and indirectly interact with a clinician. Self-monitoring/testing applications in the first report required direct interaction with a clinician. The goal of this report is to systematically review the evidence in the clinical areas of pediatric and obstetric telemedicine as well as home-based telemedicine where there is indirect involvement of the health care professional. (In this report, we will refer to the latter as clinician-indirect home telemedicine.) Specifically, the report summarizes scientific evidence on the diagnostic accuracy, access, clinical outcomes, satisfaction, and cost-effectiveness of services provided by telemedicine technologies for these patient groups. It also identifies gaps in the evidence and makes recommendations for evaluating telemedicine services for these populations in the future. The evidence is clustered according to three categories of telemedicine service defined in our original report: store-and-forward, self-monitoring/testing, and clinician-interactive services. The three clinical practice areas reviewed in this report are defined as follows. The term pediatric applies to any telemedicine study in which the sample consisted wholly or partially of persons aged 18 or younger, including studies with neonatal samples. The term obstetric applies to any telemedicine study in which the sample consisted entirely of women seeking pregnancy-related care. The term clinician-indirect home telemedicine applies to home-based telemedicine (called self-monitoring/testing in our original report) where a telemedicine application used in the home has only indirect involvement by the health care professional. Interactive home telemedicine was applied in this report to all patient populations. KEY QUESTIONS: The key questions that served as a guide for reviewing the literature in the evaluation of pediatric, obstetric, and clinician-indirect home telemedicine applications were derived by consensus among the evidence-review team based on the analytic framework established for the original evidence report. For the current report, the questions were applied to studies in all three practice areas as a whole group within each of the three categories of telemedicine services: store-and-forward; self-monitoring/testing; and clinician-interactive. The specific key questions were: 1. Does telemedicine result in comparable diagnosis and appropriateness of recommendations for management? 2. Does the availability of telemedicine provide comparable access to care? 3. Does telemedicine result in comparable health outcomes? 4. Does telemedicine result in comparable patient or clinician satisfaction with care? 5. Does telemedicine result in comparable costs of care and/or cost-effectiveness? METHODS: We searched for peer-reviewed literature using several bibliographic databases. In addition, we conducted hand searches of leading telemedicine journals and identified key papers from the reference lists of journal articles. For our original evidence report on telemedicine for the Medicare population, we designed a search to find any publications about telemedicine and used it to search the MEDLINE, CINAHL, and HealthSTAR databases for all years the databases were available. Through this process, we captured studies of pediatric, obstetric, and clinician-indirect home telemedicine; however, they were excluded from the original report since they were outside its scope. For this supplemental report, we reviewed our original search results and identified studies relevant to this report. We identified additional studies from the reference lists of included papers and from hand searching two peer-reviewed telemedicine publications, the Journal of Telemedicine and Telecare and Telemedicine Journal. We critically appraised the included studies for each study area and key question and discussed the strengths and limitations of the most important studies at weekly meetings of the research team. We also developed recommendations for research to address telemedicine knowledge gaps. To match these gaps with the capabilities of specific research methods, we classified the telemedicine services according to the type of evidence that would be needed to determine whether the specific goals of covering such services had been met. We emphasized the relationship between the type and level of evidence found in the systematic review of effectiveness and the types of studies that might be funded to address the gaps in knowledge in this growing field of research. FINDINGS: We identified a total of 28 eligible studies. In the new clinical areas, we found few studies in store-and-forward telemedicine. There is some evidence of comparable diagnosis and management decisions made using store-and-forward telemedicine from the areas of pediatric dental screening, pediatric ophthalmology, and neonatalogy. In self-monitoring/testing telemedicine for the areas of pediatrics, obstetrics, and clinician-indirect home telemedicine, there is evidence that access to care can be improved when patients and families have the opportunity to receive telehealth care at home rather than in-person care in a clinic or hospital. Access is particularly enhanced when the telehealth system enables timely communication between patients or families and care providers that allows self-management and necessary adjustments that may prevent hospitalization. There is some evidence that this form of telemedicine improves health outcomes, but the study sample sizes are usually small, and even when they are not, the treatment effects are small. There is also some evidence for the efficacy of clinician-interactive telemedicine, but the studies do not clearly define which technologies provide benefit or cost-efficiency. Some promising areas for diagnosis include emergency medicine, psychiatry, and cardiology. Most of the studies measuring access to care provide evidence that it is improved. Although none of these studies were randomized controlled trials, they provide some evidence of access improvement over prior conditions. Clinician-interactive telemedicine was the only area for which any cost studies were found. The three cost studies did not adequately demonstrate that telemedicine reduces costs of care (except comparing only selected costs). No study addressed cost-effectiveness. CONCLUSIONS: This supplemental report covering the areas of pediatrics, obstetrics, and indirect-clinician home telemedicine echoes the findings of our initial report for the Medicare domain, which is that while the use of telemedicine is small but growing, the evidence for its efficacy is incomplete. Many of the studies are small and/or methodologically limited, so it cannot be determined whether telemedicine is efficacious. Future studies should focus on the use of telemedicine in conditions where burden of illness and/or barriers to access for care are significant. Use of recent innovations in the design of randomized controlled trials for emerging technologies would lead to higher quality studies. Journals publishing telemedicine evaluation studies must set high standards for methodologic quality so that evidence reports need not rely on studies with marginal methodologies.

Factors associated with successful answering of clinical questions using an information retrieval system.

OBJECTIVES: Despite the growing use of online databases by clinicians, there has been very little research documenting how effectively they are used. This study assessed the ability of medical and nurse-practitioner students to answer clinical questions using an information retrieval system. It also attempted to identify the demographic, experience, cognitive, personality, search mechanics, and user-satisfaction factors associated with successful use of a retrieval system. METHODS: Twenty-nine students completed questionnaires of clinical and computer experience as well as tests of cognitive abilities and personality type. They were then administered three clinical questions to answer in a medical library setting using the MEDLINE database and electronic and print full-text resources. RESULTS: Medical students were able to answer more questions correctly than nurse-practitioner students before and after searching, but both had comparable improvements in the number of correct questions before and after searching. Successful ability to answer questions was also associated with having experience in literature searching and higher standardized test-score percentiles. CONCLUSIONS: Medical and nurse-practitioner students obtained comparable benefits in the ability to answer clinical questions from use of the information retrieval system. Future research must examine strategies that improve successful search and retrieval of clinical questions posed by clinicians in practice.

Integration and beyond: linking information from disparate sources and into workflow.

The vision of integrating information-from a variety of sources, into the way people work, to improve decisions and process-is one of the cornerstones of biomedical informatics. Thoughts on how this vision might be realized have evolved as improvements in information and communication technologies, together with discoveries in biomedical informatics, and have changed the art of the possible. This review identified three distinct generations of "integration" projects. First-generation projects create a database and use it for multiple purposes. Second-generation projects integrate by bringing information from various sources together through enterprise information architecture. Third-generation projects inter-relate disparate but accessible information sources to provide the appearance of integration. The review suggests that the ideas developed in the earlier generations have not been supplanted by ideas from subsequent generations. Instead, the ideas represent a continuum of progress along the three dimensions of workflow, structure, and extraction.

Perceptions of house officers who use physician order entry.

OBJECTIVE: Describe the perceptions of housestaff physicians about their experience using computerized physician order entry (POE) in hospitals. METHODS: Qualitative study using data from participant observation, focus groups, and both formal and informal interviews. Data were analyzed by three researchers using a grounded approach to identify patterns and themes in the texts. RESULTS: Six themes were identified, including housestaff education, benefits of POE, problems with POE, feelings about POE, implementation strategies, and the future of POE. CONCLUSION: House officers felt that POE assists patient care but may undermine education. They found that POE works best when tailored to fit local and individual workflow. Implementation strategies should include mechanisms for engaging housestaff in the decision process.

Filtering Web pages for quality indicators: an empirical approach to finding high quality consumer health information on the World Wide Web.

The World Wide Web is an increasingly popular source for consumer health information, but many authors have expressed concerns about the quality of health information present on the Internet. We have developed a prototype system that responds to a consumer health query by returning a list of Web pages that are ranked according to the likely quality of the page contents. A computer program identifies some of the criteria that have been suggested for assessing the quality of health information on the Internet. It also identifies characteristics that may serve as proxies for desirable (or undesirable) qualities that are difficult to assess directly using an algorithm. Intervening in the search process and automatically analyzing the contents of each page returned by a general search engine may facilitate the search for high quality consumer health information on the Web.

How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review.

OBJECTIVE: Despite the proliferation of electronic information retrieval (IR) systems for physicians, their effectiveness has not been well assessed. The purpose of this review is to provide a conceptual framework and to apply the results of previous studies to this framework. DATA SOURCES: All sources of medical informatics and information science literature, including MEDLINE, along with bibliographies of textbooks in these areas, were searched from 1966 to January 1998. STUDY SELECTION: All articles presenting either classifications of evaluation studies or their results, with an emphasis on those studying use by physicians. DATA EXTRACTION: A framework for evaluation was developed, consisting of frequency of use, purpose of use, user satisfaction, searching utility, search failure, and outcomes. All studies were then assessed based on the framework. DATA SYNTHESIS: Due to the heterogeneity and simplistic study designs, no meta-analysis of studies could be done. General conclusions were drawn from data where appropriate. A total of 47 articles were found to include an evaluation component and were used to develop the framework. Of these, 21 articles met the inclusion criteria for 1 or more of the categories in the framework. Most use of IR systems by physicians still occurs with bibliographic rather than full-text databases. Overall use of IR systems occurs just 0.3 to 9 times per physician per month, whereas physicians have 2 unanswered questions for every 3 patients. CONCLUSIONS: Studies comparing IR systems with different searching features have not shown that advanced searching methods are significantly more effective than simple text word methods. Most searches retrieve only one fourth to one half of the relevant articles on a given topic and, once retrieved, little is known about how these articles are interpreted or applied. These studies imply that further research and development are needed to improve system utility and performance.

Developing search strategies for detecting high quality reviews in a hypertext test collection.

OBJECTIVE: To identify search strategies for retrieving high quality review studies about etiology, prognosis, therapy, and diagnosis from World Wide Web (WWW) medical documents. DESIGN: Observational study of the performance of search strategies based on terms found in high quality review articles in a collection of hypertext medical documents from the WWW. MEASUREMENTS: The sensitivity and specificity of search strategies for review articles in general and with a specific focus were determined by comparison to a manual review of a collection of hypertext medical documents. RESULTS: A total of 1058 hypertext medical documents from seven governmental and academic WWW sites were included in the study collection. About 16% of the documents in the collection met the criteria for high quality review documents. Search strategies for review documents were identified that had 87% sensitivity and 95% specificity. Compared to simple strategies combining the term "review" and the article focus, more complex strategies based on terms found in high quality studies were more sensitive in identifying review articles of a given focus. These more complex strategies had a sensitivity of 83% for diagnosis, 85% for therapy, 79% for prognosis, and 88% for etiology, while the simple strategies had a sensitivity of 88%, 74%, 38%, and 46%, respectively. In addition, the more complex strategies were more specific for high quality review articles on diagnosis and therapy. CONCLUSION: Search strategies can be identified that enhance retrieval of review documents and review documents of specific focus from a collection of WWW hypertext medical documents.

SAPHIRE International: a tool for cross-language information retrieval.

The world's foremost medical literature is written in English, yet much of the world does not speak English as a primary language. This has led to increasing research interest in cross-language information retrieval, where textual databases are queried in languages other than the one in which they are written. We describe enhancements to the SAPHIRE concept-retrieval system, which maps free-text documents and queries to concepts in the UMLS Metathesaurus, that allow it to accept text input and provide Metathesaurus concept output in any of six languages: English, German, French, Russian, Spanish, and Portuguese. An example of the use of SAPHIRE International is shown in the CliniWeb catalogue of clinically-oriented Web pages. A formative evaluation of German terms shows that additional work is required in handling plural and other suffix variants as well as expanding the breadth of synonyms in the UMLS Metathesaurus.

Automatic prediction of trauma registry procedure codes from emergency room dictations.

Current natural language processing techniques for recognition of concepts in the electronic medical record have been insufficient to allow their broad use for coding information automatically. We have undertaken a preliminary investigation into the use of machine learning methods to recognize procedure codes from emergency room dictations for a trauma registry. Our preliminary results indicate moderate success, and we believe future enhancements with additional learning techniques and selected natural language processing approaches will be fruitful.