Automated annotation and classification of BI-RADS assessment from radiology reports.

The Breast Imaging Reporting and Data System (BI-RADS) was developed to reduce variation in the descriptions of findings. Manual analysis of breast radiology report data is challenging but is necessary for clinical and healthcare quality assurance activities. The objective of this study is to develop a natural language processing (NLP) system for automated BI-RADS categories extraction from breast radiology reports. We evaluated an existing rule-based NLP algorithm, and then we developed and evaluated our own method using a supervised machine learning approach. We divided the BI-RADS category extraction task into two specific tasks: (1) annotation of all BI-RADS category values within a report, (2) classification of the laterality of each BI-RADS category value. We used one algorithm for task 1 and evaluated three algorithms for task 2. Across all evaluations and model training, we used a total of 2159 radiology reports from 18 hospitals, from 2003 to 2015. Performance with the existing rule-based algorithm was not satisfactory. Conditional random fields showed a high performance for task 1 with an F-1 measure of 0.95. Rules from partial decision trees (PART) algorithm showed the best performance across classes for task 2 with a weighted F-1 measure of 0.91 for BIRADS 0-6, and 0.93 for BIRADS 3-5. Classification performance by class showed that performance improved for all classes from Naïve Bayes to Support Vector Machine (SVM), and also from SVM to PART. Our system is able to annotate and classify all BI-RADS mentions present in a single radiology report and can serve as the foundation for future studies that will leverage automated BI-RADS annotation, to provide feedback to radiologists as part of a learning health system loop.

A Taxonomy of Delivery and Documentation Deviations During Delivery of High-Fidelity Simulations.

INTRODUCTION: We developed a taxonomy of simulation delivery and documentation deviations noted during a multicenter, high-fidelity simulation trial that was conducted to assess practicing physicians' performance. Eight simulation centers sought to implement standardized scenarios over 2 years. Rules, guidelines, and detailed scenario scripts were established to facilitate reproducible scenario delivery; however, pilot trials revealed deviations from those rubrics. A taxonomy with hierarchically arranged terms that define a lack of standardization of simulation scenario delivery was then created to aid educators and researchers in assessing and describing their ability to reproducibly conduct simulations. METHODS: Thirty-six types of delivery or documentation deviations were identified from the scenario scripts and study rules. Using a Delphi technique and open card sorting, simulation experts formulated a taxonomy of high-fidelity simulation execution and documentation deviations. The taxonomy was iteratively refined and then tested by 2 investigators not involved with its development. RESULTS: The taxonomy has 2 main classes, simulation center deviation and participant deviation, which are further subdivided into as many as 6 subclasses. Inter-rater classification agreement using the taxonomy was 74% or greater for each of the 7 levels of its hierarchy. Cohen kappa calculations confirmed substantial agreement beyond that expected by chance. All deviations were classified within the taxonomy. CONCLUSIONS: This is a useful taxonomy that standardizes terms for simulation delivery and documentation deviations, facilitates quality assurance in scenario delivery, and enables quantification of the impact of deviations upon simulation-based performance assessment.

Evidence-Based Practice Knowledge, Perceptions, and Behavior: A Multi-Institutional, Cross-Sectional Study of a Population of U.S. Dental Students.

The aim of this study was to help inform faculty and curriculum leaders in academic dental institutions about the knowledge, skills, perceptions, and behavior of an institutionally diverse population of dental students with respect to evidence-based practice (EBP). A survey utilizing the validated Knowledge, Attitudes, Access, and Confidence Evaluation instrument developed by Hendricson et al. was conducted in 2012 with fourth-year dental students at seven geographically dispersed U.S. dental schools. The survey addressed elements of EBP knowledge, attitudes toward EBP, behavior in accessing evidence, and perceptions of competence in statistical analysis. A total of 138 students from the seven schools participated. A slight majority of these students correctly responded to the knowledge of critical appraisal questions. While the students demonstrated positive attitudes about EBP, they did not report high levels of confidence in their critical appraisal skills. The findings also showed that the students accessed various sources of evidence with differing frequencies. The most frequently accessed resources were colleagues, the Internet (excluding Cochrane Database of Systematic Reviews), and textbooks. The results of this study help to identify areas for improvement in EBP education in order to advance dental students' preparation to become evidence-based practitioners.

Building a gold standard to construct search filters: a case study with biomarkers for oral cancer.

OBJECTIVE: To support clinical researchers, librarians and informationists may need search filters for particular tasks. Development of filters typically depends on a "gold standard" dataset. This paper describes generalizable methods for creating a gold standard to support future filter development and evaluation using oral squamous cell carcinoma (OSCC) as a case study. OSCC is the most common malignancy affecting the oral cavity. Investigation of biomarkers with potential prognostic utility is an active area of research in OSCC. The methods discussed here should be useful for designing quality search filters in similar domains. METHODS: The authors searched MEDLINE for prognostic studies of OSCC, developed annotation guidelines for screeners, ran three calibration trials before annotating the remaining body of citations, and measured inter-annotator agreement (IAA). RESULTS: We retrieved 1,818 citations. After calibration, we screened the remaining citations (n = 1,767; 97.2%); IAA was substantial (kappa = 0.76). The dataset has 497 (27.3%) citations representing OSCC studies of potential prognostic biomarkers. CONCLUSIONS: The gold standard dataset is likely to be high quality and useful for future development and evaluation of filters for OSCC studies of potential prognostic biomarkers. IMPLICATIONS: The methodology we used is generalizable to other domains requiring a reference standard to evaluate the performance of search filters. A gold standard is essential because the labels regarding relevance enable computation of diagnostic metrics, such as sensitivity and specificity. Librarians and informationists with data analysis skills could contribute to developing gold standard datasets and subsequent filters tuned for their patrons' domains of interest.

A Prototype for a Hybrid System to Support Systematic Review Teams: A Case Study of Organ Transplantation.

We describe a prototype for a hybrid system designed to reduce the number of citations needed to re-screen (NNRS) by systematic reviewers, where citations include titles, abstracts, and metadata. The system obviates the need for screening the entire set of citations a second time, which is typically done to control human error. The reference set is based on a complex review about organ transplantation (N=10,796 citations). Data were split into 50% training and test sets, randomly stratified for percentage eligible citations. The system consists of a rule-based module and a machine-learning (ML) module. The former substantially reduces the number of negative citations passed to the ML module and improves imbalance. Relative to the baseline, the system reduces classification error (5.6% vs 2.9%) thereby reducing NNRS by 47.3% (300 vs 158). We discuss the implications of de-emphasizing sensitivity (recall) in favor of specificity and negative predictive value to reduce screening burden.

Feature engineering and a proposed decision-support system for systematic reviewers of medical evidence.

OBJECTIVES: Evidence-based medicine depends on the timely synthesis of research findings. An important source of synthesized evidence resides in systematic reviews. However, a bottleneck in review production involves dual screening of citations with titles and abstracts to find eligible studies. For this research, we tested the effect of various kinds of textual information (features) on performance of a machine learning classifier. Based on our findings, we propose an automated system to reduce screeing burden, as well as offer quality assurance. METHODS: We built a database of citations from 5 systematic reviews that varied with respect to domain, topic, and sponsor. Consensus judgments regarding eligibility were inferred from published reports. We extracted 5 feature sets from citations: alphabetic, alphanumeric(+), indexing, features mapped to concepts in systematic reviews, and topic models. To simulate a two-person team, we divided the data into random halves. We optimized the parameters of a Bayesian classifier, then trained and tested models on alternate data halves. Overall, we conducted 50 independent tests. RESULTS: All tests of summary performance (mean F3) surpassed the corresponding baseline, P<0.0001. The ranks for mean F3, precision, and classification error were statistically different across feature sets averaged over reviews; P-values for Friedman's test were .045, .002, and .002, respectively. Differences in ranks for mean recall were not statistically significant. Alphanumeric(+) features were associated with best performance; mean reduction in screening burden for this feature type ranged from 88% to 98% for the second pass through citations and from 38% to 48% overall. CONCLUSIONS: A computer-assisted, decision support system based on our methods could substantially reduce the burden of screening citations for systematic review teams and solo reviewers. Additionally, such a system could deliver quality assurance both by confirming concordant decisions and by naming studies associated with discordant decisions for further consideration.

Are dentists interested in the oral-systemic disease connection? A qualitative study of an online community of 450 practitioners.

BACKGROUND: Dentists in the US see an increasing number of patients with systemic conditions. These patients are challenging to care for when the relationship between oral and systemic disease is not well understood. The prevalence of professional isolation exacerbates the problem due to the difficulty in finding expert advice or peer support. This study aims to identify whether dentists discuss the oral-systemic connection and what aspects they discuss; to understand their perceptions of and attitudes toward the connection; and to determine what information they need to treat patients with systemic conditions. METHODS: We retrieved 14,576 messages posted to the Internet Dental Forum from April 2008 to May 2009. Using natural language processing and human classification, we identified substantive phrases and keywords and used them to retrieve 141messages on the oral-systemic connection. We then conducted coding and thematic analysis to identify recurring themes on the topic. RESULTS: Dentists discuss a variety of topics on oral diseases and systemic health, with the association between periodontal and systemic diseases, the effect of dental materials or procedures on general health, and the impact of oral-systemic connection on practice behaviors as the leading topics. They also disseminate and share research findings on oral and systemic health with colleagues online. However, dentists are very cautious about the nature of the oral-systemic linkage that may not be causal. Nonetheless, they embrace the positive association as a motivating point for patients in practice. When treating patients with systemic conditions, dentists enquire about the cause of less common dental diseases potentially in relation to medical conditions in one-third of the cases and in half of the cases seek clinical guidelines and evidence-based interventions on treating dental diseases with established association with systemic conditions. CONCLUSIONS: Dentists' unmet information needs call for more research into the association between less studied dental conditions and systemic diseases, and more actionable clinical guidelines for well-researched disease connections. To improve dissemination and foster behavioral change, it is imperative to understand what information clinicians need and in which situations. Leveraging peer influence via social media could be a useful strategy to achieve the goal.

Comparative effectiveness research designs: an analysis of terms and coverage in Medical Subject Headings (MeSH) and Emtree.

OBJECTIVES: We analyzed the extent to which comparative effectiveness research (CER) organizations share terms for designs, analyzed coverage of CER designs in Medical Subject Headings (MeSH) and Emtree, and explored whether scientists use CER design terms. METHODS: We developed local terminologies (LTs) and a CER design terminology by extracting terms in documents from five organizations. We defined coverage as the distribution over match type in MeSH and Emtree. We created a crosswalk by recording terms to which design terms mapped in both controlled vocabularies. We analyzed the hits for queries restricted to titles and abstracts to explore scientists' language. RESULTS: Pairwise LT overlap ranged from 22.64% (12/53) to 75.61% (31/41). The CER design terminology (n = 78 terms) consisted of terms for primary study designs and a few terms useful for evaluating evidence, such as opinion paper and systematic review. Patterns of coverage were similar in MeSH and Emtree (gamma = 0.581, P = 0.002). CONCLUSIONS: Stakeholder terminologies vary, and terms are inconsistently covered in MeSH and Emtree. The CER design terminology and crosswalk may be useful for expert searchers. For partially mapped terms, queries could consist of free text for modifiers such as nonrandomized or interrupted added to broad or related controlled terms.

Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers.

OBJECTIVES: To investigate whether (1) machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers; (2) classifier performance varies with optimization; and (3) the number of citations to screen can be reduced. METHODS: We used an open-source, data-mining suite to process and classify biomedical citations that point to mostly nonrandomized studies from 2 systematic reviews. We built training and test sets for citation portions and compared classifier performance by considering the value of indexing, various feature sets, and optimization. We conducted our experiments in 2 phases. The design of phase I with no optimization was: 4 classifiers × 3 feature sets × 3 citation portions. Classifiers included k-nearest neighbor, naïve Bayes, complement naïve Bayes, and evolutionary support vector machine. Feature sets included bag of words, and 2- and 3-term n-grams. Citation portions included titles, titles and abstracts, and full citations with metadata. Phase II with optimization involved a subset of the classifiers, as well as features extracted from full citations, and full citations with overweighted titles. We optimized features and classifier parameters by manually setting information gain thresholds outside of a process for iterative grid optimization with 10-fold cross-validations. We independently tested models on data reserved for that purpose and statistically compared classifier performance on 2 types of feature sets. We estimated the number of citations needed to screen by reviewers during a second pass through a reduced set of citations. RESULTS: In phase I, the evolutionary support vector machine returned the best recall for bag of words extracted from full citations; the best classifier with respect to overall performance was k-nearest neighbor. No classifier attained good enough recall for this task without optimization. In phase II, we boosted performance with optimization for evolutionary support vector machine and complement naïve Bayes classifiers. Generalization performance was better for the latter in the independent tests. For evolutionary support vector machine and complement naïve Bayes classifiers, the initial retrieval set was reduced by 46% and 35%, respectively. CONCLUSIONS: Machine learning classifiers can help identify nonrandomized studies eligible for full-text screening by systematic reviewers. Optimization can markedly improve performance of classifiers. However, generalizability varies with the classifier. The number of citations to screen during a second independent pass through the citations can be substantially reduced.

Using natural language processing to enable in-depth analysis of clinical messages posted to an Internet mailing list: a feasibility study.

BACKGROUND: An Internet mailing list may be characterized as a virtual community of practice that serves as an information hub with easy access to expert advice and opportunities for social networking. We are interested in mining messages posted to a list for dental practitioners to identify clinical topics. Once we understand the topical domain, we can study dentists' real information needs and the nature of their shared expertise, and can avoid delivering useless content at the point of care in future informatics applications. However, a necessary first step involves developing procedures to identify messages that are worth studying given our resources for planned, labor-intensive research. OBJECTIVES: The primary objective of this study was to develop a workflow for finding a manageable number of clinically relevant messages from a much larger corpus of messages posted to an Internet mailing list, and to demonstrate the potential usefulness of our procedures for investigators by retrieving a set of messages tailored to the research question of a qualitative research team. METHODS: We mined 14,576 messages posted to an Internet mailing list from April 2008 to May 2009. The list has about 450 subscribers, mostly dentists from North America interested in clinical practice. After extensive preprocessing, we used the Natural Language Toolkit to identify clinical phrases and keywords in the messages. Two academic dentists classified collocated phrases in an iterative, consensus-based process to describe the topics discussed by dental practitioners who subscribe to the list. We then consulted with qualitative researchers regarding their research question to develop a plan for targeted retrieval. We used selected phrases and keywords as search strings to identify clinically relevant messages and delivered the messages in a reusable database. RESULTS: About half of the subscribers (245/450, 54.4%) posted messages. Natural language processing (NLP) yielded 279,193 clinically relevant tokens or processed words (19% of all tokens). Of these, 2.02% (5634 unique tokens) represent the vocabulary for dental practitioners. Based on pointwise mutual information score and clinical relevance, 325 collocated phrases (eg, fistula filled obturation and herpes zoster) with 108 keywords (eg, mercury) were classified into 13 broad categories with subcategories. In the demonstration, we identified 305 relevant messages (2.1% of all messages) over 10 selected categories with instances of collocated phrases, and 299 messages (2.1%) with instances of phrases or keywords for the category systemic disease. CONCLUSIONS: A workflow with a sequence of machine-based steps and human classification of NLP-discovered phrases can support researchers who need to identify relevant messages in a much larger corpus. Discovered phrases and keywords are useful search strings to aid targeted retrieval. We demonstrate the potential value of our procedures for qualitative researchers by retrieving a manageable set of messages concerning systemic and oral disease.

Barriers to implementing evidence-based clinical guidelines: a survey of early adopters.

OBJECTIVE: The purpose of this study was to identify barriers that early-adopting dentists perceive as common and challenging when implementing recommendations from evidence-based (EB) clinical guidelines. METHOD: This is a cross-sectional study. Dentists who attended the 2008 Evidence-based Dentistry Champion Conference were eligible for inclusion. Forty-three dentists (34%) responded to a 22-item questionnaire administered online. Two investigators independently coded and categorized responses to open-ended items. Descriptive statistics were computed to assess the frequency of barriers and perceived challenges. RESULTS: The most common barriers to implementation were difficulty in changing current practice model, resistance and criticism from colleagues, and lack of trust in evidence or research. Barriers perceived as serious problems had to do with lack of up-to-date evidence, lack of clear answers to clinical questions, and contradictory information in the scientific literature. CONCLUSIONS: Knowledge of barriers will help improve translation of biomedical research for dentists. Information in guidelines needs to be current, clear, and simplified for use at chairside; dentists' fears need to be addressed.

Towards automating the initial screening phase of a systematic review.

Systematic review authors synthesize research to guide clinicians in their practice of evidence-based medicine. Teammates independently identify provisionally eligible studies by reading the same set of hundreds and sometimes thousands of citations during an initial screening phase. We investigated whether supervised machine learning methods can potentially reduce their workload. We also extended earlier research by including observational studies of a rare condition. To build training and test sets, we used annotated citations from a search conducted for an in-progress Cochrane systematic review. We extracted features from titles, abstracts, and metadata, then trained, optimized, and tested several classifiers with respect to mean performance based on 10-fold cross-validations. In the training condition, the evolutionary support vector machine (EvoSVM) with an Epanechnikov or radial kernel is the best classifier: mean recall=100%; mean precision=48% and 41%, respectively. In the test condition, EvoSVM performance degrades: mean recall=77%, mean precision ranges from 26% to 37%. Because near-perfect recall is essential in this context, we conclude that supervised machine learning methods may be useful for reducing workload under certain conditions.

Conceptual biology, hypothesis discovery, and text mining: Swanson's legacy.

Innovative biomedical librarians and information specialists who want to expand their roles as expert searchers need to know about profound changes in biology and parallel trends in text mining. In recent years, conceptual biology has emerged as a complement to empirical biology. This is partly in response to the availability of massive digital resources such as the network of databases for molecular biologists at the National Center for Biotechnology Information. Developments in text mining and hypothesis discovery systems based on the early work of Swanson, a mathematician and information scientist, are coincident with the emergence of conceptual biology. Very little has been written to introduce biomedical digital librarians to these new trends. In this paper, background for data and text mining, as well as for knowledge discovery in databases (KDD) and in text (KDT) is presented, then a brief review of Swanson's ideas, followed by a discussion of recent approaches to hypothesis discovery and testing. 'Testing' in the context of text mining involves partially automated methods for finding evidence in the literature to support hypothetical relationships. Concluding remarks follow regarding (a) the limits of current strategies for evaluation of hypothesis discovery systems and (b) the role of literature-based discovery in concert with empirical research. Report of an informatics-driven literature review for biomarkers of systemic lupus erythematosus is mentioned. Swanson's vision of the hidden value in the literature of science and, by extension, in biomedical digital databases, is still remarkably generative for information scientists, biologists, and physicians.

High-soluble-fiber foods in conjunction with a telephone-based, personalized behavior change support service result in favorable changes in lipids and lifestyles after 7 weeks.

OBJECTIVE: To evaluate whether an intervention of foods high in soluble fiber from psyllium and/or oats plus a telephone-based, personalized behavior change support service improves serum lipids and elicits cholesterol-managing lifestyle changes vs usual care. DESIGN: 7-week randomized, controlled intervention. SUBJECTS/SETTING: 150 moderately hypercholesterolemic men and women, age range 25 to 70 years. INTERVENTION: The intervention group consumed 4 servings/day of high-fiber foods and had weekly telephone conversations with a personal coach who offered support and guidance in making lifestyle changes consistent with the National Cholesterol Education Program's (NCEP) cholesterol-lowering guidelines. The usual care group received a handout describing the NCEP Step-1 diet. MAIN OUTCOME MEASURES: Serum lipids and lipoproteins and self-reported lifestyle changes. STATISTICAL ANALYSES: For physiologic and dietary changes, mixed linear models for repeated measures were applied. Models were simplified using analysis of covariance where age in years was the covariate. Traditional general linear models were used to assess lifestyle changes. RESULTS: In the intervention group total cholesterol (TC) decreased 5.6%, low-density lipoprotein (LDL) cholesterol 7.1%, LDL/high-density lipoprotein (HDL) cholesterol ratio 5.6%, and triglycerides (TG) 14.2% (P<.0167); decreases in TC and LDL were significantly different from the usual care group. In the usual care group TC decreased 1.9%, LDL 1.2%, LDL/HDL 1.9%, and TG 4.4% (all not significant). The intervention group also reported an increase in their knowledge, ability, and confidence to make cholesterol-managing diet and exercise changes compared with the usual care group (P<.05). The intervention group had a greater decrease in energy intake from saturated fat (-1.6%) and increase in soluble fiber intake (7.3%) than the usual care group (P<.05). The intervention group reported an increase in exercise vs the usual care group (P<.05). Both intervention and control groups had a minimal reduction (<1%) in body weight compared with baseline (P<.0167). APPLICATIONS/CONCLUSIONS: A 7-week intervention that includes both functional foods and individualized, interactive support for behavior change could be an effective model for dietitians to use with patients at risk for CVD, pending results of long-term studies.