Statistical visualization for assessing performance of methods for safety surveillance using electronic databases.

PURPOSE: The success of an epidemiological study for drug safety surveillance or comparative effectiveness depends largely on design and analysis strategies besides data quality. The Observational Medical Outcomes Partnership (OMOP) methods community implemented a collection of statistical methods with extensive parameters allowing a wide variety of designs and analyses. Our objective was to develop a visualization tool to explore which parameter settings may enable better predictive properties for a given method in a database. METHODS: Performance measures were produced for each setting, including sensitivity (recall), specificity (1-FPR), AUC, MAP, and P(k). Multiple regressions with relevant parameters as main effects were run for performance measures on all test cases and subgroups. Heatmaps with sequential palettes to indicate the parameters' impacts on performance measures were generated based on matrices of the standardized coefficients (t-statistics) by parameter settings and test case subgroups. RESULTS: Heatmaps help researchers to explore design and analysis options of methods for evaluating a variety of drug-outcome relationships and also to explore data issues. CONCLUSIONS: Statistical visualization through heatmaps is a useful tool for summarizing and presenting method performance results and for the exploration of the parameter settings for method performance characteristics and data limitations.

An evaluation of the UMLS in representing corpus derived clinical concepts.

We performed an evaluation of the Unified Medical Language System (UMLS) in representing concepts derived from medical narrative documents from three domains: chest x-ray reports, discharge summaries and admission notes. We detected concepts in these documents by identifying noun phrases (NPs) and N-grams, including unigrams (single words), bigrams (word pairs) and trigrams (word triples). After removing NPs and N-grams that did not represent discrete clinical concepts, we processed the remaining with the UMLS MetaMap program. We manually reviewed the results of MetaMap processing to determine whether MetaMap found full, partial or no representation of the concept. For full representations, we determined whether post-coordination was required. Our results showed that a large portion of concepts found in clinical narrative documents are either unrepresented or poorly represented in the current version of the UMLS Metathesaurus and that post-coordination was often required in order to fully represent a concept.

Comparing methods for identifying pancreatic cancer patients using electronic data sources.

We sought to determine the accuracy of two electronic methods of identifying pancreatic cancer in a cohort of pancreatic cyst patients, and to examine the reasons for identification failure. We used the International Classification of Diseases, 9(th) Edition (ICD-9) codes and natural language processing (NLP) technology to identify pancreatic cancer in these patients. We compared both methods to a human-validated gold-standard surgical database. Both ICD-9 codes and NLP technology achieved high sensitivity for identifying pancreatic cancer, but the ICD-9 code method achieved markedly lower specificity and PPV compared to the NLP method. The NLP method required only slightly greater expenditures of time and effort compared to the ICD-9 code method. We identified several variables influencing the accuracy of ICD-9 codes to identify cancer patients including: the identification algorithm, kind of cancer to be identified, presence of other conditions similar to cancer, and presence of conditions that are precancerous.

An empiric modification to the probabilistic record linkage algorithm using frequency-based weight scaling.

OBJECTIVE: To incorporate value-based weight scaling into the Fellegi-Sunter (F-S) maximum likelihood linkage algorithm and evaluate the performance of the modified algorithm. Background Because healthcare data are fragmented across many healthcare systems, record linkage is a key component of fully functional health information exchanges. Probabilistic linkage methods produce more accurate, dynamic, and robust matching results than rule-based approaches, particularly when matching patient records that lack unique identifiers. Theoretically, the relative frequency of specific data elements can enhance the F-S method, including minimizing the false-positive or false-negative matches. However, to our knowledge, no frequency-based weight scaling modification to the F-S method has been implemented and specifically evaluated using real-world clinical data. METHODS: The authors implemented a value-based weight scaling modification using an information theoretical model, and formally evaluated the effectiveness of this modification by linking 51,361 records from Indiana statewide newborn screening data to 80,089 HL7 registration messages from the Indiana Network for Patient Care, an operational health information exchange. In addition to applying the weight scaling modification to all fields, we examined the effect of selectively scaling common or uncommon field-specific values. RESULTS: The sensitivity, specificity, and positive predictive value for applying weight scaling to all field-specific values were 95.4, 98.8, and 99.9%, respectively. Compared with nonweight scaling, the modified F-S algorithm demonstrated a 10% increase in specificity with a 3% decrease in sensitivity. CONCLUSION: By eliminating false-positive matches, the value-based weight modification can enhance the specificity of the F-S method with minimal decrease in sensitivity.

Participation in a prototype federated system for sharing medical content.

As part of the Markle Foundation's Connecting For Health (CFH) Reference Implementation Project, Regenstrief Institute participated in the design, creation, and public distribution of server and client code that allows for the secure electronic exchange of protected health information between disparate computer systems.