ABSTRACT
BACKGROUND: Clinical practice and epidemiological information aggregation require knowing when, how long, and in what sequence medically relevant events occur. The Temporal Awareness and Reasoning Systems for Question Interpretation (TARSQI) Toolkit (TTK) is a complete, open source software package for the temporal ordering of events within narrative text documents. TTK was developed on newspaper articles. We extended TTK to support medical notes using veterans' affairs (VA) clinical notes and compared it to TTK. METHODS: We used a development set consisting of 200 VA clinical notes to modify and append rules to TTK's time tagger, creating Med-TTK. We then evaluated the performances of TTK and Med-TTK on an independent random selection of 100 clinical notes. Evaluation tasks were to identify and classify time-referring expressions as one of four temporal classes (DATE, TIME, DURATION, and SET). The reference standard for this test set was generated by dual human manual review with disagreements resolved by a third reviewer. Outcome measures included recall and precision for each class, and inter-rater agreement scores. RESULTS: There were 3146 temporal expressions in the reference standard. TTK identified 1595 temporal expressions. Recall was 0.15 (95% confidence interval [CI] 0.12-0.15) and precision was 0.27 (95% CI 0.25-0.29) for TTK. Med-TTK identified 3174 expressions. Recall was 0.86 (95% CI 0.84-0.87) and precision was 0.85 (95% CI 0.84-0.86) for Med-TTK. CONCLUSION: The algorithms for identifying and classifying temporal expressions in medical narratives developed within Med-TTK significantly improved performance compared to TTK. Natural language processing applications such as Med-TTK provide a foundation for meaningful longitudinal mapping of patient history events among electronic health records. The tool can be accessed at the following site: http://code.google.com/p/med-ttk/.
