Combining human and machine intelligence for clinical trial eligibility querying.
J Am Med Inform Assoc
; 29(7): 1161-1171, 2022 06 14.
Article
in English
| MEDLINE | ID: covidwho-1795239
ABSTRACT
OBJECTIVE:
To combine machine efficiency and human intelligence for converting complex clinical trial eligibility criteria text into cohort queries. MATERIALS ANDMETHODS:
Criteria2Query (C2Q) 2.0 was developed to enable real-time user intervention for criteria selection and simplification, parsing error correction, and concept mapping. The accuracy, precision, recall, and F1 score of enhanced modules for negation scope detection, temporal and value normalization were evaluated using a previously curated gold standard, the annotated eligibility criteria of 1010 COVID-19 clinical trials. The usability and usefulness were evaluated by 10 research coordinators in a task-oriented usability evaluation using 5 Alzheimer's disease trials. Data were collected by user interaction logging, a demographic questionnaire, the Health Information Technology Usability Evaluation Scale (Health-ITUES), and a feature-specific questionnaire.RESULTS:
The accuracies of negation scope detection, temporal and value normalization were 0.924, 0.916, and 0.966, respectively. C2Q 2.0 achieved a moderate usability score (3.84 out of 5) and a high learnability score (4.54 out of 5). On average, 9.9 modifications were made for a clinical study. Experienced researchers made more modifications than novice researchers. The most frequent modification was deletion (5.35 per study). Furthermore, the evaluators favored cohort queries resulting from modifications (score 4.1 out of 5) and the user engagement features (score 4.3 out of 5). DISCUSSION ANDCONCLUSION:
Features to engage domain experts and to overcome the limitations in automated machine output are shown to be useful and user-friendly. We concluded that human-computer collaboration is key to improving the adoption and user-friendliness of natural language processing.Keywords
Full text:
Available
Collection:
International databases
Database:
MEDLINE
Main subject:
COVID-19
Type of study:
Cohort study
/
Experimental Studies
/
Observational study
/
Prognostic study
/
Randomized controlled trials
Limits:
Humans
Language:
English
Journal:
J Am Med Inform Assoc
Journal subject:
Medical Informatics
Year:
2022
Document Type:
Article
Affiliation country:
Jamia
Similar
MEDLINE
...
LILACS
LIS