Intra- and Inter-Rater Agreement Describing Myometrial Lesions Using Morphologic Uterus Sonographic Assessment: A Pilot Study.

OBJECTIVES: To evaluate the intra- and inter-rater agreement for myometrial lesions using Morphologic Uterus Sonographic Assessment terminology. METHODS: Thirteen raters with high (n = 6) or medium experience (n = 7) assessed 30 3-dimensional ultrasound clips with (n = 20) and without (n = 10) benign myometrial lesions. Myometrial lesions were reported as poorly or well defined and then systematically evaluated for the presence of individual features. The clips were blindly assessed twice (at a 2-month interval). Intra- and inter-rater agreements were calculated with κ statistics. RESULTS: The reporting of poorly defined lesions reached moderate intra-rater agreement (κ = 0.49 [high experience] and 0.47 [medium experience]) and poor inter-rater agreement (κ = 0.39 [high experience] and 0.25 [medium experience]). The reporting of well-defined lesions reached good to very good intra-rater agreement (κ = 0.73 [high experience] and 0.82 [medium experience]) and good inter-rater agreement (κ = 0.75 [high experience] and 0.63 [medium experience]). Most individual features associated with ill-defined lesions reached moderate intra- and inter-rater agreement among highly experienced raters (κ = 0.41-0.60). The least reproducible features were myometrial cysts, hyperechoic islands, subendometrial lines and buds, and translesional flow (κ = 0.11-0.34). Most individual features associated with well-defined lesions reached moderate to good intra- and inter-rater agreement among all observers (κ = 0.41-0.80). The least reproducible features were a serosal contour, asymmetry, a hyperechoic rim, and fan-shaped shadows (κ = 0.00-0.35). CONCLUSIONS: The reporting of well-defined lesions showed excellent agreement, whereas the agreement for poorly defined lesions was low, even among highly experienced raters. The agreement on identifying individual features varied, especially for features associated with ill-defined lesions. Guidelines on minimum requirements for features associated with ill-defined lesions to be interpreted as poorly defined lesions may improve agreement.

Clinical data miner: an electronic case report form system with integrated data preprocessing and machine-learning libraries supporting clinical diagnostic model research.

BACKGROUND: Using machine-learning techniques, clinical diagnostic model research extracts diagnostic models from patient data. Traditionally, patient data are often collected using electronic Case Report Form (eCRF) systems, while mathematical software is used for analyzing these data using machine-learning techniques. Due to the lack of integration between eCRF systems and mathematical software, extracting diagnostic models is a complex, error-prone process. Moreover, due to the complexity of this process, it is usually only performed once, after a predetermined number of data points have been collected, without insight into the predictive performance of the resulting models. OBJECTIVE: The objective of the study of Clinical Data Miner (CDM) software framework is to offer an eCRF system with integrated data preprocessing and machine-learning libraries, improving efficiency of the clinical diagnostic model research workflow, and to enable optimization of patient inclusion numbers through study performance monitoring. METHODS: The CDM software framework was developed using a test-driven development (TDD) approach, to ensure high software quality. Architecturally, CDM's design is split over a number of modules, to ensure future extendability. RESULTS: The TDD approach has enabled us to deliver high software quality. CDM's eCRF Web interface is in active use by the studies of the International Endometrial Tumor Analysis consortium, with over 4000 enrolled patients, and more studies planned. Additionally, a derived user interface has been used in six separate interrater agreement studies. CDM's integrated data preprocessing and machine-learning libraries simplify some otherwise manual and error-prone steps in the clinical diagnostic model research workflow. Furthermore, CDM's libraries provide study coordinators with a method to monitor a study's predictive performance as patient inclusions increase. CONCLUSIONS: To our knowledge, CDM is the only eCRF system integrating data preprocessing and machine-learning libraries. This integration improves the efficiency of the clinical diagnostic model research workflow. Moreover, by simplifying the generation of learning curves, CDM enables study coordinators to assess more accurately when data collection can be terminated, resulting in better models or lower patient recruitment costs.