Interobserver agreement between eight observers using IOTA simple rules and O-RADS lexicon descriptors for adnexal masses.

PURPOSE: To evaluate interobserver agreement in assigning imaging features and classifying adnexal masses using the IOTA simple rules versus O-RADS lexicon and identify causes of discrepancy. METHODS: Pelvic ultrasound (US) examinations in 114 women with 118 adnexal masses were evaluated by eight radiologists blinded to the final diagnosis (4 attendings and 4 fellows) using IOTA simple rules and O-RADS lexicon. Each feature category was analyzed for interobserver agreement using intraclass correlation coefficient (ICC) for ordinal variables and free marginal kappa for nominal variables. The two-tailed significance level (a) was set at 0.05. RESULTS: For IOTA simple rules, interobserver agreement was almost perfect for three malignant lesion categories (M2-4) and substantial for the remaining two (M1, M5) with k-values of 0.80-0.82 and 0.68-0.69, respectively. Interobserver agreement was almost perfect for two benign feature categories (B2, B3), substantial for two (B4, B5) and moderate for one (B1) with k-values of 0.81-0.90, 0.69-0.70 and 0.60, respectively. For O-RADS, interobserver agreement was almost perfect for two out of ten feature categories (ascites and peritoneal nodules) with k-values of 0.89 and 0.97. Interobserver agreement ranged from fair to substantial for the remaining eight feature categories with k-values of 0.39-0.61. Fellows and attendings had ICC values of 0.725 and 0.517, respectively. CONCLUSION: O-RADS had variable interobserver agreement with overall good agreement. IOTA simple rules had more uniform interobserver agreement with overall excellent agreement. Greater reader experience did not improve interobserver agreement with O-RADS.

Ultrasound Liver Imaging Reporting and Data System (US LI-RADS) Visualization Score: a reliability analysis on inter-reader agreement.

BACKGROUND & AIM: The American College of Radiology Ultrasound Liver Imaging Reporting and Data System (ACR US LI-RADS) Visualization Score conveys the expected level of sensitivity of screening and surveillance ultrasound exams in patients at risk for hepatocellular carcinoma (HCC). We sought to determine inter-reader agreement of the Visualization Score which is currently unknown. METHODS: Consecutive 6998 ultrasound HCC screening and surveillance studies in 3115 patients from 2017 to 2020 were retrospectively retrieved. Of these, 6154 (87.9%) studies were Visualization A (No or minimal limitations), 709 (10.1%) were Visualization B (Moderate limitations), and 135 (1.9%) were Visualization C (Severe limitations). Randomly sampled 90 studies, with 30 studies in each Visualization category, were included for analysis. Nine radiologists (3 senior attendings, 3 junior attendings and 3 body imaging fellows) blinded to the original categorization independently reviewed each study and assigned a Visualization Score. Intraclass correlation coefficient (ICC) was used to quantify inter-reader agreement. RESULTS: ICC among all 9 radiologists was 0.70 (95% CI 0.63-0.77). ICCs among senior attendings, junior attendings and body imaging fellows were 0.68 (CI 0.58-0.76), 0.72 (CI 0.62-0.80) and 0.76 (CI 0.68-0.83), respectively. Subgroup analysis by liver parenchyma was further performed. ICC was highest in the patient group with normal liver parenchyma (0.69, CI 0.56-0.81), followed by steatosis (0.66, CI 0.54-0.79) and cirrhosis (0.58, CI 0.43-0.73), respectively. CONCLUSIONS: US LI-RADS Visualization Score is a reliable tool with good inter-reader agreement that can be used to indicate the expected level of sensitivity of a screening and surveillance ultrasound examination for detecting focal liver observations.