The utility of COVID-19 reporting templates for chest radiography (CXR): an Australian single centre study

ABSTRACT

Purpose: Chest radiographs (CXRs) are an important tool for COVID-19 disease management and progression.1 A number of international thoracic/radiology societies have developed structured reporting templates to reduce interpretation variability and measure concordance.2 Recent research into COVID-19 templates have used small data sets, small numbers of radiologists and focused on predictive disease progression. The aim of this study is to evaluate the utility of three (Australian, British, Dutch) CXR qualitative reporting templates in a large Sydney teaching hospital. Methods and Materials Twelve staff specialist radiologists were recruited to participate in the study. Retrospective CXRs between 2020-21 were collected and organized into test sets, containing the CXRs of 50 PCR-positive COVID-19 patients and classified as 'classic/ characteristic' for radiology signs of COVID-19 disease (30 cases) or 'indeterminate' for radiological signs (20 cases) by two expert radiologists. A further 10 CXRs of patients who were imaged for an 'alternative' diagnosis and 10 CXRs that had been reported 'normal' were included. Radiologists were assigned to one of three image sets, with the 70 images randomized and instructed to report the cases using three templates the Royal Australian and New Zealand College of Radiology (RANZCR), British Society of Thoracic Imaging (BSTI) and modified Co-RADS (Dutch)3. Fleiss' Kappa Coefficient was used for agreement between radiologists on each template (inter-reader), and for radiologist's agreement within themselves using the three templates (intra-reader). Result(s) Inter-reader agreement between radiologists was highest for the BSTI template at 0.46 (moderate agreement), followed by RANZCR (0.36) and modified CO-RADS (0.31) (fair agreement). For all templates, agreement was highest for 'normal CXRs" and lowest for 'indeterminate', with moderate or fair agreement for the 'classic' COVID-19 cases (BSTI 0.44;RANZCR 0.31;mod-CO-RADS 0.31). The intra-reader agreement across the 3 templates for 'classic/characteristic' COVID cases was 0.61, for 'normal' cases 0.76 and 'alternative' 0.68 (all at substantial agreement), indicating that radiologists largely were consistent with classification of cases. Conclusion(s) The BSTI template yielded the highest agreement among radiologists for reporting COVID-19 CXRs as well as for 'normal' and 'alternative'. There was a large range of intra-reader agreement for all 4 types of patient presentations, however the level of agreement equal to or higher than 'moderate'. Structured reporting templates have yielded promising results for concordance between radiologists and reliability within radiologist's reporting of CXRs. Further investigation of radiology lexicon within templates is required to seek reasons for variation in concordance, as well as the demographics of readers.