Inter-Rater Reliability and Agreement of Fetal Echocardiography Measurements Among Investigators From Multiple Sites

ABSTRACT

Background: Single institutions specialising in fetal cardiology often have too few cases to develop robust prognostic indicators for specific conditions. The Australia and New Zealand (ANZ) Fetal Cardiology Working Group instigated a multicentre study to examine fetal risk factors for early postnatal intervention in Tetralogy of Fallot (ToF). Centralised data analysis was not possible due to COVID-19 travel restrictions and ethical constraints related to sharing of retrospectively acquired images. A study of inter-observer agreement of standardised in utero cardiac measurements was undertaken to assess the feasibility of combining data from multiple centres. Methods: Ten fetuses with ToF were randomly identified. Deidentified images were distributed securely to ten ANZ fetal cardiologists. The pulmonary valve (PV) annulus, main pulmonary artery (MPA), branch pulmonary arteries (BPA), aortic valve (AV) annulus, ascending aorta (AA), and ductus arteriosus (DA) were measured in triplicate following a defined protocol. Inter-rater reliability was assessed using a two-way random effects model to calculate the intra-class coefficient (ICC). Results: Measurements were available for seven ANZ fetal cardiologists. There was moderate inter-observer agreement for PV (ICC 0.74, 95% CI 0.49–0.91) and AV (0.71, 95% CI 0.48–0.90), and good agreement for MPA (0.81, 95% CI 0.62–0.94) and AA (0.91, 95% CI 0.80–0.97). Inadequate data were available for BPA and DA analysis. Multi-variate analysis found no patient or investigator factors that influenced measurement variability. Conclusion: Fetal cardiac structures can be measured using a defined measurement protocol by multiple investigators with at least moderate agreement. Analysis of large datasets by multiple investigators is a reasonable alternative to centralised data analysis.