Inter-rater reliability and prospective validation of a clinical prediction rule for SARS-CoV-2 infection.

ABSTRACT

OBJECTIVES: Accurate estimation of the risk of SARS-CoV-2 infection based on bedside data alone has importance to emergency department (ED) operations and throughput. The 13-item CORC (COVID [or coronavirus] Rule-out Criteria) rule had good overall diagnostic accuracy in retrospective derivation and validation. The objective of this study was to prospectively test the inter-rater reliability and diagnostic accuracy of the CORC score and rule (score ≤ 0 negative, > 0 positive) and compare the CORC rule performance with physician gestalt. METHODS: This noninterventional study was conducted at an urban academic ED from February 2021 to March 2021. Two practitioners were approached by research coordinators and asked to independently complete a form capturing the CORC criteria for their shared patient and their gestalt binary prediction of the SARS-CoV-2 test result and confidence (0%-100%). The criterion standard for SARS-CoV-2 was from reverse transcriptase polymerase chain reaction performed on a nasopharyngeal swab. The primary analysis was from weighted Cohen's kappa and likelihood ratios (LRs). RESULTS: For 928 patients, agreement between observers was good for the total CORC score, κ = 0.613 (95% confidence interval [CI] = 0.579-0.646), and for the CORC rule, κ = 0.644 (95% CI = 0.591-0.697). The agreement for clinician gestalt binary determination of SARs-CoV-2 status was κ = 0.534 (95% CI = 0.437-0.632) with median confidence of 76% (first-third quartile = 66-88.5). For 425 patients who had the criterion standard, a negative CORC rule (both observers scored CORC < 0), the sensitivity was 88%, and specificity was 51%, with a negative LR (LR-) of 0.24 (95% CI = 0.10-0.50). Among patients with a mean CORC score of >4, the prevalence of a positive SARS-CoV-2 test was 58% (95% CI = 28%-85%) and positive LR was 13.1 (95% CI = 4.5-37.2). Clinician gestalt demonstrated a sensitivity of 51% and specificity of 86% with a LR- of 0.57 (95% CI = 0.39-0.74). CONCLUSION: In this prospective study, the CORC score and rule demonstrated good inter-rater reliability and reproducible diagnostic accuracy for estimating the pretest probability of SARs-CoV-2 infection.