Improving the Reliability of Peer Review Without a Gold Standard.

Peer review plays a crucial role in accreditation and credentialing processes as it can identify outliers and foster a peer learning approach, facilitating error analysis and knowledge sharing. However, traditional peer review methods may fall short in effectively addressing the interpretive variability among reviewing and primary reading radiologists, hindering scalability and effectiveness. Reducing this variability is key to enhancing the reliability of results and instilling confidence in the review process. In this paper, we propose a novel statistical approach called "Bayesian Inter-Reviewer Agreement Rate" (BIRAR) that integrates radiologist variability. By doing so, BIRAR aims to enhance the accuracy and consistency of peer review assessments, providing physicians involved in quality improvement and peer learning programs with valuable and reliable insights. A computer simulation was designed to assign predefined interpretive error rates to hypothetical interpreting and peer-reviewing radiologists. The Monte Carlo simulation then sampled (100 samples per experiment) the data that would be generated by peer reviews. The performances of BIRAR and four other peer review methods for measuring interpretive error rates were then evaluated, including a method that uses a gold standard diagnosis. Application of the BIRAR method resulted in 93% and 79% higher relative accuracy and 43% and 66% lower relative variability, compared to "Single/Standard" and "Majority Panel" peer review methods, respectively. Accuracy was defined by the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. Variability was defined by the 95% CI around the median difference of Monte Carlo simulations between measured and pre-defined "actual" interpretive error rates. BIRAR is a practical and scalable peer review method that produces more accurate and less variable assessments of interpretive quality by accounting for variability within the group's radiologists, implicitly applying a standard derived from the level of consensus within the group across various types of interpretive findings.

Automated CT scoring of airway diseases: preliminary results.

RATIONALE AND OBJECTIVES: The aim of this study was to retrospectively evaluate an automated global scoring system for evaluating the extent and severity of disease in a known cohort of patients with documented bronchiectasis. On the basis of a combination of validated three-dimensional automated algorithms for bronchial tree extraction and quantitative airway measurements, global scoring combines the evaluation of bronchial lumen-to-artery ratios and bronchial wall-to-artery ratios, as well as the detection of mucoid-impacted airways. The result is an automatically generated global computed tomographic (CT) score designed to simplify and standardize the interpretation of scans in patients with chronic airway infections. MATERIALS AND METHODS: Twenty high-resolution CT data sets were used to evaluate an automated CT scoring method that combines algorithms for airway quantitative analysis that have been individually tested and validated. Patients with clinically documented atypical mycobacterial infections with visually assessed CT evidence of bronchiectasis varying from mild to severe were retrospectively selected. These data sets were evaluated by two independent experienced radiologists and by computer scoring, with the results compared statistically, including Spearman's rank correlation. RESULTS: Computer evaluation required 3 to 5 minutes per data set, compared to 12 to 15 minutes for manual scoring. Initial Spearman's rank tests showed positive correlations between automated and readers' global scores (r = 0.609, P = .01), extent of bronchiectasis (r = 0.69, P = .0004), and severity of bronchiectasis (r = 0.61, P = .01), while mucus plug detection showed a lesser extent of positive correlation between the scoring methods (r = 0.42, P = .07) and wall thickness a negative weak correlation (r = -0.10, P = .40). Further retrospective review of 24 lobes in which wall thickness scores showed the highest discrepancy between manual and automated methods was then performed, using electronic calipers and perpendicular cross-sections to reassess airway measurements. This resulted in an improved Spearman's rank correlation to r = 0.62 (P = .009), for a global score of r = 0.67 (P = .001). CONCLUSION: Automated computerized scoring shows considerable promise for providing a standardized, quantitative method, demonstrating overall good correlation with the results of experienced readers' evaluation of the extent and severity of bronchiectasis. It is speculated that this technique may also be applicable to a wide range of other conditions associated with chronic bronchial inflammation, as well as of potential value for monitoring response to therapy in these same populations.

Computer-aided diagnosis of the airways: beyond nodule detection.

Although to date, the major impetus for the development of computer-assisted diagnosis (CAD) has been the detection of pulmonary nodules, CAD should properly be viewed as a potential tool for assisting radiologic interpretation of the entire gamut of chest diseases, including not just enhanced detection of disease but also characterization and quantification, ideally leading to improved patient management. The use of CAD to improve visualization of the airways using advanced computer techniques, including sophisticated methods for obtaining 3-dimensional segmentation of the central airways and, in particular, the development of virtual bronchoscopy has been recently studied. In this paper, the authors review the development of a specific series of CAD applications enabling automated identification and characterization of chronically inflamed airways. The advantages to the use of computer methodologies to quantify peripheral airway disease include reproducible visualization methods to display the location, severity, and extent of airway dilatation, bronchial wall thickening, and the presence of mucoid impacted airways. Currently, a number of semiquantitative global scoring systems have been proposed to assess disease extent and severity in patients with bronchiectasis. Unfortunately, with the exception of patients with cystic fibrosis, these are rarely if ever employed, largely owing to the considerable inconvenience of measuring individual airway dimensions and computing a global score. It is apparent that for this specific purpose, CAD may be ideally suited. Automated staging allows for more complete assessment of the entire bronchial tree while providing improved standardization and eliminating an otherwise tedious and time-consuming task.

Boundary-specific cost functions for quantitative airway analysis.

Computed tomography (CT) images of the lungs provide high resolution views of the airways. Quantitative measurements such as lumen diameter and wall thickness help diagnose and localize airway diseases, assist in surgical planning, and determine progress of treatment. Automated quantitative analysis of such images is needed due to the number of airways per patient. We present an approach involving dynamic programming coupled with boundary-specific cost functions that is capable of differentiating inner and outer borders. The method allows for precise delineation of the inner lumen and outer wall. The results are demonstrated on synthetic data, evaluated on human datasets compared to human operators, and verified on phantom CT scans to sub-voxel accuracy.