Interobserver Agreement and Performance of Concurrent AI Assistance for Radiographic Evaluation of Knee Osteoarthritis.

Background Due to conflicting findings in the literature, there are concerns about a lack of objectivity in grading knee osteoarthritis (KOA) on radiographs. Purpose To examine how artificial intelligence (AI) assistance affects the performance and interobserver agreement of radiologists and orthopedists of various experience levels when evaluating KOA on radiographs according to the established Kellgren-Lawrence (KL) grading system. Materials and Methods In this retrospective observer performance study, consecutive standing knee radiographs from patients with suspected KOA were collected from three participating European centers between April 2019 and May 2022. Each center recruited four readers across radiology and orthopedic surgery at in-training and board-certified experience levels. KL grading (KL-0 = no KOA, KL-4 = severe KOA) on the frontal view was assessed by readers with and without assistance from a commercial AI tool. The majority vote of three musculoskeletal radiology consultants established the reference standard. The ordinal receiver operating characteristic method was used to estimate grading performance. Light kappa was used to estimate interrater agreement, and bootstrapped t statistics were used to compare groups. Results Seventy-five studies were included from each center, totaling 225 studies (mean patient age, 55 years ± 15 [SD]; 113 female patients). The KL grades were KL-0, 24.0% (n = 54); KL-1, 28.0% (n = 63); KL-2, 21.8% (n = 49); KL-3, 18.7% (n = 42); and KL-4, 7.6% (n = 17). Eleven readers completed their readings. Three of the six junior readers showed higher KL grading performance with versus without AI assistance (area under the receiver operating characteristic curve, 0.81 ± 0.017 [SEM] vs 0.88 ± 0.011 [P < .001]; 0.76 ± 0.018 vs 0.86 ± 0.013 [P < .001]; and 0.89 ± 0.011 vs 0.91 ± 0.009 [P = .008]). Interobserver agreement for KL grading among all readers was higher with versus without AI assistance (κ = 0.77 ± 0.018 [SEM] vs 0.85 ± 0.013; P < .001). Board-certified radiologists achieved almost perfect agreement for KL grading when assisted by AI (κ = 0.90 ± 0.01), which was higher than that achieved by the reference readers independently (κ = 0.84 ± 0.017; P = .01). Conclusion AI assistance increased junior readers' radiographic KOA grading performance and increased interobserver agreement for osteoarthritis grading across all readers and experience levels. Published under a CC BY 4.0 license. Supplemental material is available for this article.

Constructing a clinical radiographic knee osteoarthritis database using artificial intelligence tools with limited human labor: A proof of principle.

OBJECTIVE: To create a scalable and feasible retrospective consecutive knee osteoarthritis (OA) radiographic database with limited human labor using commercial and custom-built artificial intelligence (AI) tools. METHODS: We applied four AI tools, two commercially available and two custom-built tools, to analyze 6 years of clinical consecutive knee radiographs from patients aged 35-79 at the University of Copenhagen Hospital, Bispebjerg-Frederiksberg Hospital, Denmark. The tools provided Kellgren-Lawrence (KL) grades, joint space widths, patella osteophyte detection, radiographic view detection, knee joint implant detection, and radiographic marker detection. RESULTS: In total, 25,778 knee radiographs from 8575 patients were included in the database after excluding inapplicable radiographs, and 92.5% of the knees had a complete OA dataset. Using the four AI tools, we saved about 800 hours of radiologist reading time and only manually reviewed 16.0% of the images in the database. CONCLUSIONS: This study shows that clinical knee OA databases can be built using AI with limited human reading time for uniform grading and measurements. The concept is scalable temporally and across geographic regions and could help diversify further OA research by efficiently including radiographic knee OA data from different populations globally. We can prevent data dredging and overfitting OA theories on existing trite cohorts by including various gene pools and continuous expansion of new clinical cohorts. Furthermore, the suggested tools and applied approaches provide an ability to retest previous hypotheses and test new hypotheses on real-life clinical data with current disease prevalence and trends.

Should artificial intelligence have lower acceptable error rates than humans?

The first patient was misclassified in the diagnostic conclusion according to a local clinical expert opinion in a new clinical implementation of a knee osteoarthritis artificial intelligence (AI) algorithm at Bispebjerg-Frederiksberg University Hospital, Copenhagen, Denmark. In preparation for the evaluation of the AI algorithm, the implementation team collaborated with internal and external partners to plan workflows, and the algorithm was externally validated. After the misclassification, the team was left wondering: what is an acceptable error rate for a low-risk AI diagnostic algorithm? A survey among employees at the Department of Radiology showed significantly lower acceptable error rates for AI (6.8 %) than humans (11.3 %). A general mistrust of AI could cause the discrepancy in acceptable errors. AI may have the disadvantage of limited social capital and likeability compared to human co-workers, and therefore, less potential for forgiveness. Future AI development and implementation require further investigation of the fear of AI's unknown errors to enhance the trustworthiness of perceiving AI as a co-worker. Benchmark tools, transparency, and explainability are also needed to evaluate AI algorithms in clinical implementations to ensure acceptable performance.

External validation of an artificial intelligence tool for radiographic knee osteoarthritis severity classification.

PURPOSE: To externally validate an artificial intelligence (AI) tool for radiographic knee osteoarthritis severity classification on a clinical dataset. METHOD: This retrospective, consecutive patient sample, external validation study used weight-bearing, non-fixed-flexion posterior-anterior knee radiographs from a clinical production PACS. The index test was ordinal Kellgren-Lawrence grading by an AI tool, two musculoskeletal radiology consultants, two reporting technologists, and two resident radiologists. Grading was repeated by all readers after at least four weeks. Reference test was the consensus of the two consultants. The primary outcome was quadratic weighted kappa. Secondary outcomes were ordinal weighted accuracy, multiclass accuracy and F1-score. RESULTS: 50 consecutive patients between September 24, 2019 and October 22, 2019 were retrospectively included (3 excluded) totaling 99 knees (1 excluded). Quadratic weighted kappa for the AI tool and the consultant consensus was 0.88 CI95% (0.82-0.92). Agreement between the consultants was 0.89 CI95% (0.85-0.93). Intra-rater agreements for the consultants were 0.96 CI95% (0.94-0.98) and 0.94 CI95% (0.91-0.96) respectively. For the AI tool it was 1 CI95% (1-1). For the AI tool, ordinal weighted accuracy was 97.8% CI95% (96.9-98.6 %). Average multiclass accuracy and F1-score were 84% (83/99) CI95% (77-91%) and 0.67 CI95% (0.51-0.81). CONCLUSIONS: The AI tool achieved the same good-to-excellent agreement with the radiology consultant consensus for radiographic knee osteoarthritis severity classification as the consultants did with each other.

Effect of Platelet-Rich Plasma on Nonsurgically Treated Acute Achilles Tendon Ruptures: A Randomized, Double-Blinded Prospective Study.

BACKGROUND: An acute Achilles tendon rupture (ATR) is a long-lasting and devastating injury. Possible biological augmentation to promote and strengthen tendon healing after an ATR would be desirable. PURPOSE: To determine whether the application of a platelet-rich plasma (PRP) injection in nonsurgically treated ATRs may promote healing and thereby improve functional outcomes. STUDY DESIGN: Randomized controlled trial; Level of evidence, 2. METHODS: A total of 40 men (aged 18-60 years) with an ATR incurred within 72 hours were included, and 38 were followed for 12 months. All patients were treated with an orthosis with 3 wedges for 8 weeks; full weightbearing from day 1 was allowed, combined with either 4 PRP or 4 placebo injections (a few drops of saline, <0.5 mL, under the skin) 14 days apart. All patients received the same instructions on an exercise program starting from week 9. Outcomes included the self-reported Achilles tendon Total Rupture Score (ATRS) as well as heel-rise work, heel-rise height, tendon elongation, calf circumference, and ankle dorsiflexion range of motion. RESULTS: The mean ATRS score improved in both groups at all time points (P < .001), but there was no difference between the groups at any time points (12 months: 90.1 points in PRP group and 88.8 points in placebo group). No differences in all functional outcomes at any time points were seen between the groups. At 12 months, the injured leg did not reach normal functional values compared with the uninjured leg. CONCLUSION: The application of PRP in nonsurgically treated ATRs did not appear to show any superior clinical and functional improvement. REGISTRATION: NCT02417922 (ClinicalTrials.gov identifier).