Sequestration methodology in practice through evaluation of joint demographic distributions of 54,185 patients in the Medical Imaging and Data Resource Center (MIDRC) data commons

ABSTRACT

The Medical Imaging and Data Resource Center (MIDRC) is a multi-institutional effort to accelerate medical imaging machine intelligence research and create a publicly available data commons as well as a sequestered commons for performance evaluation of algorithms. This work sought to evaluate the currently implemented methodology for apportioning data to the public and sequestered data commons by investigating the resulting distributions of joint demographic characteristics between the public and sequestered commons. 54,185 patients whose de-identified imaging studies and metadata had been submitted to MIDRC were previously separated into public and sequestered commons using a multi-dimensional stratified sampling method, resulting in 41,556 patients (77%) in the public commons and 12,629 patients (23%) in the sequestered commons. To compare the balance obtained in the joint distributions of patient characteristics from use of the developed sequestration method, patients from each commons were separated into bins, representing a unique combination of the demographic variables of COVID-19 status, age, race, and sex assigned at birth. The joint distributions of patients were visualized, and the absolute and percent difference in each bin from an exact 7723 split of the data were calculated. Results indicated 75.9% of bins obtained differences of less than 15 patients, with a median difference of 3.6 from the total data for both public and sequestered commons. Joint distributions of patient characteristics in both the public and sequestered commons closely matched each other as well as that of the total data, indicating the sequestration by stratified sampling method has operated as intended. © 2023 SPIE.