Clinical validation of digital assessment tools and machine learning models for remote measurement of psoriasis and psoriatic arthritis: a proof-of-concept study.

OBJECTIVE: Psoricatic disease remains underdiagnosed and undertreated. We developed and validated a suite of novel, smartphone sensor-based assessments that can be self-administered to measure cutaneous and musculoskeletal signs and symptoms of psoriatic disease. METHODS: Participants with psoriasis, psoriatic arthritis, or healthy controls were recruited between June 5, 2019, and November 10, 2021, at two academic medical centers. Concordance and accuracy of digital measures and image-based machine learning models were compared to their analogous clinical measures from trained rheumatologists and dermatologists. RESULTS: Of 104 study participants, 51 (49%) were female and 53 (51%) were male, with a mean age of 42.3 years (SD: 12.6). Seventy-nine (76%) participants had psoriatic arthritis, 16 (15.4%) had psoriasis and 9 (8.7%) were healthy controls. Digital patient assessment of percent body surface area (BSA) affected with psoriasis demonstrated very strong concordance (CCC = 0.94, [95%CI = 0.91-0.96]) with physician-assessed BSA. The in-clinic and remote target-lesion Physician Global Assessments showed fair to moderate concordance (CCCerythema=0.72 [0.59-0.85]; CCCinduration=0.72 [0.62-0.82]; CCCscaling=0.60 [0.48-0.72]). Machine learning models of hand photos taken by patients accurately identified clinically-diagnosed nail psoriasis with an accuracy of 0.76. The Digital Jar Open assessment categorized physician-assessed upper extremity involvement, considering joint tenderness or enthesitis (AUROC = 0.68 (0.47-0.85)). CONCLUSION: The Psorcast digital assessments achieved significant clinical validity, although they require further validation in larger cohorts before use in evidence-based medicine or clinical trial settings. The smartphone software and analysis pipelines from the Psorcast suite are open source and freely available.

Remote smartphone monitoring of Parkinson's disease and individual response to therapy.

Remote health assessments that gather real-world data (RWD) outside clinic settings require a clear understanding of appropriate methods for data collection, quality assessment, analysis and interpretation. Here we examine the performance and limitations of smartphones in collecting RWD in the remote mPower observational study of Parkinson's disease (PD). Within the first 6 months of study commencement, 960 participants had enrolled and performed at least five self-administered active PD symptom assessments (speeded tapping, gait/balance, phonation or memory). Task performance, especially speeded tapping, was predictive of self-reported PD status (area under the receiver operating characteristic curve (AUC) = 0.8) and correlated with in-clinic evaluation of disease severity (r = 0.71; P < 1.8 × 10-6) when compared with motor Movement Disorder Society-Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Although remote assessment requires careful consideration for accurate interpretation of RWD, our results support the use of smartphones and wearables in objective and personalized disease assessments.

Smartphone-Based VO2max Measurement With Heart Snapshot in Clinical and Real-world Settings With a Diverse Population: Validation Study.

BACKGROUND: Maximal oxygen consumption (VO2max) is one of the most predictive biometrics for cardiovascular health and overall mortality. However, VO2max is rarely measured in large-scale research studies or routine clinical care because of the high cost, participant burden, and requirement for specialized equipment and staff. OBJECTIVE: To overcome the limitations of clinical VO2max measurement, we aim to develop a digital VO2max estimation protocol that can be self-administered remotely using only the sensors within a smartphone. We also aim to validate this measure within a broadly representative population across a spectrum of smartphone devices. METHODS: Two smartphone-based VO2max estimation protocols were developed: a 12-minute run test (12-MRT) based on distance measured by GPS and a 3-minute step test (3-MST) based on heart rate recovery measured by a camera. In a 101-person cohort, balanced across age deciles and sex, participants completed a gold standard treadmill-based VO2max measurement, two silver standard clinical protocols, and the smartphone-based 12-MRT and 3-MST protocols in the clinic and at home. In a separate 120-participant cohort, the video-based heart rate measurement underlying the 3-MST was measured for accuracy in individuals across the spectrum skin tones while using 8 different smartphones ranging in cost from US $99 to US $999. RESULTS: When compared with gold standard VO2max testing, Lin concordance was pc=0.66 for 12-MRT and pc=0.61 for 3-MST. However, in remote settings, the 12-MRT was significantly less concordant with the gold standard (pc=0.25) compared with the 3-MST (pc=0.61), although both had high test-retest reliability (12-MRT intraclass correlation coefficient=0.88; 3-MST intraclass correlation coefficient=0.86). On the basis of the finding that 3-MST concordance was generalizable to remote settings whereas 12-MRT was not, the video-based heart rate measure within the 3-MST was selected for further investigation. Heart rate measurements in any of the combinations of the six Fitzpatrick skin tones and 8 smartphones resulted in a concordance of pc≥0.81. Performance did not correlate with device cost, with all phones selling under US $200 performing better than pc>0.92. CONCLUSIONS: These findings demonstrate the importance of validating mobile health measures in the real world across a diverse cohort and spectrum of hardware. The 3-MST protocol, termed as heart snapshot, measured VO2max with similar accuracy to supervised in-clinic tests such as the Tecumseh (pc=0.94) protocol, while also generalizing to remote and unsupervised measurements. Heart snapshot measurements demonstrated fidelity across demographic variation in age and sex, across diverse skin pigmentation, and between various iOS and Android phone configurations. This software is freely available for all validation data and analysis code.

Combining accurate tumor genome simulation with crowdsourcing to benchmark somatic structural variant detection.

BACKGROUND: The phenotypes of cancer cells are driven in part by somatic structural variants. Structural variants can initiate tumors, enhance their aggressiveness, and provide unique therapeutic opportunities. Whole-genome sequencing of tumors can allow exhaustive identification of the specific structural variants present in an individual cancer, facilitating both clinical diagnostics and the discovery of novel mutagenic mechanisms. A plethora of somatic structural variant detection algorithms have been created to enable these discoveries; however, there are no systematic benchmarks of them. Rigorous performance evaluation of somatic structural variant detection methods has been challenged by the lack of gold standards, extensive resource requirements, and difficulties arising from the need to share personal genomic information. RESULTS: To facilitate structural variant detection algorithm evaluations, we create a robust simulation framework for somatic structural variants by extending the BAMSurgeon algorithm. We then organize and enable a crowdsourced benchmarking within the ICGC-TCGA DREAM Somatic Mutation Calling Challenge (SMC-DNA). We report here the results of structural variant benchmarking on three different tumors, comprising 204 submissions from 15 teams. In addition to ranking methods, we identify characteristic error profiles of individual algorithms and general trends across them. Surprisingly, we find that ensembles of analysis pipelines do not always outperform the best individual method, indicating a need for new ways to aggregate somatic structural variant detection approaches. CONCLUSIONS: The synthetic tumors and somatic structural variant detection leaderboards remain available as a community benchmarking resource, and BAMSurgeon is available at https://github.com/adamewing/bamsurgeon .

Crowdsourcing biomedical research: leveraging communities as innovation engines.

The generation of large-scale biomedical data is creating unprecedented opportunities for basic and translational science. Typically, the data producers perform initial analyses, but it is very likely that the most informative methods may reside with other groups. Crowdsourcing the analysis of complex and massive data has emerged as a framework to find robust methodologies. When the crowdsourcing is done in the form of collaborative scientific competitions, known as Challenges, the validation of the methods is inherently addressed. Challenges also encourage open innovation, create collaborative communities to solve diverse and important biomedical problems, and foster the creation and dissemination of well-curated data repositories.

Combining tumor genome simulation with crowdsourcing to benchmark somatic single-nucleotide-variant detection.

The detection of somatic mutations from cancer genome sequences is key to understanding the genetic basis of disease progression, patient survival and response to therapy. Benchmarking is needed for tool assessment and improvement but is complicated by a lack of gold standards, by extensive resource requirements and by difficulties in sharing personal genomic information. To resolve these issues, we launched the ICGC-TCGA DREAM Somatic Mutation Calling Challenge, a crowdsourced benchmark of somatic mutation detection algorithms. Here we report the BAMSurgeon tool for simulating cancer genomes and the results of 248 analyses of three in silico tumors created with it. Different algorithms exhibit characteristic error profiles, and, intriguingly, false positives show a trinucleotide profile very similar to one found in human tumors. Although the three simulated tumors differ in sequence contamination (deviation from normal cell sequence) and in subclonality, an ensemble of pipelines outperforms the best individual pipeline in all cases. BAMSurgeon is available at https://github.com/adamewing/bamsurgeon/.

Summary of the DREAM8 Parameter Estimation Challenge: Toward Parameter Identification for Whole-Cell Models.

Whole-cell models that explicitly represent all cellular components at the molecular level have the potential to predict phenotype from genotype. However, even for simple bacteria, whole-cell models will contain thousands of parameters, many of which are poorly characterized or unknown. New algorithms are needed to estimate these parameters and enable researchers to build increasingly comprehensive models. We organized the Dialogue for Reverse Engineering Assessments and Methods (DREAM) 8 Whole-Cell Parameter Estimation Challenge to develop new parameter estimation algorithms for whole-cell models. We asked participants to identify a subset of parameters of a whole-cell model given the model's structure and in silico "experimental" data. Here we describe the challenge, the best performing methods, and new insights into the identifiability of whole-cell models. We also describe several valuable lessons we learned toward improving future challenges. Going forward, we believe that collaborative efforts supported by inexpensive cloud computing have the potential to solve whole-cell model parameter estimation.

Enabling transparent and collaborative computational analysis of 12 tumor types within The Cancer Genome Atlas.

The Cancer Genome Atlas Pan-Cancer Analysis Working Group collaborated on the Synapse software platform to share and evolve data, results and methodologies while performing integrative analysis of molecular profiling data from 12 tumor types. The group's work serves as a pilot case study that provides (i) a template for future large collaborative studies; (ii) a system to support collaborative projects; and (iii) a public resource of highly curated data, results and automated systems for the evaluation of community-developed models.

Systematic analysis of challenge-driven improvements in molecular prognostic models for breast cancer.

Although molecular prognostics in breast cancer are among the most successful examples of translating genomic analysis to clinical applications, optimal approaches to breast cancer clinical risk prediction remain controversial. The Sage Bionetworks-DREAM Breast Cancer Prognosis Challenge (BCC) is a crowdsourced research study for breast cancer prognostic modeling using genome-scale data. The BCC provided a community of data analysts with a common platform for data access and blinded evaluation of model accuracy in predicting breast cancer survival on the basis of gene expression data, copy number data, and clinical covariates. This approach offered the opportunity to assess whether a crowdsourced community Challenge would generate models of breast cancer prognosis commensurate with or exceeding current best-in-class approaches. The BCC comprised multiple rounds of blinded evaluations on held-out portions of data on 1981 patients, resulting in more than 1400 models submitted as open source code. Participants then retrained their models on the full data set of 1981 samples and submitted up to five models for validation in a newly generated data set of 184 breast cancer patients. Analysis of the BCC results suggests that the best-performing modeling strategy outperformed previously reported methods in blinded evaluations; model performance was consistent across several independent evaluations; and aggregating community-developed models achieved performance on par with the best-performing individual models.

Transient transcapillary exchange of water driven by osmotic forces in the heart.

Osmotic transient responses in organ weight after changes in perfusate osmolarity have implied steric hindrance to small-molecule transcapillary exchange, but tracer methods do not. We obtained osmotic weight transient data in isolated, Ringer-perfused rabbit hearts with NaCl, urea, glucose, sucrose, raffinose, inulin, and albumin and analyzed the data with a new anatomically and physicochemically based model accounting for 1) transendothelial water flux, 2) two sizes of porous passages across the capillary wall, 3) axial intracapillary concentration gradients, and 4) water fluxes between myocytes and interstitium. During steady-state conditions approximately 28% of the transcapillary water flux going to form lymph was through the endothelial cell membranes [capillary hydraulic conductivity (Lp) = 1.8 +/- 0.6 x 10-8 cm. s-1. mmHg-1], presumably mainly through aquaporin channels. The interendothelial clefts (with Lp = 4.4 +/- 1.3 x 10-8 cm. s-1. mmHg-1) account for 67% of the water flux; clefts are so wide (equivalent pore radius was 7 +/- 0.2 nm, covering approximately 0.02% of the capillary surface area) that there is no apparent hindrance for molecules as large as raffinose. Infrequent large pores account for the remaining 5% of the flux. During osmotic transients due to 30 mM increases in concentrations of small solutes, the transendothelial water flux was in the opposite direction and almost 800 times as large and was entirely transendothelial because no solute gradient forms across the pores. During albumin transients, gradients persisted for long times because albumin does not permeate small pores; the water fluxes per milliosmolar osmolarity change were 200 times larger than steady-state water flux. The analysis completely reconciles data from osmotic transient, tracer dilution, and lymph sampling techniques.

An integrative model of coupled water and solute exchange in the heart.

Physiologists have devised many models for interpreting water and solute exchange data in whole organs, but the models have typically neglected key aspects of the underlying physiology to present the simplest possible model for a given experimental situation. We have developed a physiologically realistic model of microcirculatory water and solute exchange and applied it to diverse observations of water and solute exchange in the heart. Model simulations are consistent with the results of osmotic weight transient, tracer indicator dilution, and steady-state lymph sampling experiments. The key model features that permit this unification are the use of an axially distributed blood-tissue exchange region, inclusion of a lymphatic drain in the interstitium, and the independent computation of transcapillary solute and solvent fluxes through three different pathways.