Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers

OBJECTIVE: To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. MATERIALS AND METHODS: PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals. RESULTS: Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals. CONCLUSION: Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

Clinical Validation of the Psychotic Depression Assessment Scale, Hamilton Depression Rating Scale-6, and Brief Psychiatric Rating Scale-5: Results from the Clinical Research Center for Depression Study

OBJECTIVE: The aim of this study was to validate the psychotic depression assessment scale (PDAS), which includes the six-item melancholia subscale from the Hamilton depression rating scale (HAMD-6) and the five-item psychosis subscale from the brief psychiatric rating scale (BPRS-5). Data from the Clinical Research Center for Depression (CRESCEND) study, which is a 52-week naturalistic trial, were analyzed. METHODS: Fifty-two patients with psychotic depression from the CRESCEND study met our inclusion criteria. The patients underwent the following psychometric assessments: the PDAS, including HAMD-6 and BPRS-5, the clinical global impression scales, the HAMD, the positive symptom subscale, and the negative symptom subscale. Assessments were performed at the baseline and then at weeks 1, 2, 4, 8, 12, 24, and 52. Spearman correlation analyses were used to assess the clinical validity and responsiveness of the PDAS. RESULTS: The clinical validity and responsiveness of the PDAS, including HAMD-6 and BPRS-5, were acceptable, with the exception of the clinical responsiveness of the PDAS for positive symptoms and the clinical responsiveness of BPRS-5 for negative symptoms. CONCLUSION: The clinical relevance of the PDAS has been confirmed and this clinical validation will enhance its clinical utility and availability.

Clinical Validation of AdvanSure GenoBlot Assay as Primary Screening and Test of Cure for Human Papillomavirus Infection

BACKGROUND: Clinical specificity and sensitivity are essential factors in the adoption of a human papillomavirus (HPV) test as a primary screening tool and test of cure after treatment of cervical cancer and precancerous lesions (High-Risk-Lesion). Using histologically-confirmed High-Risk-Lesion-patient specimens with postoperative follow-ups, we performed clinical validation of the AdvanSure GenoBlot Assay (GenoBlot; LG Life Sciences, Korea). METHODS: The study population included 100 cases with High-Risk-Lesion, 96 with high-risk genotype positive and cervical intraepithelial neoplasia (CIN) 1 or better, and 39 with HR-negative and better than CIN 1. Forty-eight High-Risk-Lesion cases received follow-up HPV exams after surgery. For validation as a test of cure, 48 preoperative specimens (PreOP) and 78 postoperative specimens (PostOP) from 48 subjects were separately analyzed. The results of HPV DNA chip tests (HPVDNAChip; BioMedLab Co., Korea) and sequencing were cross-compared. RESULTS: The concordance rates for each genotype between HPVDNAChip and GenoBlot were between 96.3-100%. The accuracy of HPVDNAChip and GenoBlot was 87.9% and 96.6%, respectively. Genotype-based specificity for High-Risk-Lesion detection was higher than 87% for both assays; genotype 16 showed the highest sensitivity. In the PostOP group, the positive rates for HPVDNAChip and GenoBlot were 30.8% and 47.4%, respectively. CONCLUSIONS: GenoBlot showed a higher positive rate than HPVDNAChip for each genotype, with concordance rate and accuracy being similar to previous reports. As a test of cure, GenoBlot performed better than the HPVDNAChip.

Detección de virus papiloma humano en la prevención del cáncer cérvico-uterino / Human papilomavirus detection in cervical cancer prevention

El cáncer cérvico-uterino (CCU), que está fuertemente asociado a la infección por virus papiloma humano de alto riesgo (VPH-AR), sigue siendo un problema de salud pública en Latinoamérica. El uso de la citología para la detección de lesiones pre-cancerosas no ha tenido mayor impacto en las tasas de incidencia y mortalidad del CCU, que aún se mantienen altas en la región. La disponibilidad de nuevas técnicas de tamizaje para la detección de lesiones pre-cancerosas y de vacunas altamente eficaces que previenen casi todas las lesiones relacionadas con los VPH-AR de alto potencial oncogénico VPH 16 y 18, en mujeres no expuestas previamente al virus brindan una gran oportunidad para la prevención del CCU. La detección de VPH-AR representa actualmente un valioso componente de las guías clínicas para el tamizaje, manejo y tratamiento del CCU y sus lesiones precursoras. Se han desarrollado estrategias metodológicas que detectan un amplio espectro de tipos de VPH-AR; sin embargo, solo un pequeño subgrupo de ellas ha documentado la validación clínica para cualquiera de las indicaciones habituales de la detección de estos virus. Las pruebas de VPH que no estén validadas y que no hayan demostrado confiabilidad, reproducibilidad y exactitud no deben ser usadas en el manejo clínico. Una vez incorporada una prueba de VPH en el laboratorio, es esencial que el procedimiento completo sea sometido a un continuo y riguroso control de calidad para evitar prácticas subóptimas, potencialmente dañinas. Este artículo discute los recientes progresos y el estado actual de estos métodos.

Cervical cancer (CC), which is strongly associated to high-risk human papillomavirus (hr-HPV) infection, continues being a significant health problem in Latin America. The use of conventional cytology to detect precancerous cervical lesions has had no major impact on reducing CC incidence and mortality rates, which are still high in the region. New screening tools to detect precancerous lesions became available, which provide great opportunities for CC prevention, as do highly efficacious HPV vaccines able to prevent nearly all lesions associated with HPV-16 and -18 when applied before viral exposure. Currently, hr-HPV testing represents an invaluable component of clinical guidelines for screening, management and treatment of CC and their precursor lesions. Many testing strategies have been developed that can detect a broad spectrum of hr-HPV types in a single assay; however, only a small subset of them has documented clinical performance for any of the standard HPV testing indications. HPV tests that have not been validated and lack proof of reliability, reproducibility and accuracy should not be used in clinical management. Once incorporated into the lab, it is essential to submit the whole procedure of HPV testing to continuous and rigorous quality assurance to avoid sub-optimal, potentially harmful practices. Recent progress and current status of these methods are discussed in this article.

Diagnóstico de débito cardíaco diminuído: validação clínica no pós-operatório de cirurgia cardíaca / Diagnóstico de débito cardíaco diminuído: validação clínica no pós-operatório de cirurgia cardíaca

O objetivo deste estudo foi observar o comportamento das características definidoras do diagnóstico de débito cardíaco diminuído. Procedimentos de análise e síntese reduziram 86 possíveis características definidoras de débito cardíaco diminuído identificadas na literatura a 42 variáveis para o estudo clínico, das quais 32 constituíram as variáveis independentes. A presença ou não das CD foi estudada pelo próprio pesquisador no momento da mensuração do débito cardíaco por termodiluição em 49 pacientes em pós-operatório de cirurgias cardíacas eletivas, com circulação extracorpórea. Os pacientes formaram dois grupos: um com índice cardíaco diminuído ('menor OU igual' 2,5 l/min/metros quadrados) e outro com índice cardíaco não diminuído (maior 2,5 l/min/metros quadrados). Dos 49 pacientes 29 (59,2%) eram mulheres, a idade variou de 29 a 82 anos (média = 63,4). Foram aplicados testes estatísticos não-paramétricos para analisar as associações entre as características definidoras e o índice cardíaco diminuído ou não. Definiu-se como significante p 'menor OU igual' 0,05. Das 32 características definidoras estudadas como variáveis independentes, pulso periférico filiforme e diminuição da perfusão periférica foram mais freqüentes nos pacientes com índice cardíaco diminuído (p=0,001 e p=0,002). Oito CD não ocorreram em nenhum doente, uma CD não foi possível ser avaliada e 23 CD não mostraram diferença entre os grupos. Concluiu-se que pulso periférico filiforme e diminuição perfusão periférica foram indicadores importantes de débito cardíaco diminuído na amostra estudada. O fato de 30, das 32 CD estudadas, não serem diferentes entre os dois grupos sugere que esse diagnóstico deve ser extensivamente estudado com diferentes populações

The aim of this study was to observe the behaviour of defining characteristics of diagnosis decreased cardiac output. Procedures from analysis and synthesis reduced from 87 to 42 possible variables for decreased cardiac output defining characteristics on the literature. From this 42 defining characteristics, 32 constituted the independent variables. The presence or not of the defining characteristics was studied by the researcher at the moment of measurement of cardiac output by thermodilution in 49 post-operative heart surgery patients. The patients formed two groups: decreased cardic index: (smaller OR equal 2,5 1/min/meter square) and non decreased cardic index. From 49 patients 29 (59,2%) were women, the age changed 29 and 82 (average = 63,4). Non-parametric statistical tests were performed in order to identify the associations between defining characteristics and the decreased cardiac index or not. The p smaller OR equal 0,05 was defined as significant. Decreased peripheral pulse and decreased peripheral perfusion showed statistically significant results (p equal 0,001 and p equal 0,002), between the 32 defining characteristics studied on the group with decreased cardiac index. Eight defining characteristcs did not occur in any patient. One defining characteristcs was not possible to be evaluated and defining characteristcs did not show differences between groups. We concluded that decreased peripheral pulse and decreased peripheral perfusion important indicatives for decreased cardic output in this sample. The fact that 30 of 32 defining characteristcs studied do not present difference suggests that this diagnosis must be studied extensively with diferent population