Selection and Reporting of Statistical Methods to Assess Reliability of a Diagnostic Test: Conformity to Recommended Methods in a Peer-Reviewed Journal

OBJECTIVE: To evaluate the frequency and adequacy of statistical analyses in a general radiology journal when reporting a reliability analysis for a diagnostic test. MATERIALS AND METHODS: Sixty-three studies of diagnostic test accuracy (DTA) and 36 studies reporting reliability analyses published in the Korean Journal of Radiology between 2012 and 2016 were analyzed. Studies were judged using the methodological guidelines of the Radiological Society of North America-Quantitative Imaging Biomarkers Alliance (RSNA-QIBA), and COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. DTA studies were evaluated by nine editorial board members of the journal. Reliability studies were evaluated by study reviewers experienced with reliability analysis. RESULTS: Thirty-one (49.2%) of the 63 DTA studies did not include a reliability analysis when deemed necessary. Among the 36 reliability studies, proper statistical methods were used in all (5/5) studies dealing with dichotomous/nominal data, 46.7% (7/15) of studies dealing with ordinal data, and 95.2% (20/21) of studies dealing with continuous data. Statistical methods were described in sufficient detail regarding weighted kappa in 28.6% (2/7) of studies and regarding the model and assumptions of intraclass correlation coefficient in 35.3% (6/17) and 29.4% (5/17) of studies, respectively. Reliability parameters were used as if they were agreement parameters in 23.1% (3/13) of studies. Reproducibility and repeatability were used incorrectly in 20% (3/15) of studies. CONCLUSION: Greater attention to the importance of reporting reliability, thorough description of the related statistical methods, efforts not to neglect agreement parameters, and better use of relevant terminology is necessary.

Estimativas de repetibilidade para caracteres forrageiros em Panicum maximum / Repeatability estimates for forage characters in Panicum maximum

Objetiva-se com este trabalho estimar a repetibilidade para caracteres forrageiros de Panicum, e determinar o número de cortes de avaliação necessários para a seleção de genótipos de Panicum, com confiabilidade. Utilizaram-se os dados de um ensaio conduzido no período de 21/11/2002 a 08/04/2005, no Campo Experimental da Embrapa Gado de Leite, localizado em Valença-RJ, onde foram realizados 15 cortes de avaliação. No ensaio, foram avaliados 23 genótipos de Panicum maximum, em parcelas experimentais, dispostas no delineamento de blocos casualizados, com três repetições. Foram estimados os coeficientes de repetibilidade para as características produção de matéria verde de forragem (PMV); produção de matéria seca de forragem (PMS) e de folhas (PMSF); porcentagem de folhas na PMS ( por centoFOL) e altura da planta (AP), utilizando os métodos da análise de variância, componentes principais e análise estrutural. Para todas as características avaliadas os efeitos de genótipos, cortes e interação genótipos x cortes foram significativos (P<0,01). Quando se considerou o coeficiente de determinação de 85 por cento, o número de avaliações (cortes) necessários para a determinação do valor real dos genótipos foram de 10, 9, 7, 11 e 3, respectivamente, para PMV, PMS, PMSF, por centoFOL e AP. O método dos componentes principais e o da análiseestrutural (baseado na matriz de correlações) foram concordantes para todas as características avaliadas. A realização de 10 cortes de avaliação permite discriminar o valor real dos genótipos de Panicum, com confiabilidade superior a 85 por cento, para a maioria das características avaliadas.

The objective of this work was to estimate the repeatability for forage characters of Panicum and to determinate the necessary number of evaluation cuts to select Panicum genotypes with confidence. Data of a trial with 15 cuts, carried out between 21/11/2002 and 08/04/2005 in the experimental station of Embrapa Gado de Leite located in Valença, RJ, Brazil, were used. In this study. 23 genotypes of " Panicum maximum" were evaluated, in a complete randomized block, with three replications. The coefficient of repeatability for fresh forage production (PMV), total plant dry matter production (PMS) and leaves dry matter production (PMSF) were recorded along with leaves percentage in PMS ( percentFOL) and plant hight (AP), using the variance analysis, main components and structural analysis methods. For all evaluated parameters the effects of genotype, cut and genotype x cut interaction were significant (P<0.01). When considering the determination coefficient as 85 percent, the required number of measures (cuts) to determine the real value of genotypes were 10, 9, 7, 11 and 3, for PMV, PMS, PMSF, percentFOL and AP, respectively. Main components and structural analysis methods (based on the correlation matrix) did agree for all features evaluated. The utilization of data from 10 cuts allows to discriminate, with confidence higher than 85 percent, the real value for most of the evaluated characters of Panicum genotypes.