Better to be in agreement than in bad company : A critical analysis of many kappa-like tests.

We assessed several agreement coefficients applied in 2x2 contingency tables, which are commonly applied in research due to dichotomization. Here, we not only studied some specific estimators but also developed a general method for the study of any estimator candidate to be an agreement measurement. This method was developed in open-source R codes and it is available to the researchers. We tested this method by verifying the performance of several traditional estimators over all possible configurations with sizes ranging from 1 to 68 (total of 1,028,789 tables). Cohen's kappa showed handicapped behavior similar to Pearson's r, Yule's Q, and Yule's Y. Scott's pi, and Shankar and Bangdiwala's B seem to better assess situations of disagreement than agreement between raters. Krippendorff's alpha emulates, without any advantage, Scott's pi in cases with nominal variables and two raters. Dice's F1 and McNemar's chi-squared incompletely assess the information of the contingency table, showing the poorest performance among all. We concluded that Cohen's kappa is a measurement of association and McNemar's chi-squared assess neither association nor agreement; the only two authentic agreement estimators are Holley and Guilford's G and Gwet's AC1. The latter two estimators also showed the best performance over the range of table sizes and should be considered as the first choices for agreement measurement in contingency 2x2 tables. All procedures and data were implemented in R and are available to download from Harvard Dataverse https://doi.org/10.7910/DVN/HMYTCK.

Intra- and Inter-Rater Reliability and Agreement of Ultrasound Imaging of Muscle Architecture and Patellar Tendon in Post-COVID-19 Patients Who Had Experienced Moderate or Severe COVID-19 Infection.

COVID-19 is associated with musculoskeletal disorders. Ultrasound is a tool to assess muscle architecture and tendon measurements, offering an idea of the proportion of the consequences of the disease, since significant changes directly reflect the reduction in the ability to produce force and, consequently, in the functionality of the patient; however, its application in post-COVID-19 infection needs to be determined. We aimed to assess the intra- and inter-rater reliability of ultrasound measures of the architecture of the vastus lateralis (VL), rectus femoris (RF), vastus medialis (VM), gastrocnemius lateralis (GL), gastrocnemius medialis (GM), soleus (SO), and tibialis anterior (TA) muscles, as well as the patellar tendon (PT) cross-sectional area (CSA) in post-COVID-19 patients. An observational, prospective study with repeated measures was designed to evaluate 20 post-COVID-19 patients, who were measured for the pennation angle (θp), fascicular length (Lf), thickness, echogenicity of muscles, CSA and echogenicity of the PT. The intra-class correlation coefficient (ICC) and 95% limits of agreement were used. The intra-rater reliability presented high or very high correlations (ICC = 0.71-1.0) for most measures, except the θp of the TA, which was classified as moderate (ICC = 0.69). Observing the inter-rater reliability, all the evaluations of the PT, thickness and echogenicity of the muscles presented high or very high correlations. For the Lf, only the RF showed as low (ICC = 0.43), for the θp, RF (ICC = 0.68), GL (ICC = 0.70) and TA (ICC = 0.71) moderate and the SO (ICC = 0.40) low. The ultrasound reliability was acceptable for the muscle architecture, muscle and tendon echogenicity, and PT CSA, despite the low reliability for the Lf and θp of the RF and SO, respectively.

Cultural adaptation and reliability of the ISTAP Skin Tear Classification System to Chilean Spanish.

OBJECTIVE: The International Skin Tear Advisory Panel (ISTAP) created the ISTAP Skin Tear Classification System with the aim of promoting a common and universal language to describe, classify and document skin tears, and increase awareness of the high prevalence of these wounds. Although there is a Spanish version of the ISTAP Skin Tear Classification System, the authors considered it relevant to have a specific Chilean Spanish version. The aim was to achieve the cultural adaptation of the ISTAP Skin Tear Classification System to Chilean Spanish, and verify its content validity and inter-rater reliability. METHOD: The study consisted of two phases: cultural adaptation of the ISTAP Skin Tear Classification System into Chilean Spanish, and validation of the culturally adapted system. A convenience sample of 150 health professionals classified 30 photographs of skin tears, using the same photographs used in the original ISTAP skin tear classification and validation study. Additionally, the clinical application of the classification system was tested among 20 nurses, who assessed and classified the skin tears (n=24) of hospitalised patients. For analysis of the inter-rater reliability, Fleiss' kappa was used. RESULTS: The differences found in the translation referred to a synonym of the terms 'skin' or 'cutaneous', and the terms 'flap' or 'tear'. Once analysed and discussed, the term 'desgarro' was maintained, which is the translation of the English term 'flap'. There is no equivalent term for 'skin tears' in Spanish, but consensus was reached by researchers and collaborators to use the phrase: 'desgarro de piel'. Once a consensus was reached on the wording for the translation, back-translation was completed and compared with the original English version and reviewed by the original author of the classification for accuracy. The content validity of the translated version of the ISTAP Skin Tear Classification System into Chilean Spanish showed a moderate agreement for the non-specialised nurses' group (0.4804) and for the specialised nurses' group (0.5308). Inter-rater reliability was achieved by obtaining a moderate agreement (Fleiss' kappa=0.53) and an almost perfect level of agreement for clinical application (Fleiss' kappa=0.83). CONCLUSION: The reported content validity and inter-rater reliability support the applicability of the cultural adaptation of the ISTAP Skin Tear Classification System to Chilean Spanish into practice.

Classifying maternal deaths in Suriname using WHO ICD-MM: different interpretation by Physicians, National and International Maternal Death Review Committees.

The World Health Organization (WHO) provides a framework (ICD-MM) to classify pregnancy-related deaths systematically, which enables global comparison among countries. We compared the classification of pregnancy-related deaths in Suriname by the attending physician and by the national maternal death review (MDR) committee and among the MDR committees of Suriname, Jamaica and the Netherlands. There were 89 possible pregnancy-related deaths in Suriname between 2010 and 2014. Nearly half (47%) were classified differently by the Surinamese MDR committee as compared to the classification of the attending physicians. All three MDR committees agreed that 18% (n = 16/89) of the cases were no maternal deaths. Out of the remaining 73 cases, there was disagreement regarding whether 15% (n = 11) were maternal deaths. The Surinamese and Jamaican MDR committees achieved greater consensus in classification than the Surinamese and the Netherlands MDR committees. The Netherlands MDR committee classified more deaths as unspecified than Surinamese and the Jamaican MDR committees. Underlying causes that achieved a high level of agreement among the three committees were abortive outcomes and obstetric hemorrhage, while little agreement was reported for unspecified and other direct causes. The issues encountered during maternal death classification using the ICD-MM guidelines included classification of suicide during early pregnancy; when to assume pregnancy without objective evidence; how to count maternal deaths occurring outside the country of residence; the relevance of direct or indirect cause attribution; and how to select the underlying cause when direct and indirect conditions or multiple comorbidities co-occur. Addressing these classification barriers in future revisions of the ICD-MM guidelines could enhance the feasibility of maternal death classification and facilitate global comparison. BACKGROUND: Insight into the underlying causes of pregnancy-related deaths is essential to develop policies to avert preventable deaths. The WHO International Classification of Diseases-Maternal Mortality (ICD-MM) guidelines provide a framework to standardize maternal death classifications and enable comparison in and among countries over time. However, despite the implementation of these guidelines, differences in classification remain. We evaluated consensus on maternal death classification using the ICD-MM guidelines. METHODS: The classification of pregnancy-related deaths in Suriname during 2010-2014 was compared in the country (between the attending physician and the national maternal death review (MDR) committee), and among the MDR committees from Suriname, Jamaica and the Netherlands. All reviewers applied the ICD-MM guidelines. The inter-rater reliability (Fleiss kappa [κ]) was used to measure agreement. RESULTS: Out of the 89 cases certified by attending physicians, 47% (n = 42) were classified differently by the Surinamese MDR committee. The three MDR committees agreed that 18% (n = 16/89) of these cases were no maternal deaths, and, therefore, excluded from further analyses. However, opinions differed whether 15% (n = 11) of the remaining 73 cases were maternal deaths. The MDR committees achieved moderate agreement classifying the deaths into type (direct, indirect and unspecified) (κ = 0.53) and underlying cause group (κ = 0.52). The Netherlands MDR committee classified more maternal deaths as unspecified (19%), than the Jamaican (7%) and Surinamese (4%) committees did. The mutual agreement between the Surinamese and Jamaican MDR committees (κ = 0.69 vs κ = 0.63) was better than between the Surinamese and the Netherlands MDR committees (κ = 0.48 vs κ = 0.49) for classification into type and underlying cause group, respectively. Agreement on the underlying cause category was excellent for abortive outcomes (κ = 0.85) and obstetric hemorrhage (κ = 0.74) and fair for unspecified (κ = 0.29) and other direct causes (κ = 0.32). CONCLUSIONS: Maternal death classification differs in Suriname and among MDR committees from different countries, despite using the ICD-MM guidelines on similar cases. Specific challenges in applying these guidelines included attribution of underlying cause when comorbidities occurred, the inclusion of deaths from suicides, and maternal deaths that occurred outside the country of residence.

Geração de itens e validação de conteúdo de domínios e de itens para o indivíduo com osteoartrite do joelho: um estudo de método misto / Item generation and content validation of domains with item pool for the individual with knee osteoarthritis: a mixed-method study

INTRODUÇÃO: Existem várias escalas para avaliar as percepções subjetivas e os componentes individuais em indivíduos com osteoartrite (OA) de joelho. Até o momento, não há escalas disponíveis conhecidas para medir o equilíbrio combinado entre mobilidade, AVD e QV em OA de joelho com base na Classificação Internacional de Funcionalidade, Incapacidade e Saúde (CIF). OBJETIVO: Gerar itens e domínios relacionados aos problemas enfrentados pelos indivíduos com OA de joelhos e validar o conteúdo por especialistas. MÉTODOS: Os domínios e itens foram gerados através de pesquisa bibliográfica extensa (ELS) para extrair itens relacionados a equilíbrio, mobilidade, ADL e QV em indivíduos com OA em joelhos baseados na CIF e através de entrevista aprofundada direta (EAD) em 13 pessoas com OA de joelhos e três especialistas. A validação de conteúdo dos domínios e itens gerados foi validada por 10 especialistas por meio da pesquisa Delphi online. O índice mínimo de validação de conteúdo em nível de item (I-CVI) de 0,80 foi considerado para validar os itens identificados e o índice de validação de conteúdo em nível de escala geral (S-CVI) de 0,90 foi fixado para validar os itens gerados para uso no processo de desenvolvimento da escala. RESULTADOS: Os 117 itens gerados por EAD na ELS foram inicialmente agrupados em 18 domínios. A validação de conteúdo pelo método Delphi resultou em uma diminuição para 56 itens agrupados em 14 domínios com SCVI de 0, 93. CONCLUSÃO: O conjunto abrangente de itens de deficiência, limitação de atividade e restrição de participação para indivíduos com OA de joelhos nos domínios propostos foi desenvolvido e o conteúdo validado. Esses itens são recomendados para uso no desenvolvimento de uma nova escala abrangente de índice de osteoartrite do joelho (CKOAI).

INTRODUCTION: There are several scales to evaluate subjective perceptions and individual components in individuals with knee osteoarthritis (IKOA). Till date, no scale is available to measure the combined balance, mobility, ADL and QoL in IKOA based on the International Classification of Functioning, Disability and Health (ICF). OBJECTIVE: The purpose of the study was to generate items and domains related to problems faced by IKOA and to validate the content by experts. METHODS: The domains and items were generated through extensive literature search (ELS) to extract items related to symptoms, balance, mobility, ADL and QoL in IKOA based on the International Classification of Functioning, Disability and Health (ICF) and through in-depth direct interview (IDDI) from 13 IKOA and three experts. The content validation of domains and items generated were validated by 10 experts through online Delphi survey. Minimum itemlevel content validation index (I-CVI) of 0.80 was considered to validate the identified items and the overall scale-level content validation index (S-CVI) of 0.90 was fixed to validate the generated items to use in scale development process. RESULTS: 117 items generated by IDDI and ELS were grouped under 18 domains initially. Content validation by Delphi method resulted in reduction with 56 item pool being grouped under the 14 domains with SCVI is 0.93. CONCLUSION: The comprehensive impairment, activity limitation and participation restriction item pool for IKOA under the proposed domains, have been developed and content validated. These items are recommended for their use in development of new comprehensive knee osteoarthritis index scale (CKOAI).

Concordancia entre observadores de videos de pacientes con sospecha diagnóstica de autismo y psicosis infantiles / Inter-rater Reliability in Videos of Patients with a Suspected Diagnostic of Autism and Child Psychosis

RESUMEN Introducción: Dados la complejidad del abordaje y el tratamiento del autismo y las psicosis infantiles y el impacto que tienen en la vida del niño y su familia, es fundamental ser cautos a la hora de hacer un diagnóstico que tiene implicaciones pronosticas. Se utiliza la perspectiva psicodinámica para acceder a la posibilidad de incluir en el diagnóstico aspectos tanto patológicos como sanos y promover el potencial evolutivo de cada paciente. Objetivo: Determinar la concordancia entre evaluadores y evaluar cuantitativa y cualitativa mente las variables que influyen en el diagnóstico de pacientes con sospecha de autismo y psicosis infantiles desde la perspectiva psicodinámica. Métodos: Se realizó un estudio de concordancia entre observadores basado en la evaluación diagnóstica de videos de pacientes con sospecha de autismo o psicosis infantiles que asisten a junta diagnóstica (sesión de observación) en el Instituto de Ortopedia Infantil Roosevelt. Resultados: Se obtuvieron valores de kappa que oscilaron en general entre 0,24 y 0,50, con una fuerza de concordancia que varió de leve a moderada y K = 0,24 para la organización de personalidad. Esta manera de diagnosticar toma en consideración los aspectos tanto patológicos como saludables que constituyen la organización de personalidad. Al final se discuten las limitaciones y los puntos que tener en cuenta en los estudios posteriores. Conclusiones: Se recomienda que evalúe al niño con perturbaciones mayores un equipo interdisciplinario, y no un solo observador, pues este espacio favorece la discusión, que es crucial a la hora de establecer un diagnóstico y evita lecturas parciales del funcionamiento psíquico del paciente que consulta.

ABSTRACT Introduction: Diagnosing and treating autism and child psychoses is very difficult; these pathologies impact not only the child's life but also the family as a whole. Therefore caution is required when giving a diagnosis with prognostic implications. We use the psychodynamic perspective in order to take into consideration both pathological and healthy diagnostic aspects, and to promote evolutionary potential of each patient. Objective: To determine the inter-rater reliability and to test quantitatively and qualitati vely the variables involved in the diagnoses of patients with suspected autism and child psychoses, based upon psychodynamic concepts. Methods: An inter-rater reliability study was carried on, based upon the diagnostic evaluation of videos of patients with suspected autism or child psychoses who attended the diagnostic meeting (observation session) at the Instituto de Ortopedia Infantil Roosevelt. Results: Kappa values were obtained, ranging from .24 to .50, with a reliability force varying from slight to moderate, and K=.24 for personality organization. This type of diagnosis takes into account both the pathological and healthy aspects which make up personality orga nization. Finally, limitations and aspects that should be considered in further studies were discussed. Conclusions: The results reinforce the need to evaluate a child with major disorders in an interdisciplinary team, and but a single observer, in order to allow discussion, and debate, and therefore, avoid partial readings of the patient's psychological functioning.

Confiabilidad interevaluador de la prueba video head impulse test en estudiantes de primer año de la generación 2016 de la Escuela de Fonoaudiología de la Universidad de Valparaíso Casa Central / Inter-rater reliability of the video head impulse test in students from the School of Speech and Language therapist of the University of Valparaiso main campus class 2016

RESUMEN Introducción: El video head impulse test es un método rápido, no invasivo y cómodo para evaluar el reflejo vestíbulo ocular, el cual está siendo utilizado con mayor frecuencia en la evaluación de pacientes con alteraciones del equilibrio. Sin embargo, en Chile se evidencia una falta de estudios relacionados a la confiabilidad interevaluador de dicha prueba. Objetivo: Determinar el nivel de concordancia de los resultados de la eficiencia del reflejo vestíbulo ocular de los canales semicirculares horizontales, obtenidos mediante la aplicación de la prueba video head impulse test, por parte de 3 evaluadores. Material y método: Estudio cuantitativo, correlacional, no experimental transversal descriptivo, con muestreo no probabilístico, de participación voluntaria. La prueba video head impulse test fue aplicada por 3 examinadores a 30 estudiantes que cumplieron con los criterios de inclusión. Resultados: El método de Bland y Altman mostró gran concordancia en las mediciones realizadas entre los 3 examinadores. La correlación intraclase para la ganancia del canal semicircular horizontal derecho e izquierdo fue de 0,85 y 0,91 respectivamente y de 0,75 para el valor de asimetría. Conclusión: El estudio mostró una confiabilidad interevaluador adecuada de la prueba video head impulse test para la evaluación de los canales semicirculares horizontales.

ABSTRACT Introduction: The video head impulse test is a rapid, non-invasive and comfortable method for the evaluation of the vestibulo-ocular reflex that is being used more often in the evaluation of patients with balance disorders. In Chile there is a lack of studies related to the inter-rater reliability for this test. Aim: To determine the level of agreement on the results for the vestibulo-ocular reflex of the horizontal semicircular canals obtained by the execution of the video head impulse test by 3 raters. Material and method: Quantitative, correlational, non-experimental, cross-sectional descriptive study with a type of non-probability sampling with voluntary participation. The video head impulse test was applied by 3 raters to 30 students who fulfilled the inclusion criteria. Results: The Bland-Altman method demonstrated good agreement in the measurements made between the 3 raters. The inter-rater reliability for the gain of the left and right horizontal semicircular canals was 0.85 and 0.91 respectively and 0.75 for the asymmetry value. Conclusion: The study showed an adequate inter-rater reliability for the video head impulse test in the evaluation of the horizontal semicircular canals.

Lung Ultrasonography: A Viable Alternative to Chest Radiography in Children with Suspected Pneumonia?

OBJECTIVE: To determine the interrater reliability (IRR) of lung ultrasonography (LUS) and chest radiography (CXR) and evaluate the accuracy of LUS compared with CXR for detecting pediatric pneumonia compared with chest computed tomography (CT) scan. STUDY DESIGN: This was a prospective cohort study of children aged 3 months to 18 years with a CXR and LUS performed between May 1, 2012, and January 31, 2014 with or without a clinical diagnosis of pneumonia. Four pediatric radiologists blinded to clinical information reported findings for the CXR and LUS images. IRR was estimated for 50 LUS and CXR images. The main outcome was the finding from CT ordered clinically or the probability of the CT finding for patients clinically requiring CT. Two radiologists reviewed CT scans to determine an overall finding. Latent class analysis was used to evaluate the sensitivity and specificity for findings (eg, consolidation) for LUS and CXR compared with CT. RESULTS: Of the 132 patients in the cohort, 36 (27%) had CT performed for a clinical reason. Pneumonia was clinically documented in 47 patients (36%). The IRR for lung consolidation was 0.55 (95% CI, 0.40-0.70) for LUS and 0.36 (95% CI, 0.21-0.51) for CXR. The sensitivity for detecting consolidation, interstitial disease, and pleural effusion was statistically similar for LUS and CXR compared with CT; however, specificity was higher for CXR. The negative predictive value was similar for CXR and LUS. CONCLUSIONS: LUS has a sufficiently high IRR for detection of consolidation. Compared with CT, LUS and CXR have similar sensitivity, but CXR is more specific for findings indicating pneumonia.

Specific algorithm method of scoring the Clock Drawing Test applied in cognitively normal elderly / Método com algoritmo específico para pontuação do teste do desenho do relógio em idosos cognitivamente normais

ABSTRACT. The Clock Drawing Test (CDT) is an inexpensive, fast and easily administered measure of cognitive function, especially in the elderly. This instrument is a popular clinical tool widely used in screening for cognitive disorders and dementia. The CDT can be applied in different ways and scoring procedures also vary. OBJECTIVE: The aims of this study were to analyze the performance of elderly on the CDT and evaluate inter-rater reliability of the CDT scored by using a specific algorithm method adapted from Sunderland et al. (1989). METHODS: We analyzed the CDT of 100 cognitively normal elderly aged 60 years or older. The CDT ("free-drawn") and Mini-Mental State Examination (MMSE) were administered to all participants. Six independent examiners scored the CDT of 30 participants to evaluate inter-rater reliability. RESULTS AND CONCLUSION: A score of 5 on the proposed algorithm ("Numbers in reverse order or concentrated"), equivalent to 5 points on the original Sunderland scale, was the most frequent (53.5%). The CDT specific algorithm method used had high inter-rater reliability (p<0.01), and mean score ranged from 5.06 to 5.96. The high frequency of an overall score of 5 points may suggest the need to create more nuanced evaluation criteria, which are sensitive to differences in levels of impairment in visuoconstructive and executive abilities during aging.

RESUMO. O Teste do Desenho do Relógio (TDR) é uma barata e rápida medida de função cognitiva, de fácil aplicação, especialmente em idosos. Este instrumento é uma ferramenta clínica muito conhecida, amplamente utilizada no rastreamento de transtornos cognitivos e demência. O TDR pode ser aplicado de diferentes formas e a sua pontuação também varia. OBJETIVO: Os objetivos deste estudo foram analisar o desempenho dos idosos no TDR e avaliar a confiabilidade inter-examinadores do TDR pontuado por um método com algoritmo específico, adaptado a partir dos critérios estabelecidos por Sunderland et al. (1989). MÉTODOS: Analisamos o TDR de 100 idosos cognitivamente saudáveis com 60 anos de idade ou mais. O TDR ("desenho livre") e o Mini-Exame do Estado Mental (MEEM) foram administrados em todos os participantes. Seis avaliadores independentes pontuaram 30 TDR para avaliar a confiabilidade inter-examinadores. RESULTADOS E CONCLUSÃO: A pontuação 5 do algoritmo proposto ("Os números em ordem inversa ou concentrados") equivalente a 5 pontos na escala original de Sunderland foi a mais frequente (53,5%). O método com algoritmo específico do TDR utilizado teve alta confiabilidade entre avaliadores (p<0,01), e a média da pontuação variou entre 5,06 e 5,96. A alta frequência de 5 pontos na pontuação geral pode sugerir a necessidade da elaboração de critérios de avaliação mais sutis, que sejam sensíveis às diferenças entre indícios de comprometimento nas habilidades visuoconstrutivas e executivas durante o envelhecimento.

Specific algorithm method of scoring the Clock Drawing Test applied in cognitively normal elderly.

The Clock Drawing Test (CDT) is an inexpensive, fast and easily administered measure of cognitive function, especially in the elderly. This instrument is a popular clinical tool widely used in screening for cognitive disorders and dementia. The CDT can be applied in different ways and scoring procedures also vary. OBJECTIVE: The aims of this study were to analyze the performance of elderly on the CDT and evaluate inter-rater reliability of the CDT scored by using a specific algorithm method adapted from Sunderland et al. (1989). METHODS: We analyzed the CDT of 100 cognitively normal elderly aged 60 years or older. The CDT ("free-drawn") and Mini-Mental State Examination (MMSE) were administered to all participants. Six independent examiners scored the CDT of 30 participants to evaluate inter-rater reliability. RESULTS AND CONCLUSION: A score of 5 on the proposed algorithm ("Numbers in reverse order or concentrated"), equivalent to 5 points on the original Sunderland scale, was the most frequent (53.5%). The CDT specific algorithm method used had high inter-rater reliability (p<0.01), and mean score ranged from 5.06 to 5.96. The high frequency of an overall score of 5 points may suggest the need to create more nuanced evaluation criteria, which are sensitive to differences in levels of impairment in visuoconstructive and executive abilities during aging.

O Teste do Desenho do Relógio (TDR) é uma barata e rápida medida de função cognitiva, de fácil aplicação, especialmente em idosos. Este instrumento é uma ferramenta clínica muito conhecida, amplamente utilizada no rastreamento de transtornos cognitivos e demência. O TDR pode ser aplicado de diferentes formas e a sua pontuação também varia. OBJETIVO: Os objetivos deste estudo foram analisar o desempenho dos idosos no TDR e avaliar a confiabilidade inter-examinadores do TDR pontuado por um método com algoritmo específico, adaptado a partir dos critérios estabelecidos por Sunderland et al. (1989). MÉTODOS: Analisamos o TDR de 100 idosos cognitivamente saudáveis com 60 anos de idade ou mais. O TDR ("desenho livre") e o Mini-Exame do Estado Mental (MEEM) foram administrados em todos os participantes. Seis avaliadores independentes pontuaram 30 TDR para avaliar a confiabilidade inter-examinadores. RESULTADOS E CONCLUSÃO: A pontuação 5 do algoritmo proposto ("Os números em ordem inversa ou concentrados") equivalente a 5 pontos na escala original de Sunderland foi a mais frequente (53,5%). O método com algoritmo específico do TDR utilizado teve alta confiabilidade entre avaliadores (p<0,01), e a média da pontuação variou entre 5,06 e 5,96. A alta frequência de 5 pontos na pontuação geral pode sugerir a necessidade da elaboração de critérios de avaliação mais sutis, que sejam sensíveis às diferenças entre indícios de comprometimento nas habilidades visuoconstrutivas e executivas durante o envelhecimento.