Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model.

Reliability of measurement instruments providing quantitative outcomes is usually assessed by an intraclass correlation coefficient. When participants are repeatedly measured by a single rater or device, or, are each rated by a different group of raters, the intraclass correlation coefficient is based on a one-way analysis of variance model. When planning a reliability study, it is essential to determine the number of participants and measurements per participant (i.e. number of raters or number of repeated measurements). Three different sample size determination approaches under the one-way analysis of variance model were identified in the literature, all based on a confidence interval for the intraclass correlation coefficient. Although eight different confidence interval methods can be identified, Wald confidence interval with Fisher's large sample variance approximation remains most commonly used despite its well-known poor statistical properties. Therefore, a first objective of this work is comparing the statistical properties of all identified confidence interval methods-including those overlooked in previous studies. A second objective is developing a general procedure to determine the sample size using all approaches since a closed-form formula is not always available. This procedure is implemented in an R Shiny app. Finally, we provide advice for choosing an appropriate sample size determination method when planning a reliability study.

Statistical inference for agreement between multiple raters on a binary scale.

Agreement studies often involve more than two raters or repeated measurements. In the presence of two raters, the proportion of agreement and of positive agreement are simple and popular agreement measures for binary scales. These measures were generalized to agreement studies involving more than two raters with statistical inference procedures proposed on an empirical basis. We present two alternatives. The first is a Wald confidence interval using standard errors obtained by the delta method. The second involves Bayesian statistical inference not requiring any specific Bayesian software. These new procedures show better statistical behaviour than the confidence intervals initially proposed. In addition, we provide analytical formulas to determine the minimum number of persons needed for a given number of raters when planning an agreement study. All methods are implemented in the R package simpleagree and the Shiny app simpleagree.

Chasing consistency: On the measurement error in self-reported affect in experiments.

How feelings change over time is a central topic in emotion research. To study these affective fluctuations, researchers often ask participants to repeatedly indicate how they feel on a self-report rating scale. Despite widespread recognition that this kind of data is subject to measurement error, the extent of this error remains an open question. Complementing many daily-life studies, this study aimed to investigate this question in an experimental setting. In such a setting, multiple trials follow each other at a fast pace, forcing experimenters to use a limited number of questions to measure affect during each trial. A total of 1398 participants completed a probabilistic reward task in which they were unknowingly presented with the same string of outcomes multiple times throughout the study. This allowed us to assess the test-retest consistency of their affective responses to the rating scales under investigation. We then compared these consistencies across different types of rating scales in hopes of finding out whether a given type of scale led to a greater consistency of affective measurements. Overall, we found moderate to good consistency of the affective measurements. Surprisingly, however, we found no differences in consistency across rating scales, which suggests that the specific rating scale that is used does not influence the measurement consistency.

Reliability and agreement in intrapartum fetal heart rate monitoring interpretation: A systematic review.

INTRODUCTION: Fetal heart rate (FHR) monitoring is routine in intrapartum care worldwide and one of the most common obstetrical procedures. Intrapartum FHR monitoring helps assess fetal wellbeing and interpretation of the FHR help form decisions for clinical management and intervention. It relies on the observers' subjective assessments, with variation in interpretations leading to variations in intrapartum care. The purpose of this systematic review was to summarize and evaluate extant inter- and intrarater reliability research on the human interpretation of intrapartum FHR monitoring. MATERIAL AND METHODS: We searched for the terms "fetal heart rate monitoring," "interpretation agreement" and related concepts on Embase, Medline, Maternity and Infant Care Database and CINAHL. The last search was made on January 31, 2022. The protocol for the study was prospectively registered in PROSPERO (CRD42021260937). Studies that assess inter- and intrarater reliability and agreement of health professionals' intrapartum FHR monitoring were included and studies including other assessment of fetal wellbeing excluded. We extracted data in reviewer pairs using quality appraisal tool for studies of diagnostic reliability (QAREL) forms. The data retrieved from the studies are presented as narrative synthesis and in additional tables. RESULTS: Forty-nine articles concerning continuous FHR monitoring were included in the study. For interrater reliability and agreement, in total 577 raters assessed 6315 CTG tracings. There was considerable heterogeneity in quality and measures across the included articles. We found higher reliability and agreement for the basic FHR features than for overall classification and higher agreement for intrarater reliability and agreement than for their interrater counterparts. CONCLUSIONS: There is great variation in reliability and agreement measures for continuous intrapartum FHR monitoring, implying that intrapartum CTG should be used with caution for clinical decision making given its questionable reliability. We found few high-quality studies and noted methodological concerns in the studies. We recommend a more standardized approach to future reliability studies on FHR monitoring.

Affective Symptoms and Oropharyngeal Dysphagia in Head-and-Neck Cancer Patients: A Systematic Review.

Oropharyngeal dysphagia (OD) is a high impact morbidity in head-and-neck cancer (HNC) patients. A wide variety of instruments are developed to screen for affective symptoms and OD. The current paper aims to systematically review and appraise the literature to obtain insight into the prevalence, strength, and causal direction of the relationship between affective symptoms and OD in HNC patients. This review was conducted in accordance with the PRISMA statement. A systematic search of the literature was performed using PubMed, PsycINFO, Cochrane, and Embase. All available publications reporting on the relationship between affective conditions and swallowing function in HNC patients were included. Conference papers, tutorials, reviews, and studies with less than 5 patients were excluded. Fifteen studies met the inclusion criteria. The level of evidence and methodological quality were assessed using the ABC-rating scale and QualSyst critical appraisal tool. Eleven studies reported a positive relationship between affective symptoms and OD. The findings of this paper highlight the importance of affective symptom screening in dysphagic HNC patients as clinically relevant affective symptoms and OD seems to be prevalent and coincident in this population. Considering the impact of affective symptoms and OD on patients' daily life, early detection and an integrated interdisciplinary approach are recommended. However, due to the heterogeneity of study designs, outcomes, and outcome measures, the generalization of study results is limited.

Impact of the biopsy forceps size on histological analysis and performances of the histological scoring systems.

To improve the reliability of the quantitative scorings of the synovial biopsies, we evaluate whether diameter of arthroscopic forceps influences histological quality of synovial tissue and/or histological scores and we compare the intra- and inter-observer performances of the main histological scoring systems. Synovial biopsies were retrieved in the same part of the joint using 1, 2 and 4 mm diameters grasping forceps. After standard staining and immunohistochemistry with anti-CD68 antibody, slides were scored blindly by 2 independent experienced operators for tissue quality and with Krenn score, de Bois-Tak score and CD68 semi-quantitative score. Four samples did not pass quality control. No difference other than a higher number of vessels in the 4 mm versus 2 mm forceps (p = 0.01) was found among the 3 groups. CD68 score was significantly higher in the 2 versus 4 mm forceps (p = 0.009). So we concluded that only vessels quantification and CD68 semi-quantitative score seemed affected by the forceps size. The intra-reader agreement was variable across observers and features: 0.78 (0.66-0.87) for the Krenn scoring system, 0.89 (0.78-0.97) for the de Bois-Tak score and 0.93 (0.81-1.00) for the CD68 score. Interobserver reliabilities of Krenn score, de Bois-Tak score and CD68 scores were satisfactory: 0.95 (0.92-0.99) for Krenn, 0.98 (0.96-0.99) for de Bois-Tak and 0.80 (0.71-0.89) for CD68.

Punch Trackers: Correct Recognition Depends on Punch Type and Training Experience.

To determine the ability of different punch trackers (PT) (Corner (CPT), Everlast (EPT), and Hykso (HPT)) to recognize specific punch types (lead and rear straight punches, lead and rear hooks, and lead and rear uppercuts) thrown by trained (TR, n = 10) and untrained punchers (UNTR, n = 11), subjects performed different punch combinations, and PT data were compared to data from video recordings to determine how well each PT recognized the punches that were actually thrown. Descriptive statistics and multilevel modelling were used to analyze the data. The CPT, EPT and HPT detected punches more accurately in TR than UNTR, evidenced by a lower percentage error in TR (p = 0.007). The CPT, EPT, and HPT detected straight punches better than uppercuts and hooks, with a lower percentage error for straight punches (p < 0.001). The recognition of punches with CPT and HPT depended on punch order, with earlier punches in a sequence recognized better. The same may or may not have occurred with EPT, but EPT does not allow for data to be exported, meaning the order of individual punches could not be analyzed. The CPT and HPT both seem to be viable options for tracking punch count and punch type in TR and UNTR.

Voice and Vocal Fold Condition Following Short-Term General Anesthesia: A Prospective Study.

BACKGROUND: Dysphonia, with or without laryngeal changes, has been reported as a complication following prolonged intubation. In contrast, it is unknown if laryngeal changes also occur following short-term airway instrumentation. The objectives of this study were to determine the prevalence of laryngeal changes in patients undergoing short-term routine general anesthesia using an endotracheal tube (ETT) or supraglottic airway (SGA), and to identify predictors to these changes. METHODS: Standardized voice assessments were performed preoperatively, postoperatively, and at follow-up on adults undergoing general anesthesia for an elective procedure of less than three hours requiring an ETT or a SGA. The standardized voice assessment protocol comprised a rigid videolaryngostroboscopy, the Voice Handicap Index (VHI), and acoustic voice analysis. The effects of demographic and anesthetic characteristics and type of airway instrumentation on the videolaryngostroboscopic variables were studied using multilevel logistic regression. Multilevel linear regression was used to reveal preoperative versus postoperative changes in VHI and acoustic voice scores. RESULTS: Overall, the prevalence of postoperative laryngeal changes was low. Significant postoperative laryngeal changes were found for the variables right-sided vocal fold redness in the ETT group (P = 0.048) and right-sided vocal fold blood vessels in both groups (ETT versus SGA). However, after adjustment for all demographic and anesthetic characteristics in the regression model, the effect of the type of airway instrumentation (ETT versus SGA) on the variable right-sided vocal fold redness was no longer significant. CONCLUSIONS: ETT and SGA short-term airway instrumentation are vocal fold function sparing techniques with negligible laryngeal changes.

Affective symptoms and swallow-specific quality of life in total laryngectomy patients.

BACKGROUND: The aim of this study is to determine the prevalence of clinically relevant affective symptoms and level of swallow-specific quality of life (QoL) in dysphagic patients with total laryngectomy (TL) and to explore the relationship between affective symptoms and swallow-specific QoL. METHODS: Thirty-five TL patients completed the Hospital Anxiety and Depression Scale (HADS) and the MD Anderson Dysphagia Inventory (MDADI). Student's t test and linear regression were used. RESULTS: Eight (23%) patients showed clinically relevant symptoms of anxiety, 8 (23%) of depression, and 11 (31%) showed either one. These groups had significantly lower mean MDADI scores. One-point increase in HADS-anxiety or HADS-depression subscale score corresponds with a decrease of 2.7 or 3.0 points, on average, respectively, of the MDADI total score. CONCLUSIONS: Clinically relevant affective symptoms were present in approximately one-third of the TL patients. These preliminary results show that increased affective symptom scores correlate with a decreased swallow-specific QoL.

The use of spectrograms improves the classification of wheezes and crackles in an educational setting.

Chest auscultation is a widely used method in the diagnosis of lung diseases. However, the interpretation of lung sounds is a subjective task and disagreements arise. New technological developments like the use of visSual representation of sounds through spectrograms could improve the agreement when classifying lung sounds, but this is not yet known. In this study, we tested if the use of spectrograms improves the agreement when classifying wheezes and crackles. To do this, we asked twenty-three medical students at UiT the Arctic University of Norway to classify 30 lung sounds recordings for the presence of wheezes and crackles. The sample contained 15 normal recordings and 15 with wheezes or crackles. The students classified the recordings in a random order twice. First sound only, then sound with spectrograms. We calculated kappa values for the agreement between each student and the expert classification with and without display of spectrograms and tested for significant improvement between these two coefficients. We also calculated Fleiss kappa for the 23 observers with and without the spectrogram. In an individual analysis comparing each student to an expert annotated reference standard we found that 13 out of 23 students had a positive change in kappa when classifying wheezes with the help of spectrograms. When classifying crackles 16 out of 23 showed improvement when spectrograms were used. In a group analysis we observed that Fleiss kappa values were k = 0.51 and k = 0.56 (p = 0.63) for classifying wheezes without and with spectrograms. For crackles, these values were k = 0.22 and k = 0.40 (p = <0.01) in the same order. Thus, we conclude that the use of spectrograms had a positive impact on the inter-rater agreement and the agreement with experts. We observed a higher improvement in the classification of crackles compared to wheezes.

Evaluating Electrocardiography-Based Identification of Cardiac Resynchronization Therapy Responders Beyond Current Left Bundle Branch Block Definitions.

OBJECTIVES: This study aimed to evaluate the association of 4 left bundle branch block (LBBB) definitions and their individual ECG characteristics with clinical outcome. Furthermore, it aimed to combine relevant outcome-associated electrocardiographic (ECG) characteristics into a novel outcome-based definition. BACKGROUND: LBBB morphology is associated with positive response to cardiac resynchronization therapy. However, there are multiple LBBB definitions. Associations with outcomes may differ between definitions and depend on varying contributions of the individual ECG characteristics that these LBBB definitions are composed of. METHODS: A retrospective multicenter study was conducted in 1,492 cardiac resynchronization therapy patients. Patients were classified as LBBB or non-LBBB according to definitions provided by the European Society of Cardiology, American Heart Association, MADIT-CRT (Multicenter Automatic Defibrillator Implantation with Cardiac Resynchronization Therapy) trial, and according to Strauss et al., the primary endpoint was left ventricular assist device implantation, cardiac transplantation, and all-cause mortality. RESULTS: LBBB classification differed significantly between the 4 definitions (kappa coefficients ranging from 0.09 to 0.92). The American Heart Association definition correlated the least (0.09 to 0.12) with the other definitions. Only 13.8% of patients were classified as LBBB by all definitions. During a follow-up period of 3.4 ± 2.4 years, 472 (32%) patients experienced the primary endpoint. For each LBBB definition survival analysis showed a significant association of LBBB with outcome, with relative risk reduction ranging from 39% to 43%. Each LBBB definition included characteristics that were not associated with outcome. Combining outcome-associated ECG characteristics into a novel prediction model did not significantly improve diagnostic performance (relative risk reduction 43%). CONCLUSIONS: The classification of LBBB is highly dependent on the LBBB definition used. However, each LBBB definition provides a comparable difference in risk of adverse clinical events between LBBB and non-LBBB patients. Combining individual outcome-associated ECG-characteristics into a novel prediction model does not improve association with outcome.

Medically Unexplained Oropharyngeal Dysphagia at the University Hospital ENT Outpatient Clinic for Dysphagia: A Cross-Sectional Cohort Study.

Medically unexplained oropharyngeal dysphagia (MUNOD) is a rare condition. It presents without demonstrable abnormalities in the anatomy of the upper aero-digestive tract and/or swallowing physiology. This study investigates whether MUNOD is related to affective or other psychiatric conditions. The study included patients with dysphagic complaints who had no detectible structural or physiological abnormalities upon swallowing examination. Patients with any underlying disease or disorder that could explain the oropharyngeal dysphagia were excluded. All patients underwent a standardized examination protocol, with FEES examination, the Hospital Anxiety and Depression Scale (HADS), and the Dysphagia Severity Scale (DSS). Two blinded judges scored five different FEES variables. None of the 14 patients included in this study showed any structural or physiological abnormalities during FEES examination. However, the majority did show abnormal piecemeal deglutition, which could be a symptom of MUNOD. Six patients (42.8%) had clinically relevant symptoms of anxiety and/or depression. The DSS scores did not differ significantly between patients with and without affective symptoms. Affective symptoms are common in patients with MUNOD, and their psychiatric conditions could possibly be related to their swallowing problems.

Asymptotic variability of (multilevel) multirater kappa coefficients.

Agreement studies are of paramount importance in various scientific domains. When several observers classify objects on categorical scales, agreement can be quantified through multirater kappa coefficients. In most statistical packages, the standard error of these coefficients is only available under the null hypothesis that the coefficient is equal to zero, preventing the construction of confidence intervals in the general case. The aim of this paper is triple. First, simple analytic formulae for the standard error of multirater kappa coefficients will be given in the general case. Second, these formulae will be extended to the case of multilevel data structures. The formulae are based on simple matrix algebra and are implemented in the R package "multiagree". Third, guidelines on the choice between the different mulitrater kappa coefficients will be provided.

Modeling agreement on bounded scales.

Agreement is an important concept in medical and behavioral sciences, in particular in clinical decision making where disagreements possibly imply a different patient management. The concordance correlation coefficient is an appropriate measure to quantify agreement between two scorers on a quantitative scale. However, this measure is based on the first two moments, which could poorly summarize the shape of the score distribution on bounded scales. Bounded outcome scores are common in medical and behavioral sciences. Typical examples are scores obtained on visual analog scales and scores derived as the number of positive items on a questionnaire. These kinds of scores often show a non-standard distribution, like a J- or U-shape, questioning the usefulness of the concordance correlation coefficient as agreement measure. The logit-normal distribution has shown to be successful in modeling bounded outcome scores of two types: (1) when the bounded score is a coarsened version of a latent score with a logit-normal distribution on the [0,1] interval and (2) when the bounded score is a proportion with the true probability having a logit-normal distribution. In the present work, a model-based approach, based on a bivariate generalization of the logit-normal distribution, is developed in a Bayesian framework to assess the agreement on bounded scales. This method permits to directly study the impact of predictors on the concordance correlation coefficient and can be simply implemented in standard Bayesian softwares, like JAGS and WinBUGS. The performances of the new method are compared to the classical approach using simulations. Finally, the methodology is used in two different medical domains: cardiology and rheumatology.

Observer Agreement for Measurements in Videolaryngostroboscopy.

OBJECTIVE: This study evaluated the levels of intraobserver and interobserver agreement for measurements of visuoperceptual variables in videolaryngostroboscopic examinations and compared the observers' behavior during independent versus consensus panel rating. STUDY DESIGN: This is a retrospective study. SETTING: This study was conducted in a single-center tertiary care facility. PARTICIPANTS: Sixty-four patients with dysphonia of heterogeneous etiology were included. EXPOSURE: All subjects underwent a standardized videolaryngostroboscopic examination. MAIN OUTCOME AND MEASURES: Two experienced and trained observers scored exactly the same examinations, first independently and then on a consensus panel. Specific visuoperceptual variables and the clinical diagnosis (as recommended by the Committee on Phoniatrics and the Phonosurgery Committee of the European Laryngological Society and advised by the American Speech-Language-Hearing Association) were scored. Descriptive and kappa statistics were used. RESULTS: In general, intraobserver agreement was better than agreement between observers for measurements of several variables. The intrapanel observer agreement levels were slightly higher than the intraobserver agreement levels on the independent rating task. When rating on the consensus panel, the observers deviated considerably from the scores they had previously given on the independent rating task. CONCLUSION AND RELEVANCE: Observer agreement in videolaryngostroboscopic assessment has important implications not only for the diagnosis and treatment of dysphonic patients but also for the interpretation of the results of scientific studies using videolaryngostroboscopic outcome parameters. The identification of factors that can influence the levels of observer agreement can provide a better understanding of the rating process and its limitations. The results of this study suggest that future research could achieve better agreement levels by rating the visuoperceptual variables in a panel setting.

Affective symptoms in patients with oropharyngeal dysphagia: A systematic review.

OBJECTIVE: Affective disorders are prevalent in different somatic conditions and influence somatic symptom bother and quality of life. Mood and anxiety disorders impact patients' compliance and adherence to treatment. This systematic review summarizes published studies on affective complaints in patients with oropharyngeal dysphagia (OD) in order to determine the quality of studies concerning any association of OD with symptoms of depression and/or anxiety. METHODS: A literature search was carried out using electronic databases Embase, Medline, Web-of-science, PsycINFO, Cochrane Library, and Google scholar. Two reviewers made the preselecting cut by screening all articles on title and abstract and independently screened the full texts of this initial set of articles. Methodological quality of the studies that met the inclusion criteria was assessed independently. RESULTS: Twenty-six articles were included in the analysis after full-text screening and by applying the inclusion and exclusion criteria. All studies concluded that symptoms of depression were associated with impaired swallowing function, and 9 out of 12 studies concluded that symptoms of anxiety were associated with functional impairment of swallowing. The reviewers found heterogeneous outcomes and methodological limitations, which prevented data from pooling. CONCLUSION: Although no meta-analytic conclusions can be drawn, it appears that symptoms of anxiety and depression are common in OD. Caregivers have to be aware of this in order to detect affective comorbidity. Given that affective conditions influence patients' treatment adherence and compliance, integrated care approaches should be advocated in case of comorbidity. Studies on treatment effect are lacking and well-designed prospective research is needed.

Comparing dependent kappa coefficients obtained on multilevel data.

Reliability and agreement are two notions of paramount importance in medical and behavioral sciences. They provide information about the quality of the measurements. When the scale is categorical, reliability and agreement can be quantified through different kappa coefficients. The present paper provides two simple alternatives to more advanced modeling techniques, which are not always adequate in case of a very limited number of subjects, when comparing several dependent kappa coefficients obtained on multilevel data. This situation frequently arises in medical sciences, where multilevel data are common. Dependent kappa coefficients can result from the assessment of the same individuals at various occasions or when each member of a group is compared to an expert, for example. The method is based on simple matrix calculations and is available in the R package "multiagree". Moreover, the statistical properties of the proposed method are studied using simulations. Although this paper focuses on kappa coefficients, the method easily extends to other statistical measures.

International perception of lung sounds: a comparison of classification across some European borders.

INTRODUCTION: Lung auscultation is helpful in the diagnosis of lung and heart diseases; however, the diagnostic value of lung sounds may be questioned due to interobserver variation. This situation may also impair clinical research in this area to generate evidence-based knowledge about the role that chest auscultation has in a modern clinical setting. The recording and visual display of lung sounds is a method that is both repeatable and feasible to use in large samples, and the aim of this study was to evaluate interobserver agreement using this method. METHODS: With a microphone in a stethoscope tube, we collected digital recordings of lung sounds from six sites on the chest surface in 20 subjects aged 40 years or older with and without lung and heart diseases. A total of 120 recordings and their spectrograms were independently classified by 28 observers from seven different countries. We employed absolute agreement and kappa coefficients to explore interobserver agreement in classifying crackles and wheezes within and between subgroups of four observers. RESULTS: When evaluating agreement on crackles (inspiratory or expiratory) in each subgroup, observers agreed on between 65% and 87% of the cases. Conger's kappa ranged from 0.20 to 0.58 and four out of seven groups reached a kappa of ≥0.49. In the classification of wheezes, we observed a probability of agreement between 69% and 99.6% and kappa values from 0.09 to 0.97. Four out of seven groups reached a kappa ≥0.62. CONCLUSIONS: The kappa values we observed in our study ranged widely but, when addressing its limitations, we find the method of recording and presenting lung sounds with spectrograms sufficient for both clinic and research. Standardisation of terminology across countries would improve international communication on lung auscultation findings.

Observers' Agreement on Measurements in Fiberoptic Endoscopic Evaluation of Swallowing.

This study analyzed the effect that dysphagia etiology, different observers, and bolus consistency might have on the level of agreement for measurements in FEES images reached by independent versus consensus panel rating. Sixty patients were included and divided into two groups according to dysphagia etiology: neurological or head and neck oncological. All patients underwent standardized FEES examination using thin and thick liquid consistencies. Two observers scored the same exams, first independently and then in a consensus panel. Four ordinal FEES variables were analyzed. Statistical analysis was performed using a linear weighted kappa coefficient and Bayesian multilevel model. Intra- and interobserver agreement on FEES measurements ranged from 0.76 to 0.93 and from 0.61 to 0.88, respectively. Dysphagia etiology did not influence observers' agreement level. However, bolus consistency resulted in decreased interobserver agreement for all measured FEES variables during thin liquid swallows. When rating on the consensus panel, the observers deviated considerably from the scores they had previously given on the independent rating task. Observer agreement on measurements in FEES exams was influenced by bolus consistency, not by dysphagia etiology. Therefore, observer agreement on FEES measurements should be analyzed by taking bolus consistency into account, as it might affect the interpretation of the outcome. Identifying factors that might influence agreement levels could lead to better understanding of the rating process and assist in developing a more precise measurement scale that would ensure higher levels of observer agreement for measurements in FEES exams.