Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability.

Inter-rater reliability (IRR) is one of the commonly used tools for assessing the quality of ratings from multiple raters. However, applicant selection procedures based on ratings from multiple raters usually result in a binary outcome; the applicant is either selected or not. This final outcome is not considered in IRR, which instead focuses on the ratings of the individual subjects or objects. We outline the connection between the ratings' measurement model (used for IRR) and a binary classification framework. We develop a simple way of approximating the probability of correctly selecting the best applicants which allows us to compute error probabilities of the selection procedure (i.e., false positive and false negative rate) or their lower bounds. We draw connections between the IRR and the binary classification metrics, showing that binary classification metrics depend solely on the IRR coefficient and proportion of selected applicants. We assess the performance of the approximation in a simulation study and apply it in an example comparing the reliability of multiple grant peer review selection procedures. We also discuss other possible uses of the explored connections in other contexts, such as educational testing, psychological assessment, and health-related measurement, and implement the computations in the R package IRR2FPR.

Combining Item Purification and Multiple Comparison Adjustment Methods in Detection of Differential Item Functioning.

Many of the differential item functioning (DIF) detection methods rely on a principle of testing for DIF item by item, while considering the rest of the items or at least some of them being DIF-free. Computational algorithms of these DIF detection methods involve the selection of DIF-free items in an iterative procedure called item purification. Another aspect is the need to correct for multiple comparisons, which can be done with a number of existing multiple comparison adjustment methods. In this article, we demonstrate that implementation of these two controlling procedures together may have an impact on which items are detected as DIF items. We propose an iterative algorithm combining item purification and adjustment for multiple comparisons. Pleasant properties of the newly proposed algorithm are shown with a simulation study. The method is demonstrated on a real data example.

COMIRESTROKE-A clinical study protocol for monitoring clinical effect and molecular biological readouts of COMprehensive Intensive REhabilitation program after STROKE: A four-arm parallel-group randomized double blinded controlled trial with a longitudinal design.

Introduction: While the role of physiotherapy as part of a comprehensive inpatient rehabilitation is indisputable, clear evidence concerning the effectiveness of different rehabilitation managements [interdisciplinary implementing the International Classification of Functioning, disability and health (ICF) vs. multidisciplinary model] and physiotherapy categories (neuroproprioceptive "facilitation, inhibition" vs. motor/skill acquisitions using technologies) are still lacking. In this study, four kinds of comprehensive inpatient rehabilitation with different management and content of physical therapy will be compared. Moreover, focus will be placed on the identification of novel biological molecules reflective of effective rehabilitation. Long non-coding RNAs (lncRNAs) are transcripts (>200 bps) of limited coding potential, which have recently been recognized as key factors in neuronal signaling pathways in ischemic stroke and as such, may provide a valuable readout of patient recovery and neuroprotection during therapeutic progression. Methods and analysis: Adults after the first ischemic stroke in an early sub-acute phase with motor disability will be randomly assigned to one of four groups and undergo a 3 weeks comprehensive inpatient rehabilitation of different types: interdisciplinary team work using ICF model as a guide; multidisciplinary teamwork implementing neuroproprioceptive "facilitation and inhibition" physiotherapy; multidisciplinary teamwork implementing technology-based physiotherapy; and standard multidisciplinary teamwork. Primary (the Goal Attainment Scale, the Patient-Reported Outcomes Measurement Information System, and the World Health Organization Disability Assessment Schedule) and secondary (motor, cognitive, psychological, speech and swallowing functions, functional independence) outcomes will be measured. A blood sample will be obtained upon consent (20 mls; representing pre-rehabilitation molecular) before and after the inpatient program. Primary outcomes will be followed up again 3 and 12 months after the end of the program. The overarching aim of this study is to determine the effectiveness of various rehabilitation managements and physiotherapeutic categories implemented by patients post ischemic stroke via analysis of primary, secondary and long non-coding RNA readouts. This clinical trial will offer an innovative approach not previously tested and will provide new complex analysis along with public assessable molecular biological evidence of various rehabilitation methodology for the alleviation of the effects of ischemic stroke. Clinical trial registration: NCT05323916, https://clinicaltrials.gov/ct2/show/NCT05323916.

Real-World Goal Setting and Use of Outcome Measures According to the International Classification of Functioning, Disability and Health: A European Survey of Physical Therapy Practice in Multiple Sclerosis.

Goal setting is a core component of physical therapy in multiple sclerosis (MS). It is unknown whether and to what extent goals are set at different levels of the International Classification of Functioning, Disability and Health (ICF), and whether, and to which, standardized outcome measures are used in real life for evaluation at the different ICF levels. Our aim was to describe the real-world use of goal setting and outcome measures in Europe. An online cross-sectional survey, completed by 212 physical therapists (PTs) specialized in MS from 26 European countries, was conducted. Differences between European regions and relationships between goals and assessments were analyzed. PTs regularly set goals, but did not always apply the Specific, Measurable, Achievable, Realistic, Timed (SMART) criteria. Regions did not differ in the range of activities assessed, but in goals set (e.g., Western and Northern regions set significantly more goals regarding leisure and work) and outcome measures used (e.g., the Berg Balance Scale was more frequently used in Northern regions). Quality of life was not routinely assessed, despite being viewed as an important therapy goal. Discrepancies existed both in goal setting and assessment across European regions. ICF assists in understanding these discrepancies and in guiding improved health-care for the future.

Content and Delivery of Physical Therapy in Multiple Sclerosis across Europe: A Survey.

Background: Guidelines and general recommendations are available for multiple sclerosis rehabilitation, but no specific guidance exists for physical therapists. Describing aspects of physical therapy content and delivery in multiple sclerosis and its determinants and analysing whether general recommendations connected with physical therapy are implemented in practice is important for interpreting clinical and research evidence. Methods: An online cross-sectional survey of physical therapists specialized in multiple sclerosis (212 specialists from 26 European countries) was used. Results: There was distinct diversity in service delivery and content across Europe. Perceived accessibility of physical therapy varied from most accessible in the Western region, and least in the Southern region. Sixty-four physical therapists adjusted their approach according to different disability levels, less so in the Eastern region. Duration, frequency and dose of sessions differed between regions, being highest in Southern and Western regions. "Hands on treatment" was the most commonly used therapeutic approach in all apart from the Northern regions, where "word instruction" (providing advice and information) prevailed. Conclusions: The content and delivery of physical therapy differs across Europe. Recommendations concerning access to treatment and adjustment according to disability do not appear to be widely implemented in clinical practice.

Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability.

Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market.

Physiotherapeutic interventions in multiple sclerosis across Europe: Regions and other factors that matter.

BACKGROUND: A wide variety of interventions exists in physical therapy (PT), but knowledge about their use across different geographical regions is limited. This study investigated the use of PT interventions in people with multiple sclerosis (MS) across Europe. It aimed to determine whether regions differ in applying interventions, and explore whether factors other than regions play a role in their use. METHODS: In an online cross-sectional survey, 212 respondents from 115 European workplaces providing PT services to people with MS representing 26 countries (four European regions) participated. Cluster analysis, Pearson Chi-squared test and a Poisson regression model were used to analyze the data. RESULTS: Thirteen of 45 listed PT interventions were used by more than 75% of centers, while nine interventions were used by less than 25%. For 12 interventions, regions differed markedly in their use. Cluster analysis of centers identified four clusters similar in their intervention use. Cluster assignment did not fully align with regions. While center region was important, center size, number and gender of physical therapists working in the center, and time since qualification also played a role. Cluster analysis exploring the use of the interventions provided the basis for a categorization of PT interventions in line with their primary focus: 1. Physical activity (fitness/endurance/resistance) training; 2. Neuroproprioceptive "facilitation/inhibition"; 3. Motor/skill acquisition (individualized therapy led); 4. Technology based interventions. CONCLUSIONS: To our knowledge this is the first study that has explored this topic in MS. The results broaden our understanding of the different PT interventions used in MS, as well as the context of their use.

Development and Validation of the Homeostasis Concept Inventory.

We present the Homeostasis Concept Inventory (HCI), a 20-item multiple-choice instrument that assesses how well undergraduates understand this critical physiological concept. We used an iterative process to develop a set of questions based on elements in the Homeostasis Concept Framework. This process involved faculty experts and undergraduate students from associate's colleges, primarily undergraduate institutions, regional and research-intensive universities, and professional schools. Statistical results provided strong evidence for the validity and reliability of the HCI. We found that graduate students performed better than undergraduates, biology majors performed better than nonmajors, and students performed better after receiving instruction about homeostasis. We used differential item analysis to assess whether students from different genders, races/ethnicities, and English language status performed differently on individual items of the HCI. We found no evidence of differential item functioning, suggesting that the items do not incorporate cultural or gender biases that would impact students' performance on the test. Instructors can use the HCI to guide their teaching and student learning of homeostasis, a core concept of physiology.

Checking Equity: Why Differential Item Functioning Analysis Should Be a Routine Part of Developing Conceptual Assessments.

We provide a tutorial on differential item functioning (DIF) analysis, an analytic method useful for identifying potentially biased items in assessments. After explaining a number of methodological approaches, we test for gender bias in two scenarios that demonstrate why DIF analysis is crucial for developing assessments, particularly because simply comparing two groups' total scores can lead to incorrect conclusions about test fairness. First, a significant difference between groups on total scores can exist even when items are not biased, as we illustrate with data collected during the validation of the Homeostasis Concept Inventory. Second, item bias can exist even when the two groups have exactly the same distribution of total scores, as we illustrate with a simulated data set. We also present a brief overview of how DIF analysis has been used in the biology education literature to illustrate the way DIF items need to be reevaluated by content experts to determine whether they should be revised or removed from the assessment. Finally, we conclude by arguing that DIF analysis should be used routinely to evaluate items in developing conceptual assessments. These steps will ensure more equitable-and therefore more valid-scores from conceptual assessments.

Validating a conceptual framework for the core concept of "cell-cell communication".

We have created and validated a conceptual framework for the core physiology concept of "cell-cell communication." The conceptual framework is composed of 51 items arranged in a hierarchy that is, in some instances, four levels deep. We have validated it with input from faculty who teach at a wide variety of institutional types. All items making up the framework were deemed essential to moderately important. However, some of the main ideas were clearly judged to be more important than others. Furthermore, the lower in the hierarchy an item is, the less important it is thought to be. Finally, there was no significant difference in the ratings given by faculty at different types of institutions.

The organisation of physiotherapy for people with multiple sclerosis across Europe: a multicentre questionnaire survey.

BACKGROUND: Understanding the organisational set-up of physiotherapy services across different countries is increasingly important as clinicians around the world use evidence to improve their practice. This also has to be taken into consideration when multi-centre international clinical trials are conducted. This survey aimed to systematically describe organisational aspects of physiotherapy services for people with multiple sclerosis (MS) across Europe. METHODS: Representatives from 72 rehabilitation facilities within 23 European countries completed an online web-based questionnaire survey between 2013 and 2014. Countries were categorised according to four European regions (defined by United Nations Statistics). Similarities and differences between regions were examined. RESULTS: Most participating centres specialized in rehabilitation (82 %) and neurology (60 %), with only 38 % specialising in MS. Of these, the Western based Specialist MS centres were predominately based on outpatient services (median MS inpatient ratio 0.14), whilst the Eastern based European services were mostly inpatient in nature (median MS inpatient ratio 0.5). In almost all participating countries, medical doctors - specialists in neurology (60 %) and in rehabilitation (64 %) - were responsible for referral to/prescription of physiotherapy. The most frequent reason for referral to/prescription of physiotherapy was the worsening of symptoms (78 % of centres). Physiotherapists were the most common members of the rehabilitation team; comprising 49 % of the team in Eastern countries compared to approximately 30 % in the rest of Europe. Teamwork was commonly adopted; 86 % of centres based in Western countries utilised the interdisciplinary model, whilst the multidisciplinary model was utilised in Eastern based countries (p = 0.046). CONCLUSION: This survey is the first to provide data about organisational aspects of physiotherapy for people with MS across Europe. Overall, care in key organisational aspects of service provision is broadly similar across regions, although some variations, for example the models of teamwork utilised, are apparent. Organisational framework specifics should be considered anytime a multi-centre study is conducted and results from such studies are applied.

Assessment set for evaluation of clinical outcomes in multiple sclerosis: psychometric properties.

PURPOSE: Multiple sclerosis (MS) manifests itself in a wide range of symptoms. Physiotherapy plays an important role in the treatment of those symptoms connected with mobility. For this therapy to be at its most effective it should be based on a systematic examination that is able to describe and classify damaged clinical functions meaningfully. The purpose of this study was to develop and validate a battery of tests and composite tests that can be used to systematically evaluate clinical features of MS treatable by physiotherapy. METHODS: The authors assembled a proposed battery of tests comprising known, standard, and validated assessments (low-contrast letter acuity testing; the Motricity Index; the Modified Ashworth Scale; the Berg Balance Scale; scales of postural reactions, tremor, dysdiadochokinesia, and dysmetria; the Nine-Hole Peg Test; the Timed 25-Foot Walk; and the 3-minute version of the Paced Auditory Serial Addition Test) and one test (knee hyperextension) of the authors' own. Normalization was calculated and six composite assessments were measured. Seventeen ambulatory subjects with MS were tested twice with the assessment set before undergoing physiotherapy, and 12 were also tested with the assessment set after the physiotherapy. The test-retest reliability, stability, internal consistency of composite measurements, sensitivity to changes after therapy, and correlation between measurements and the Kurtzke Expanded Disability Status Scale score were evaluated for all tests in the assessment set. RESULTS: A good internal consistency was confirmed for all tests in the proposed battery, and most of the tests also showed good test-retest reliability. While no significant changes occurred without treatment, significant posttreatment improvement was proved in all tests except for low-contrast letter acuity testing, where only a trend to improvement was proved. CONCLUSION: The proposed assessment set is a good tool for the evaluation of clinical features of MS treatable by physiotherapy. This battery of tests is applicable in both clinical practice and research.

Physiotherapy as an immunoactive therapy? A pilot study.

OBJECTIVES: The aim of this study was to confirm the immunoregulatory and anti-inflammatory changes in the immunologic profile after two months of the facilitation physiotherapy in patients with multiple sclerosis; and to determine whether the changes in the immunologic profile correlate with the changes in dehydroepiandrosterone, the brain microstructure and clinical functions. DESIGN & SETTING: A group of 12 patients with multiple sclerosis was examined twice: at the beginning and 2 months later after the patients had undergone the facilitation therapy. Standardized tests evaluating chosen clinical functions (balance, righting, equilibrium and protective reactions, tremor, dysdiadochokinesis, dysmetry, fine hand function and walking), immune parameters (parameters of the humoral and cellular immunity), dehydroepiandrosterone and diffusion tensor imaging (the fractional anisotropy, mean diffusivity) were measured. The patients underwent the facilitation physiotherapy in two sessions lasting two hours each week for two months. RESULTS: All clinical and diffusion tensor imaging parameters significantly improved following the therapy. Without the correction for multiple comparisons, there were significant changes in the IgG, IgG1 subclasses, in the numbers of Neutrophils and Lymphocytes, the T cells (CD3+) absolute number, the T cytotoxic subpopulation (CD3+CD8+) absolute number, B cells (CD19+) and the Natural killer cells. In addition, there was a significant correlation between the changes in the clinical functions and the changes in IgG1 (r=0.67), and between the changes in the mean diffusivity and the changes in CD3+CD8+ absolute (r=-0.61). The changes in the immune parameters and the mentioned correlations were not significant in view of the number of comparisons and thus necessitate further validation. No changes in the dehydroepiandrosterone concentration after the therapy were confirmed. CONCLUSION: The study suggests new possibilities of physiotherapy to influence the psycho-neuro-endocrine-immune response in patients with multiple sclerosis.

Fractional anisotropy and mean diffusivity in the corpus callosum of patients with multiple sclerosis: the effect of physiotherapy.

INTRODUCTION: Modulation of neurodegeneration by physical activity is an active topic in contemporary research. The purpose of this study was to investigate changes in the brain's microstructure in multiple sclerosis (MS) after facilitation physiotherapy. METHODS: Eleven patients with MS were examined using motor and neuropsychological testing and multimodal MRI at the beginning of the study, with second baseline measurement after 1 month without any therapy, and after a 2-month period of facilitation physiotherapy. Eleven healthy controls were examined at the beginning of the study and after 1 month. Fractional anisotropy (FA), mean diffusivity (MD), axial diffusivity (λ (ax)), and radial diffusivity (λ (rad)) were calculated for the whole corpus callosum (CC) in the midsagittal slice of T1W 3D MPRAGE spatially normalized images. Data were analyzed using linear mixed-effect models, paired, and two-sample tests. RESULTS: At the baseline, patients with MS showed significantly lower values in FA (p < 0.001), and significantly higher values in MD (p < 0.001), λ (ax) (p = 0.003), and λ (rad) (p < 0.001) compared to control subjects. The FA, MD, λ (ax), and λ (rad) did not change between the first and second baseline examinations in either group. Differences 2 months after initiating facilitation physiotherapy were in FA, MD, and in λ (rad) significantly higher than differences in healthy controls (p < 0.001 for FA, p = 0.02 for MD, and p = 0.002 for λ (rad)). In MS patients, FA in the CC significantly increased (p < 0.001), MD and λ (rad) significantly decreased (p = 0.014 and p = 0.002), and thus approached the values in healthy controls. CONCLUSION: The results of the study show that facilitation physiotherapy influences brain microstructure measured by DTI.