A new approach to grant review assessments: score, then rank.

BACKGROUND: In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications. METHODS: We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical "toy" examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges' evaluations. RESULTS: For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions. CONCLUSIONS: A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

Gender-based homophily in collaborations across a heterogeneous scholarly landscape.

In this article, we investigate the role of gender in collaboration patterns by analyzing gender-based homophily-the tendency for researchers to co-author with individuals of the same gender. We develop and apply novel methodology to the corpus of JSTOR articles, a broad scholarly landscape, which we analyze at various levels of granularity. Most notably, for a precise analysis of gender homophily, we develop methodology which explicitly accounts for the fact that the data comprises heterogeneous intellectual communities and that not all authorships are exchangeable. In particular, we distinguish three phenomena which may affect the distribution of observed gender homophily in collaborations: a structural component that is due to demographics and non-gendered authorship norms of a scholarly community, a compositional component which is driven by varying gender representation across sub-disciplines and time, and a behavioral component which we define as the remainder of observed gender homophily after its structural and compositional components have been taken into account. Using minimal modeling assumptions, the methodology we develop allows us to test for behavioral homophily. We find that statistically significant behavioral homophily can be detected across the JSTOR corpus and show that this finding is robust to missing gender indicators in our data. In a secondary analysis, we show that the proportion of women representation in a field is positively associated with the probability of finding statistically significant behavioral homophily.

Anti-SARS-CoV-2 seroprevalence in King County, WA-Cross-sectional survey, August 2020.

We conducted a seroprevalence survey to estimate the true number of infections with SARS-CoV-2, the virus that causes COVID-19, in King County as of August 2020 by measuring the proportion of residents from who had antibodies against the virus. Participants from 727 households took part in a cross-sectional address-based household survey with random and non-random samples and provided dried blood spots that were tested for total antibody against the viral nucleocapsid protein, with confirmatory testing for immunoglobulin G against the spike protein. The data were weighted to match King County's population based on sex, age group, income, race, and Hispanic status. After weighting and accounting for the accuracy of the tests, our best overall estimate of anti-SARS-CoV-2 seroprevalence in King County as of August 2020 is 3.9% (95% confidence interval (CI) 2.4%-6.0%) with an effective sample size of 589. Comparing seroprevalence with positive test reports, our survey suggests that viral testing underestimated incidence by a factor of about five and suggests that the proportion of cases that were serious (based on hospitalization) or fatal was 2.4% and 0.8%, respectively. Prevalence varied by subgroup; households reporting incomes at or below $100,000 in 2019 had nearly five times higher estimated antibody prevalence than those with incomes above $100,000. Those reporting non-White/non-Asian race had roughly seven times higher estimated antibody prevalence than those reporting White race. This survey was noteworthy for including people of all ages; among all age groups, the weighted estimate of prevalence was highest in older teens and young adults and lowest in young children, although these differences were not statistically significant.

Refinement: Measuring informativeness of ratings in the absence of a gold standard.

We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings - entropic refinement - as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which captures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyse refinement in real-world grant-review data, finding evidence that overall merit scores are more refined than criterion scores.

Co-clustering of Time-Dependent Data via the Shape Invariant Model.

Multivariate time-dependent data, where multiple features are observed over time for a set of individuals, are increasingly widespread in many application domains. To model these data, we need to account for relations among both time instants and variables and, at the same time, for subject heterogeneity. We propose a new co-clustering methodology for grouping individuals and variables simultaneously, designed to handle both functional and longitudinal data. Our approach borrows some concepts from the curve registration framework by embedding the shape invariant model in the latent block model, estimated via a suitable modification of the SEM-Gibbs algorithm. The resulting procedure allows for several user-defined specifications of the notion of cluster that can be chosen on substantive grounds and provides parsimonious summaries of complex time-dependent data by partitioning data matrices into homogeneous blocks. Along with the explicit modelling of time evolution, these aspects allow for an easy interpretation of the clusters, from which also low-dimensional settings may benefit.

Latent Class Analysis: Insights about design and analysis of schistosomiasis diagnostic studies.

Various global health initiatives are currently advocating the elimination of schistosomiasis within the next decade. Schistosomiasis is a highly debilitating tropical infectious disease with severe burden of morbidity and thus operational research accurately evaluating diagnostics that quantify the epidemic status for guiding effective strategies is essential. Latent class models (LCMs) have been generally considered in epidemiology and in particular in recent schistosomiasis diagnostic studies as a flexible tool for evaluating diagnostics because assessing the true infection status (via a gold standard) is not possible. However, within the biostatistics literature, classical LCM have already been criticised for real-life problems under violation of the conditional independence (CI) assumption and when applied to a small number of diagnostics (i.e. most often 3-5 diagnostic tests). Solutions of relaxing the CI assumption and accounting for zero-inflation, as well as collecting partial gold standard information, have been proposed, offering the potential for more robust model estimates. In the current article, we examined such approaches in the context of schistosomiasis via analysis of two real datasets and extensive simulation studies. Our main conclusions highlighted poor model fit in low prevalence settings and the necessity of collecting partial gold standard information in such settings in order to improve the accuracy and reduce bias of sensitivity and specificity estimates.

NIH peer review: Criterion scores completely account for racial disparities in overall impact scores.

Previous research has found that funding disparities are driven by applications' final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014-2016, we examine assigned reviewers' preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration-the process of combining criterion scores into overall impact scores-disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities-yet do not explain all of the variability-in preliminary overall impact scores.

Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability.

Ratings are present in many areas of assessment including peer review of research proposals and journal articles, teacher observations, university admissions and selection of new hires. One feature present in any rating process with multiple raters is that different raters often assign different scores to the same assessee, with the potential for bias and inconsistencies related to rater or assessee covariates. This paper analyzes disparities in ratings of internal and external applicants to teaching positions using applicant data from Spokane Public Schools. We first test for biases in rating while accounting for measures of teacher applicant qualifications and quality. Then, we develop model-based inter-rater reliability (IRR) estimates that allow us to account for various sources of measurement error, the hierarchical structure of the data, and to test whether covariates, such as applicant status, moderate IRR. We find that applicants external to the district receive lower ratings for job applications compared to internal applicants. This gap in ratings remains significant even after including measures of qualifications and quality such as experience, state licensure scores, or estimated teacher value added. With model-based IRR, we further show that consistency between raters is significantly lower when rating external applicants. We conclude the paper by discussing policy implications and possible applications of our model-based IRR estimate for hiring and selection practices in and out of the teacher labor market.

Physiotherapeutic interventions in multiple sclerosis across Europe: Regions and other factors that matter.

BACKGROUND: A wide variety of interventions exists in physical therapy (PT), but knowledge about their use across different geographical regions is limited. This study investigated the use of PT interventions in people with multiple sclerosis (MS) across Europe. It aimed to determine whether regions differ in applying interventions, and explore whether factors other than regions play a role in their use. METHODS: In an online cross-sectional survey, 212 respondents from 115 European workplaces providing PT services to people with MS representing 26 countries (four European regions) participated. Cluster analysis, Pearson Chi-squared test and a Poisson regression model were used to analyze the data. RESULTS: Thirteen of 45 listed PT interventions were used by more than 75% of centers, while nine interventions were used by less than 25%. For 12 interventions, regions differed markedly in their use. Cluster analysis of centers identified four clusters similar in their intervention use. Cluster assignment did not fully align with regions. While center region was important, center size, number and gender of physical therapists working in the center, and time since qualification also played a role. Cluster analysis exploring the use of the interventions provided the basis for a categorization of PT interventions in line with their primary focus: 1. Physical activity (fitness/endurance/resistance) training; 2. Neuroproprioceptive "facilitation/inhibition"; 3. Motor/skill acquisition (individualized therapy led); 4. Technology based interventions. CONCLUSIONS: To our knowledge this is the first study that has explored this topic in MS. The results broaden our understanding of the different PT interventions used in MS, as well as the context of their use.

A SIMULATION-BASED FRAMEWORK FOR ASSESSING THE FEASIBILITY OF RESPONDENT-DRIVEN SAMPLING FOR ESTIMATING CHARACTERISTICS IN POPULATIONS OF LESBIAN, GAY AND BISEXUAL OLDER ADULTS.

Respondent-driven sampling (RDS) is a method for sampling from a target population by leveraging social connections. RDS is invaluable to the study of hard-to-reach populations. However, RDS is costly and can be infeasible. RDS is infeasible when RDS point estimators have small effective sample sizes (large design effects) or when RDS interval estimators have large confidence intervals relative to estimates obtained in previous studies or poor coverage. As a result, researchers need tools to assess whether or not estimation of certain characteristics of interest for specific populations is feasible in advance. In this paper, we develop a simulation-based framework for using pilot data-in the form of a convenience sample of aggregated, egocentric data and estimates of subpopulation sizes within the target population-to assess whether or not RDS is feasible for estimating characteristics of a target population. in doing so, we assume that more is known about egos than alters in the pilot data, which is often the case with aggregated, egocentric data in practice. We build on existing methods for estimating the structure of social networks from aggregated, egocentric sample data and estimates of subpopulation sizes within the target population. We apply this framework to assess the feasibility of estimating the proportion male, proportion bisexual, proportion depressed and proportion infected with HIV/AIDS within three spatially distinct target populations of older lesbian, gay and bisexual adults using pilot data from the caring and Aging with Pride Study and the Gallup Daily Tracking Survey. We conclude that using an RDS sample of 300 subjects is infeasible for estimating the proportion male, but feasible for estimating the proportion bisexual, proportion depressed and proportion infected with HIV/AIDS in all three target populations.

Dealing with Reflection Invariance in Bayesian Factor Analysis.

This paper considers the reflection unidentifiability problem in confirmatory factor analysis (CFA) and the associated implications for Bayesian estimation. We note a direct analogy between the multimodality in CFA models that is due to all possible column sign changes in the matrix of loadings and the multimodality in finite mixture models that is due to all possible relabelings of the mixture components. Drawing on this analogy, we derive and present a simple approach for dealing with reflection in variance in Bayesian factor analysis. We recommend fitting Bayesian factor analysis models without rotational constraints on the loadings-allowing Markov chain Monte Carlo algorithms to explore the full posterior distribution-and then using a relabeling algorithm to pick a factor solution that corresponds to one mode. We demonstrate our approach on the case of a bifactor model; however, the relabeling algorithm is straightforward to generalize for handling multimodalities due to sign invariance in the likelihood in other factor analysis models.

Social Networks of Lesbian, Gay, Bisexual, and Transgender Older Adults.

PURPOSE: This study examines global social networks-including friendship, support, and acquaintance networks-of lesbian, gay, bisexual, and transgender (LGBT) older adults. DESIGN AND METHODS: Utilizing data from a large community-based study, we employ multiple regression analyses to examine correlates of social network size and diversity. RESULTS: Controlling for background characteristics, network size was positively associated with being female, transgender identity, employment, higher income, having a partner or a child, identity disclosure to a neighbor, engagement in religious activities, and service use. Controlling in addition for network size, network diversity was positively associated with younger age, being female, transgender identity, identity disclosure to a friend, religious activity, and service use. IMPLICATIONS: According to social capital theory, social networks provide a vehicle for social resources that can be beneficial for successful aging and well-being. This study is a first step at understanding the correlates of social network size and diversity among LGBT older adults.

Physical and mental health of transgender older adults: an at-risk and underserved population.

PURPOSE: This study is one of the first to examine the physical and mental health of transgender older adults and to identify modifiable factors that account for health risks in this underserved population. DESIGN AND METHODS: Utilizing data from a cross-sectional survey of lesbian, gay, bisexual, and transgender older adults aged 50 and older (N = 2,560), we assessed direct and indirect effects of gender identity on 4 health outcomes (physical health, disability, depressive symptomatology, and perceived stress) based on a resilience conceptual framework. RESULTS: Transgender older adults were at significantly higher risk of poor physical health, disability, depressive symptomatology, and perceived stress compared with nontransgender participants. We found significant indirect effects of gender identity on the health outcomes via fear of accessing health services, lack of physical activity, internalized stigma, victimization, and lack of social support; other mediators included obesity for physical health and disability, identity concealment for perceived stress, and community belonging for depressive symptomatology and perceived stress. Further analyses revealed that risk factors (victimization and stigma) explained the highest proportion of the total effect of gender identity on health outcomes. IMPLICATIONS: The study identifies important modifiable factors (stigma, victimization, health-related behaviors, and social support) associated with health among transgender older adults. Reducing stigma and victimization and including gender identity in nondiscrimination and hate crime statutes are important steps to reduce health risks. Attention to bolstering individual and community-level social support must be considered when developing tailored interventions to address transgender older adults' distinct health and aging needs.

Using group-based latent class transition models to analyze chronic disability data from the National Long-Term Care Survey 1984-2004.

Latent class transition models track how individuals move among latent classes through time, traditionally assuming a complete set of observations for each individual. In this paper, we develop group-based latent class transition models that allow for staggered entry and exit, common in surveys with rolling enrollment designs. Such models are conceptually similar to, but structurally distinct from, pattern mixture models of the missing data literature. We employ group-based latent class transition modeling to conduct an in-depth data analysis of recent trends in chronic disability among the U.S. elderly population. Using activities of daily living data from the National Long-Term Care Survey (NLTCS), 1982-2004, we estimate model parameters using the expectation-maximization algorithm, implemented in SAS PROC IML. Our findings indicate that declines in chronic disability prevalence, observed in the 1980s and 1990s, did not continue in the early 2000s as previous NLTCS cross-sectional analyses have indicated.

The physical and mental health of lesbian, gay male, and bisexual (LGB) older adults: the role of key health indicators and risk and protective factors.

PURPOSE: Based on resilience theory, this paper investigates the influence of key health indicators and risk and protective factors on health outcomes (including general health, disability, and depression) among lesbian, gay male, and bisexual (LGB) older adults. DESIGN AND METHODS: A cross-sectional survey was conducted with LGB older adults, aged 50 and older (N = 2,439). Logistic regressions were conducted to examine the contributions of key health indicators (access to health care and health behaviors), risk factors (lifetime victimization, internalized stigma, and sexual identity concealment), and protective factors (social support and social network size) to health outcomes, when controlling for background characteristics. RESULTS: The findings revealed that lifetime victimization, financial barriers to health care, obesity, and limited physical activity independently and significantly accounted for poor general health, disability, and depression among LGB older adults. Internalized stigma was also a significant predictor of disability and depression. Social support and social network size served as protective factors, decreasing the odds of poor general health, disability, and depression. Some distinct differences by gender and sexual orientation were also observed. IMPLICATIONS: High levels of poor general health, disability, and depression among LGB older adults are of major concern. These findings highlight the important role of key risk and protective factors, which significantly influences health outcomes among LGB older adults. Tailored interventions must be developed to address the distinct health issues facing this historically disadvantaged population.

CSF biomarker associations with change in hippocampal volume and precuneus thickness: implications for the Alzheimer's pathological cascade.

Neurofibrillary tangles (NFT) and amyloid plaques are hallmark neuropathological features of Alzheimer's disease (AD). There is some debate as to which neuropathological feature comes first in the disease process, with early autopsy studies suggesting that NFT develop first, and more recent neuroimaging studies supporting the early role of amyloid beta (Aß) deposition. Cerebrospinal fluid (CSF) biomarkers of Aß42 and hyperphosphorylated tau (p-tau) have been shown to serve as in vivo proxy measures of amyloid plaques and NFT, respectively. The aim of this study was to examine the association between CSF biomarkers and rate of atrophy in the precuneus and hippocampus. These regions were selected because the precuneus appears to be affected early and severely by Aß deposition, and the hippocampus similarly by NFT pathology. We predicted (1) baseline Aß42 would be related to accelerated rate of cortical thinning in the precuneus and volume loss in the hippocampus, with the latter relationship expected to be weaker, (2) baseline p-tau(181p) would be related to accelerated rate of hippocampal atrophy and cortical thinning in the precuneus, with the latter relationship expected to be weaker. Using all ADNI cohorts, we fitted separate linear mixed-effects models for changes in hippocampus and precuneus longitudinal outcome measures with baseline CSF biomarkers modeled as predictors. Results partially supported our hypotheses: Both baseline p-tau(181p) and Aß42 were associated with hippocampal atrophy over time. Neither p-tau(181p) nor Aß42 were significantly related to cortical thinning in the precuneus over time. However, follow-up analyses demonstrated that having abnormal levels of both Aß42 and p-tau(181p) was associated with an accelerated rate of atrophy in both the hippocampus and precuneus. Results support early effects of Aß in the Alzheimer's disease process, which are less apparent than and perhaps dependent on p-tau effects as the disease progresses. However, amyloid deposition alone may be insufficient for emergence of significant morphometric changes and clinical symptoms.

Modeling Criminal Careers as Departures from a Unimodal Population Age-Crime Curve: The Case of Marijuana Use.

A major aim of longitudinal analyses of life course data is to describe the within- and between-individual variability in a behavioral outcome, such as crime. Statistical analyses of such data typically draw on mixture and mixed-effects growth models. In this work, we present a functional analytic point of view and develop an alternative method that models individual crime trajectories as departures from a population age-crime curve. Drawing on empirical and theoretical claims in criminology, we assume a unimodal population age-crime curve and allow individual expected crime trajectories to differ by their levels of offending and patterns of temporal misalignment. We extend Bayesian hierarchical curve registration methods to accommodate count data and to incorporate influence of baseline covariates on individual behavioral trajectories. Analyzing self-reported counts of yearly marijuana use from the Denver Youth Survey, we examine the influence of race and gender categories on differences in levels and timing of marijuana smoking. We find that our approach offers a flexible model for longitudinal crime trajectories and allows for a rich array of inferences of interest to criminologists and drug abuse researchers.

Reconceptualizing the classification of PNAS articles.

PNAS article classification is rooted in long-standing disciplinary divisions that do not necessarily reflect the structure of modern scientific research. We reevaluate that structure using latent pattern models from statistical machine learning, also known as mixed-membership models, that identify semantic structure in co-occurrence of words in the abstracts and references. Our findings suggest that the latent dimensionality of patterns underlying PNAS research articles in the Biological Sciences is only slightly larger than the number of categories currently in use, but it differs substantially in the content of the categories. Further, the number of articles that are listed under multiple categories is only a small fraction of what it should be. These findings together with the sensitivity analyses suggest ways to reconceptualize the organization of papers published in PNAS.

Use of spoken and written Japanese did not protect Japanese-American men from cognitive decline in late life.

OBJECTIVES: Spoken bilingualism may be associated with cognitive reserve. Mastering a complicated written language may be associated with additional reserve. We sought to determine if midlife use of spoken and written Japanese was associated with lower rates of late life cognitive decline. METHODS: Participants were second-generation Japanese-American men from the Hawaiian island of Oahu, born 1900-1919, free of dementia in 1991, and categorized based on midlife self-reported use of spoken and written Japanese (total n included in primary analysis = 2,520). Cognitive functioning was measured with the Cognitive Abilities Screening Instrument scored using item response theory. We used mixed effects models, controlling for age, income, education, smoking status, apolipoprotein E e4 alleles, and number of study visits. RESULTS: Rates of cognitive decline were not related to use of spoken or written Japanese. This finding was consistent across numerous sensitivity analyses. DISCUSSION: We did not find evidence to support the hypothesis that multilingualism is associated with cognitive reserve.