Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions?

The replication crisis poses important challenges to modern science. Central to this challenge is re-establishing ground truths or the most fundamental theories that serve as the bedrock to a scientific community. However, the goal to identify hypotheses with the greatest support is non-trivial given the unprecedented rate of scientific publishing. In this era of high-volume science, the goal of this study is to sample from one research community within clinical neuroscience (traumatic brain injury) and track major trends that have shaped this literature over the past 50 years. To do so, we first conduct a decade-wise (1980-2019) network analysis to examine the scientific communities that shape this literature. To establish the robustness of our findings, we utilized searches from separate search engines (Web of Science; Semantic Scholar). As a second goal, we sought to determine the most highly cited hypotheses influencing the literature in each decade. In a third goal, we then searched for any papers referring to 'replication' or efforts to reproduce findings within our >50 000 paper dataset. From this search, 550 papers were analysed to determine the frequency and nature of formal replication studies over time. Finally, to maximize transparency, we provide a detailed procedure for the creation and analysis of our dataset, including a discussion of each of our major decision points, to facilitate similar efforts in other areas of neuroscience. We found that the unparalleled rate of scientific publishing within the brain injury literature combined with the scarcity of clear hypotheses in individual publications is a challenge to both evaluating accepted findings and determining paths forward to accelerate science. Additionally, while the conversation about reproducibility has increased over the past decade, the rate of published replication studies continues to be a negligible proportion of the research. Meta-science and computational methods offer the critical opportunity to assess the state of the science and illuminate pathways forward, but ultimately there is structural change needed in the brain injury literature and perhaps others.

How failure to falsify in high-volume science contributes to the replication crisis.

The number of scientific papers published every year continues to increase, but scientific knowledge is not progressing at the same rate. Here we argue that a greater emphasis on falsification - the direct testing of strong hypotheses - would lead to faster progress by allowing well-specified hypotheses to be eliminated. We describe an example from neuroscience where there has been little work to directly test two prominent but incompatible hypotheses related to traumatic brain injury. Based on this example, we discuss how building strong hypotheses and then setting out to falsify them can bring greater precision to the clinical neurosciences, and argue that this approach could be beneficial to all areas of science.

Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics.

In this critical review, we examine the application of predictive models, for example, classifiers, trained using machine learning (ML) to assist in interpretation of functional neuroimaging data. Our primary goal is to summarize how ML is being applied and critically assess common practices. Our review covers 250 studies published using ML and resting-state functional MRI (fMRI) to infer various dimensions of the human functional connectome. Results for holdout ("lockbox") performance was, on average, ∼13% less accurate than performance measured through cross-validation alone, highlighting the importance of lockbox data, which was included in only 16% of the studies. There was also a concerning lack of transparency across the key steps in training and evaluating predictive models. The summary of this literature underscores the importance of the use of a lockbox and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that, ideally, studies are motivated both by the reproducibility and generalizability of findings as well as the potential clinical significance of the insights. We offer recommendations for principled integration of machine learning into the clinical neurosciences with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks, and parsing heterogeneous patient outcomes.

Diminished neural network dynamics after moderate and severe traumatic brain injury.

Over the past decade there has been increasing enthusiasm in the cognitive neurosciences around using network science to understand the system-level changes associated with brain disorders. A growing literature has used whole-brain fMRI analysis to examine changes in the brain's subnetworks following traumatic brain injury (TBI). Much of network modeling in this literature has focused on static network mapping, which provides a window into gross inter-nodal relationships, but is insensitive to more subtle fluctuations in network dynamics, which may be an important predictor of neural network plasticity. In this study, we examine the dynamic connectivity with focus on state-level connectivity (state) and evaluate the reliability of dynamic network states over the course of two runs of intermittent task and resting data. The goal was to examine the dynamic properties of neural networks engaged periodically with task stimulation in order to determine: 1) the reliability of inter-nodal and network-level characteristics over time and 2) the transitions between distinct network states after traumatic brain injury. To do so, we enrolled 23 individuals with moderate and severe TBI at least 1-year post injury and 19 age- and education-matched healthy adults using functional MRI methods, dynamic connectivity modeling, and graph theory. The results reveal several distinct network "states" that were reliably evident when comparing runs; the overall frequency of dynamic network states are highly reproducible (r-values>0.8) for both samples. Analysis of movement between states resulted in fewer state transitions in the TBI sample and, in a few cases, brain injury resulted in the appearance of states not exhibited by the healthy control (HC) sample. Overall, the findings presented here demonstrate the reliability of observable dynamic mental states during periods of on-task performance and support emerging evidence that brain injury may result in diminished network dynamics.

A voxelwise approach to determine consensus regions-of-interest for the study of brain network plasticity.

Despite exciting advances in the functional imaging of the brain, it remains a challenge to define regions of interest (ROIs) that do not require investigator supervision and permit examination of change in networks over time (or plasticity). Plasticity is most readily examined by maintaining ROIs constant via seed-based and anatomical-atlas based techniques, but these approaches are not data-driven, requiring definition based on prior experience (e.g., choice of seed-region, anatomical landmarks). These approaches are limiting especially when functional connectivity may evolve over time in areas that are finer than known anatomical landmarks or in areas outside predetermined seeded regions. An ideal method would permit investigators to study network plasticity due to learning, maturation effects, or clinical recovery via multiple time point data that can be compared to one another in the same ROI while also preserving the voxel-level data in those ROIs at each time point. Data-driven approaches (e.g., whole-brain voxelwise approaches) ameliorate concerns regarding investigator bias, but the fundamental problem of comparing the results between distinct data sets remains. In this paper we propose an approach, aggregate-initialized label propagation (AILP), which allows for data at separate time points to be compared for examining developmental processes resulting in network change (plasticity). To do so, we use a whole-brain modularity approach to parcellate the brain into anatomically constrained functional modules at separate time points and then apply the AILP algorithm to form a consensus set of ROIs for examining change over time. To demonstrate its utility, we make use of a known dataset of individuals with traumatic brain injury sampled at two time points during the first year of recovery and show how the AILP procedure can be applied to select regions of interest to be used in a graph theoretical analysis of plasticity.

Hyperconnectivity is a fundamental response to neurological disruption.

OBJECTIVE: In the cognitive and clinical neurosciences, the past decade has been marked by dramatic growth in a literature examining brain "connectivity" using noninvasive methods. We offer a critical review of the blood oxygen level dependent functional MRI (BOLD fMRI) literature examining neural connectivity changes in neurological disorders with focus on brain injury and dementia. The goal is to demonstrate that there are identifiable shifts in local and large-scale network connectivity that can be predicted by the degree of pathology. We anticipate that the most common network response to neurological insult is hyperconnectivity but that this response depends upon demand and resource availability. METHOD: To examine this hypothesis, we initially reviewed the results from 1,426 studies examining functional brain connectivity in individuals diagnosed with multiple sclerosis, traumatic brain injury, mild cognitive impairment, and Alzheimer's disease. Based upon inclusionary criteria, 126 studies were included for detailed analysis. RESULTS: RESULTS from 126 studies examining local and whole brain connectivity demonstrated increased connectivity in traumatic brain injury and multiple sclerosis. This finding is juxtaposed with findings in mild cognitive impairment and Alzheimer's disease where there is a shift to diminished connectivity as degeneration progresses. CONCLUSION: This summary of the functional imaging literature using fMRI methods reveals that hyperconnectivity is a common response to neurological disruption and that it may be differentially observable across brain regions. We discuss the factors contributing to both hyper- and hypoconnectivity results after neurological disruption and the implications these findings have for network plasticity.

The rich get richer: brain injury elicits hyperconnectivity in core subnetworks.

There remains much unknown about how large-scale neural networks accommodate neurological disruption, such as moderate and severe traumatic brain injury (TBI). A primary goal in this study was to examine the alterations in network topology occurring during the first year of recovery following TBI. To do so we examined 21 individuals with moderate and severe TBI at 3 and 6 months after resolution of posttraumatic amnesia and 15 age- and education-matched healthy adults using functional MRI and graph theoretical analyses. There were two central hypotheses in this study: 1) physical disruption results in increased functional connectivity, or hyperconnectivity, and 2) hyperconnectivity occurs in regions typically observed to be the most highly connected cortical hubs, or the "rich club". The current findings generally support the hyperconnectivity hypothesis showing that during the first year of recovery after TBI, neural networks show increased connectivity, and this change is disproportionately represented in brain regions belonging to the brain's core subnetworks. The selective increases in connectivity observed here are consistent with the preferential attachment model underlying scale-free network development. This study is the largest of its kind and provides the unique opportunity to examine how neural systems adapt to significant neurological disruption during the first year after injury.