Protocol for the Evaluation of Data-Related Processes and Challenges at Level 1-4 Trauma Centers in Arkansas: A Mixed-Methods Case Study.

Background: In the following manuscript, we describe the detailed protocol for a mixed-methods, observational case study conducted to identify and evaluate existing data-related processes and challenges currently faced by trauma centers in a rural state. The data will be utilized to assess the impact of these challenges on registry data collection. Methods: The study relies on a series of interviews and observations to collect data from trauma registry staff at level 1-4 trauma centers across the state of Arkansas. A think-aloud protocol will be used to facilitate observations to gather keystroke-level modeling data and insight into site processes and workflows for collecting and submitting data to the Arkansas Trauma Registry. Informal, semi-structured interviews will follow the observation period to assess the participant's perspective on current processes, potential barriers to data collection or submission to the registry, and recommendations for improvement. Each session will be recorded, and de-identified transcripts and session notes will be used for analysis. Keystroke level modeling data derived from observations will be extracted and analyzed quantitatively to determine time spent performing end-to-end registry-related activities. Qualitative data from interviews will be reviewed and coded by 2 independent reviewers following a thematic analysis methodology. Each set of codes will then be adjudicated by the reviewers using a consensus-driven approach to extrapolate the final set of themes. Discussion: We will utilize a mixed methods approach to understand existing processes and barriers to data collection for the Arkansas Trauma Registry. Anticipated results will provide a baseline measure of the data collection and submission processes at various trauma centers across the state. We aim to assess strengths and limitations of existing processes and identify existing barriers to interoperability. These results will provide first-hand knowledge on existing practices for the trauma registry use case and will provide quantifiable data that can be utilized in future research to measure outcomes of future process improvement efforts. The potential implications of this study can form the basis for identifying potential solutions for streamlining data collection, exchange, and utilization of trauma registry data for clinical practice, public health, and clinical and translational research.

Measuring the Coverage of the HL7® FHIR® Standard in Supporting Data Acquisition for 3 Public Health Registries.

With the increasing need for timely submission of data to state and national public health registries, current manual approaches to data acquisition and submission are insufficient. In clinical practice, federal regulations are now mandating the use of data messaging standards, i.e., the Health Level Seven (HL7®) Fast Healthcare Interoperability Resources (FHIR®) standard, to facilitate the electronic exchange of clinical (patient) data. In both research and public health practice, we can also leverage FHIR® ‒ and the infrastructure already in place for supporting exchange of clinical practice data ‒ to enable seamless exchange between the electronic medical record and public health registries. That said, in order to understand the current utility of FHIR® for supporting the public health use case, we must first measure the extent to which the standard resources map to the required registry data elements. Thus, using a systematic mapping approach, we evaluated the level of completeness of the FHIR® standard to support data collection for three public health registries (Trauma, Stroke, and National Surgical Quality Improvement Program). On average, approximately 80% of data elements were available in FHIR® (71%, 77%, and 92%, respectively; inter-annotator agreement rates: 82%, 78%, and 72%, respectively). This tells us that there is the potential for significant automation to support EHR-to-Registry data exchange, which will reduce the amount of manual, error-prone processes and ensure higher data quality. Further, identification of the remaining 20% of data elements that are "not mapped" will enable us to improve the standard and develop profiles that will better fit the registry data model.

Protocol for the Evaluation of Data-Related Processes and Challenges at Level 1-4 Trauma Centers in Arkansas: A Mixed-Methods Case Study.

Background: In the following manuscript, we describe the detailed protocol for a mixed-methods, observational case study conducted to identify and evaluate existing data-related processes and challenges currently faced by trauma centers in a rural state. The data will be utilized to assess the impact of these challenges on registry data collection. Methods: The study relies on a series of interviews and observations to collect data from trauma registry staff at level 1-4 trauma centers across the state of Arkansas. A think-aloud protocol will be used to facilitate observations as a means to gather keystroke-level modeling data and insight into site processes and workflows for collecting and submitting data to the Arkansas Trauma Registry. Informal, semi-structured interviews will follow the observation period to assess the participant's perspective on current processes, potential barriers to data collection or submission to the registry, and recommendations for improvement. Each session will be recorded and de-identified transcripts and session notes will be used for analysis. Keystroke level modeling data derived from observations will be extracted and analyzed quantitatively to determine time spent performing end-to-end registry-related activities. Qualitative data from interviews will be reviewed and coded by 2 independent reviewers following a thematic analysis methodology. Each set of codes will then be adjudicated by the reviewers using a consensus-driven approach to extrapolate the final set of themes. Discussion: We will utilize a mixed methods approach to understand existing processes and barriers to data collection for the Arkansas Trauma Registry. Anticipated results will provide a baseline measure of the data collection and submission processes at various trauma centers across the state. We aim to assess strengths and limitations of existing processes and identify existing barriers to interoperability. These results will provide first-hand knowledge on existing practices for the trauma registry use case and will provide quantifiable data that can be utilized in future research to measure outcomes of future process improvement efforts. The potential implications of this study can form the basis for identifying potential solutions for streamlining data collection, exchange, and utilization of trauma registry data for clinical practice, public health, and clinical and translational research.

COVID-19 Vaccine Hesitancy Among Parents: A Qualitative Study.

OBJECTIVES: Addressing parental/caregivers' coronavirus disease 2019 (COVID-19) vaccine hesitancy is critical to improving vaccine uptake in children. Common concerns have been previously reported through online surveys, but qualitative data from KII and focus groups may add much-needed context. Our objective was to examine factors impacting pediatric COVID-19 vaccine decision-making in Black, Spanish-speaking, and rural white parents/caregivers to inform the content design of a mobile application to improve pediatric COVID-19 vaccine uptake. METHODS: Parents/caregivers of children aged 2 to 17 years from groups disproportionately affected by COVID-19-related vaccine hesitancy (rural-dwelling persons of any race/ethnicity, urban Black persons, and Spanish-speaking persons) were included on the basis of their self-reported vaccine hesitancy and stratified by race/ethnicity. Those expressing vaccine acceptance or refusal participated in KII, and those expressing hesitancy in focus groups. Deidentified transcripts underwent discourse analysis and thematic analysis, both individually and as a collection. Themes were revised until coders reached consensus. RESULTS: Overall, 36 participants completed the study: 4 vaccine acceptors and 4 refusers via KIIs, and the remaining 28 participated in focus groups. Participants from all focus groups expressed that they would listen to their doctor for information about COVID-19 vaccines. Infertility was a common concern, along with general concerns about vaccines. Vaccine decision-making was informed by the amount of information available to parents/caregivers, including scientific research; possible positive and negative long-term effects; and potential impacts of vaccination on preexisting medical conditions. CONCLUSIONS: Parents/caregivers report numerous addressable vaccine concerns. Our results will inform specific, targeted interventions for improving COVID-19 vaccine confidence.

Care delivery team composition effect on hospitalization risk in African Americans with congestive heart failure.

The care delivery team (CDT) is critical to providing care access and equity to patients who are disproportionately impacted by congestive heart failure (CHF). However, the specific clinical roles that are associated with care outcomes are unknown. The objective of this study was to examine the extent to which specific clinical roles within CDTs were associated with care outcomes in African Americans (AA) with CHF. Deidentified electronic medical record data were collected on 5,962 patients, representing 80,921 care encounters with 3,284 clinicians between January 1, 2014 and December 31, 2021. Binomial logistic regression assessed associations of specific clinical roles and the Mann Whitney-U assessed racial differences in outcomes. AAs accounted for only 26% of the study population but generated 48% of total care encounters, the same percentage of care encounters generated by the largest racial group (i.e., Caucasian Americans; 69% of the study population). AAs had a significantly higher number of hospitalizations and readmissions than Caucasian Americans. However, AAs had a significantly higher number of days at home and significantly lower care charges than Caucasian Americans. Among all CHF patients, patients with a Registered Nurse on their CDT were less likely to have a hospitalization (i.e. 30%) and a high number of readmissions (i.e., 31%) during the 7-year study period. When stratified by heart failure phenotype, the most severe patients who had a Registered Nurse on their CDT were 88% less likely to have a hospitalization and 50% less likely to have a high number of readmissions. Similar decreases in the likelihood of hospitalization and readmission were also found in less severe cases of heart failure. Specific clinical roles are associated with CHF care outcomes. Consideration must be given to developing and testing the efficacy of more specialized, empirical models of CDT composition to reduce the disproportionate impact of CHF.

Collaborative Program to Evaluate Real World Data for Use in Clinical Studies and Regulatory Decision Making.

The 21st Century Cures Act allows the US Food and Drug Administration to consider real world data (RWD) for new indications or post approval study requirements. However, there is limited guidance as to the relative quality of different RWD types. The ACE-RWD program will compare the quality of EHR clinical data, EHR billing data, and linked healthcare claims data to traditional clinical trial data collection methods. ACE-RWD is being conducted alongside 5-10 ancillary studies, with five sponsors, across multiple therapeutic areas. Each ancillary study will be conducted after or in parallel with its parent clinical study at a minimum of two clinical sites. Although not required, it is anticipated that EHR clinical and EHR billing data will be obtained via EHR-to-eCRF mechanisms that are based on the Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR®) standard.

Social Determinants of Health Data Quality at Different Levels of Geographic Detail.

Social determinants of health (SDOH) impact 80% of health outcomes from acute to chronic disorders, and attempts are underway to provide these data elements to clinicians. It is, however, difficult to collect SDOH data through (1) surveys, which provide inconsistent and incomplete data, or (2) aggregates at the neighborhood level. Data from these sources is not sufficiently accurate, complete, and up-to-date. To demonstrate this, we have compared the Area Deprivation Index (ADI) to purchased commercial consumer data at the individual-household level. The ADI is composed of income, education, employment, and housing quality information. Although this index does a good job of representing populations, it is not adequate to describe individuals, especially in a healthcare context. Aggregate measures are, by definition, not sufficiently granular to describe each individual within the population they represent and may result in biased or imprecise data when simply assigned to the individual. Moreover, this problem is generalizable to any community-level element, not just ADI, in so far as they are an aggregate of the individual community members.

Comparing Medical Record Abstraction (MRA) Error Rates in an Observational Study to Pooled Rates Identified in the Data Quality Literature.

Background: Medical record abstraction (MRA) is a commonly used method for data collection in clinical research, but is prone to error, and the influence of quality control (QC) measures is seldom and inconsistently assessed during the course of a study. We employed a novel, standardized MRA-QC framework as part of an ongoing observational study in an effort to control MRA error rates. In order to assess the effectiveness of our framework, we compared our error rates against traditional MRA studies that had not reported using formalized MRA-QC methods. Thus, the objective of this study was to compare the MRA error rates derived from the literature with the error rates found in a study using MRA as the sole method of data collection that employed an MRA-QC framework. Methods: Using a moderator meta-analysis employed with Q-test, the MRA error rates from the meta-analysis of the literature were compared with the error rate from a recent study that implemented formalized MRA training and continuous QC processes. Results: The MRA process for data acquisition in clinical research was associated with both high and highly variable error rates (70 - 2,784 errors per 10,000 fields). Error rates for the study using our MRA-QC framework were between 1.04% (optimistic, all-field rate) and 2.57% (conservative, populated-field rate) (or 104 - 257 errors per 10,000 fields), 4.00 - 5.53 percentage points less than the observed rate from the literature (p<0.0001). Conclusions: Review of the literature indicated that the accuracy associated with MRA varied widely across studies. However, our results demonstrate that, with appropriate training and continuous QC, MRA error rates can be significantly controlled during the course of a clinical research study.

Assessing clinical site readiness for electronic health record (EHR)-to-electronic data capture (EDC) automated data collection.

BACKGROUND: eSource software is used to automatically copy a patient's electronic health record data into a clinical study's electronic case report form. However, there is little evidence to assist sponsors in identifying the best sites for multi-center eSource studies. METHODS: We developed an eSource site readiness survey. The survey was administered to principal investigators, clinical research coordinators, and chief research information officers at Pediatric Trial Network sites. RESULTS: A total of 61 respondents were included in this study (clinical research coordinator, 22; principal investigator, 20; and chief research information officer, 19). Clinical research coordinators and principal investigators ranked medication administration, medication orders, laboratory, medical history, and vital signs data as having the highest priority for automation. While most organizations used some electronic health record research functions (clinical research coordinator, 77%; principal investigator, 75%; and chief research information officer, 89%), only 21% of sites were using Fast Healthcare Interoperability Resources standards to exchange patient data with other institutions. Respondents generally gave lower readiness for change ratings to organizations that did not have a separate research information technology group and where researchers practiced in hospitals not operated by their medical schools. CONCLUSIONS: Site readiness to participate in eSource studies is not merely a technical problem. While technical capabilities are important, organizational priorities, structure, and the site's support of clinical research functions are equally important considerations.

Informatics infrastructure in a rural pediatric clinical trials network: Matching specific clinical research needs with best practices and industry guidelines.

Children have historically been underrepresented in randomized controlled trials and multi-center studies. This is particularly true for children who reside in rural and underserved areas. Conducting multi-center trials in rural areas presents unique informatics challenges. These challenges call for increased attention towards informatics infrastructure and the need for development and application of sound informatics approaches to the collection, processing, and management of data for clinical studies. By modifying existing local infrastructure and utilizing open source tools, we have been able to successfully deploy a multi-site data coordinating and operations center. We report our implementation decisions for data collection and management for the IDeA States Pediatric Clinical Trial Network (ISPCTN) based on the functionality needed for the ISPCTN, our synthesis of the extant literature in data collection and management methodology, and Good Clinical Data Management Practices.

Error Rates of Data Processing Methods in Clinical Research: A Systematic Review and Meta-Analysis of Manuscripts Identified Through PubMed.

Background: In clinical research, prevention of systematic and random errors of data collected is paramount to ensuring reproducibility of trial results and the safety and efficacy of the resulting interventions. Over the last 40 years, empirical assessments of data accuracy in clinical research have been reported in the literature. Although there have been reports of data error and discrepancy rates in clinical studies, there has been little systematic synthesis of these results. Further, although notable exceptions exist, little evidence exists regarding the relative accuracy of different data processing methods. We aim to address this gap by evaluating error rates for 4 data processing methods. Methods: A systematic review of the literature identified through PubMed was performed to identify studies that evaluated the quality of data obtained through data processing methods typically used in clinical trials: medical record abstraction (MRA), optical scanning, single-data entry, and double-data entry. Quantitative information on data accuracy was abstracted from the manuscripts and pooled. Meta-analysis of single proportions based on the Freeman-Tukey transformation method and the generalized linear mixed model approach were used to derive an overall estimate of error rates across data processing methods used in each study for comparison. Results: A total of 93 papers (published from 1978 to 2008) meeting our inclusion criteria were categorized according to their data processing methods. The accuracy associated with data processing methods varied widely, with error rates ranging from 2 errors per 10,000 fields to 2,784 errors per 10,000 fields. MRA was associated with both high and highly variable error rates, having a pooled error rate of 6.57% (95% CI: 5.51, 7.72). In comparison, the pooled error rates for optical scanning, single-data entry, and double-data entry methods were 0.74% (0.21, 1.60), 0.29% (0.24, 0.35) and 0.14% (0.08, 0.20), respectively. Conclusions: Data processing and cleaning methods may explain a significant amount of the variability in data accuracy. MRA error rates, for example, were high enough to impact decisions made using the data and could necessitate increases in sample sizes to preserve statistical power. Thus, the choice of data processing methods can likely impact process capability and, ultimately, the validity of trial results.

Measuring and controlling medical record abstraction (MRA) error rates in an observational study.

BACKGROUND: Studies have shown that data collection by medical record abstraction (MRA) is a significant source of error in clinical research studies relying on secondary use data. Yet, the quality of data collected using MRA is seldom assessed. We employed a novel, theory-based framework for data quality assurance and quality control of MRA. The objective of this work is to determine the potential impact of formalized MRA training and continuous quality control (QC) processes on data quality over time. METHODS: We conducted a retrospective analysis of QC data collected during a cross-sectional medical record review of mother-infant dyads with Neonatal Opioid Withdrawal Syndrome. A confidence interval approach was used to calculate crude (Wald's method) and adjusted (generalized estimating equation) error rates over time. We calculated error rates using the number of errors divided by total fields ("all-field" error rate) and populated fields ("populated-field" error rate) as the denominators, to provide both an optimistic and a conservative measurement, respectively. RESULTS: On average, the ACT NOW CE Study maintained an error rate between 1% (optimistic) and 3% (conservative). Additionally, we observed a decrease of 0.51 percentage points with each additional QC Event conducted. CONCLUSIONS: Formalized MRA training and continuous QC resulted in lower error rates than have been found in previous literature and a decrease in error rates over time. This study newly demonstrates the importance of continuous process controls for MRA within the context of a multi-site clinical research study.

Evaluating Site-Level Implementations of the HL7 FHIR Standard to Support eSource Data Exchange in Clinical Research.

Direct extraction and use of electronic health record (EHR) data is a long-term and multifaceted endeavor that includes design, development, implementation and evaluation of methods and tools for semi-automating tasks in the research data collection process, including, but not limited to, medical record abstraction (MRA). A systematic mapping of study data elements was used to measure the coverage of the Health Level Seven (HL7®) Fast Healthcare Interoperability Resources (FHIR®) standard for a federally sponsored, pragmatic cardiovascular randomized controlled trial (RCT) targeting adults. We evaluated site-level implementations of the HL7® FHIR® standard to investigate study- and site-level differences that could affect coverage and offer insight into the feasibility of a FHIR-based eSource solution for multicenter clinical research.

Near Real Time EHR Data Utilization in a Clinical Study.

Extraction and use of Electronic Health Record (EHR) data is common in retrospective observational studies. However, electronic extraction and use of EHR data is rare during longitudinal prospective studies. One of the reasons is the amount of processing needed to assess data quality and assure consistency in meaning and format across multiple investigational sites. We report a case study of and lessons learned from acquisition and processing of EHR data in an ongoing basis during a clinical study.

eSource-Enabled vs. Traditional Clinical Trial Data Collection Methods: A Site-Level Economic Analysis.

Directly extracting data from site electronic health records for updating clinical trial databases (eSource) can reduce site data collection times and errors. We conducted a study to determine clinical trial characteristics that make eSource vs. traditional data collection methods more and less economically attractive. The number of patients a site enrolls, the number of study data elements, study coordinator data collection times, and the percent of study data elements that can be extracted via eSource software all impact eSource economic attractiveness. However, these factors may not impact all clinical trial designs in the same way.

Evaluating the Coverage of the HL7 ® FHIR ® Standard to Support eSource Data Exchange Implementations for use in Multi-Site Clinical Research Studies.

The direct use of EHR data in research, often referred to as 'eSource', has long-been a goal for researchers because of anticipated increases in data quality and reductions in site burden. eSource solutions should rely on data exchange standards for consistency, quality, and efficiency. The utility of any data standard can be evaluated by its ability to meet specific use case requirements. The Health Level Seven (HL7 ® ) Fast Healthcare Interoperability Resources (FHIR ® ) standard is widely recognized for clinical data exchange; however, a thorough analysis of the standard's data coverage in supporting multi-site clinical studies has not been conducted. We developed and implemented a systematic mapping approach for evaluating HL7 ® FHIR ® standard coverage in multi-center clinical trials. Study data elements from three diverse studies were mapped to HL7 ® FHIR ® resources, offering insight into the coverage and utility of the standard for supporting the data collection needs of multi-site clinical research studies.

Adapting Scoring Based Classification to Simplify and Automate Phenotype Creation for Cohort Identification in Clinical Data.

EHR-based phenotype development and validation are extremely time-consuming and have considerable monetary cost. The creation of a phenotype currently requires clinical experts and experts in the data to be queried. The new approach presented here demonstrates a computational alternative to the classification of patient cohorts based on automatic weighting of ICD codes. This approach was applied to data from six different clinics within the University of Arkansas for Medical Science (UAMS) health system. The results were compared with phenotype algorithms designed by clinicians and informaticians for asthma and melanoma. Relative to traditional phenotype development, this method shows potential to considerably reduce time requirements and monetary costs with comparable results.