Identifying assessment criteria for in vitro studies: a method and item bank.

To support the development of appraisal tools for assessing the quality of in vitro studies, we developed a method for literature-based discovery of study assessment criteria, used the method to create an item bank of assessment criteria of potential relevance to in vitro studies, and analyzed the item bank to discern and critique current approaches for appraisal of in vitro studies. We searched four research indexes and included any document that identified itself as an appraisal tool for in vitro studies, was a systematic review that included a critical appraisal step, or was a reporting checklist for in vitro studies. We abstracted, normalized, and categorized all criteria applied by the included appraisal tools to create an "item bank" database of issues relevant to the assessment of in vitro studies. The resulting item bank consists of 676 unique appraisal concepts from 67 appraisal tools. We believe this item bank is the single most comprehensive resource of its type to date, should be of high utility for future tool development exercises, and provides a robust methodology for grounding tool development in the existing literature. While we set out to develop an item bank specifically targeting in vitro studies, we found that many of the assessment concepts we discovered are readily applicable to other study designs. Item banks can be of significant value as a resource; however, there are important challenges in developing, maintaining, and extending them of which researchers should be aware.

Bias in published randomized trials that compare collagenase injection with percutaneous needle fasciotomy in the treatment of Dupuytren disease: a systematic review.

Purpose: Controversy exists regarding the comparative efficacy of collagenase injection and percutaneous needle fasciotomy in the treatment of Dupuytren contracture. The randomized controlled trials (RCTs) that have compared the two treatment methods have reported results mostly implying similar treatment efficacy, durability, and complications. We aimed to review these RCTs regarding methodical quality and risk of bias. Methods: We searched PubMed and Cochrane Library databases up to May 2023. All RCTs comparing collagenase injection with needle fasciotomy were included. Eligible articles were reviewed by two researchers, of whom one was blinded to each article's title, authors, year of publication, journal, and source of the studies. To assess methodical quality, we used the modified Jadad scale yielding a score of 0 (lowest quality) to 5 (highest quality). We assessed risk of bias with the Cochrane risk-of-bias tool (RoB 2). Results: Five studies were eligible, comprising 204 patients treated with collagenase injection and 209 patients treated with needle fasciotomy. The modified Jadad score ranged from 1 to 2 points in the five studies, and the overall risk of bias was high in all studies. Pretrial protocols could be retrieved for only two studies, revealing important discrepancies with the published articles. Conclusion: The published RCTs that have compared collagenase injection with needle fasciotomy in the treatment of Dupuytren contracture demonstrate a high risk of bias.

Scientific Committee guidance on appraising and integrating evidence from epidemiological studies for use in EFSA's scientific assessments.

EFSA requested its Scientific Committee to prepare a guidance document on appraising and integrating evidence from epidemiological studies for use in EFSA's scientific assessments. The guidance document provides an introduction to epidemiological studies and illustrates the typical biases, which may be present in different epidemiological study designs. It then describes key epidemiological concepts relevant for evidence appraisal. This includes brief explanations for measures of association, exposure assessment, statistical inference, systematic error and effect modification. The guidance then describes the concept of external validity and the principles of appraising epidemiological studies. The customisation of the study appraisal process is explained including tailoring of tools for assessing the risk of bias (RoB). Several examples of appraising experimental and observational studies using a RoB tool are annexed to the document to illustrate the application of the approach. The latter part of this guidance focuses on different steps of evidence integration, first within and then across different streams of evidence. With respect to risk characterisation, the guidance considers how evidence from human epidemiological studies can be used in dose-response modelling with several different options being presented. Finally, the guidance addresses the application of uncertainty factors in risk characterisation when using evidence from human epidemiological studies.

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews.

BACKGROUND: The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension. METHODS: A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed. RESULTS: Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT. CONCLUSION: For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

DE-PASS best evidence statement (BESt): determinants of adolescents' device-based physical activity and sedentary behaviour in settings: a systematic review and meta-analysis.

BACKGROUND: Although physical activity (PA) is associated with significant health benefits, only a small percentage of adolescents meet recommended PA levels. This systematic review with meta-analysis explored the modifiable determinants of adolescents' device-based PA and/or sedentary behaviour (SB), evaluated in previous interventions and examined the associations between PA/SB and these determinants in settings. METHODS: A search was conducted on five electronic databases, including papers published from January 2010 to July 2023. Randomized Controlled Trials (RCTs) or Controlled Trials (CTs) measuring adolescents' device-based PA/SB and their modifiable determinants at least at two time points: pre- and post-intervention were considered eligible. PA/SB and determinants were the main outcomes. Modifiable determinants were classified after data extraction adopting the social-ecological perspective. Robust Bayesian meta-analyses (RoBMA) were performed per each study setting. Outcomes identified in only one study were presented narratively. The risk of bias for each study and the certainty of the evidence for each meta-analysis were evaluated. The publication bias was also checked. PROSPERO ID: CRD42021282874. RESULTS: Fourteen RCTs (eight in school, three in school and family, and one in the family setting) and one CT (in the school setting) were included. Fifty-four modifiable determinants were identified and were combined into 33 broader determinants (21 individual-psychological, four individual-behavioural, seven interpersonal, and one institutional). RoBMAs revealed none or negligible pooled intervention effects on PA/SB or determinants in all settings. The certainty of the evidence of the impact of interventions on outcomes ranged from very low to low. Narratively, intervention effects in favour of the experimental group were detected in school setting for the determinants: knowledge of the environment for practicing PA, d = 1.84, 95%CI (1.48, 2.20), behaviour change techniques, d = 0.90, 95%CI (0.09, 1.70), choice provided, d = 0.70, 95%CI (0.36, 1.03), but no corresponding effects on PA or SB were found. CONCLUSIONS: Weak to minimal evidence regarding the associations between the identified modifiable determinants and adolescents' device-based PA/SB in settings were found, probably due to intervention ineffectiveness. Well-designed and well-implemented multicomponent interventions should further explore the variety of modifiable determinants of adolescents' PA/SB, including policy and environmental variables.

The Cochrane risk of bias assessment tool 2 (RoB 2) versus the original RoB: A perspective on the pros and cons.

Background and Aims: Critical appraisal or risk of bias assessment is a fundamental part of systematic reviews that clarifies the degree to which included research articles are qualified and reliable. Version 2 of the Cochrane tool for assessing the risk of bias in randomized trials (RoB 2), the updated version of the first tool, was released in 2019. Here, we have compared these two versions of Cochrane risk of bias assessment tools and highlighted the pros and cons of RoB 2. Methods: Statistical analysis and methodology is not applicable to this article as no new data were created or analyzed in this study. Results: The overall approach in RoB 2 is that by answering some signaling questions after the specification of results, effects of interest, and sources of information, an overall judgment for the quality of each study is reached. Accordingly, in the original version of the Cochrane RoB tool, the judgment can be in three different conclusions, including low, unclear, and high risk of bias. The most prominent difference in bias domains is the removal of "other bias" domain being replaced by "overall bias" judgment. Also, the most common presentation types of Cochrane risk of bias assessments are the "summary" and "graph" which are generated by Review Manager, web-based applications, or packages in R software. Conclusion: The RoB 2 tool, compared to the original RoB, has improved and is the recommended version by the Cochrane Collaboration for quality assessment of randomized controlled trials. It is recommended to consider funding source, duration of follow-up, declaration of data availability, the status of baseline characteristics between groups, and sample size calculation methods in further revisions of the Cochrane risk of bias assessment tools.

The suboptimal clinical applicability of prognostic prediction models for severe postpartum hemorrhage: a meta-epidemiological study.

OBJECTIVE: To systematically investigate clinical applicability of the current prognostic prediction models for severe postpartum hemorrhage (SPPH). STUDY DESIGN AND SETTING: A meta-epidemiological study of prognostic prediction models was conducted for SPPH. A pre-designed structured questionnaire was adopted to extract the study characteristics, predictors and the outcome, modelling methods, predictive performance, classification ability for high-risk individuals, and clinical use scenarios. The risk of bias among studies were assessed by the Prediction model Risk Of Bias ASsessment Tool. RESULTS: Twenty-two studies containing 27 prediction models were included. The number of predictors in the final models varied from 3 to 53. However, one-third of the models (11) did not clearly specify the timing of predictor measurement. Calibration was found to be lacking in 10 (37.0%) models. Among 20 models had an incidence rate of predicted outcome below 15.0%, none of the models estimated the area under the precision-recall curve, and all reported positive predictive values were below 40.0%. Only two (7.4%) models specified the target clinical setting, while seven (25.9%) models clarified the intended timing of model use. Lastly, all 22 studies were deemed to be at high risk of bias. CONCLUSION: Current SPPH prediction models have limited clinical applicability due to methodological flaws, including unclear predictor measurement, inadequate calibration assessment, and insufficient evaluation of classification ability. Additionally, there is a lack of clarity regarding the timing for model use, target users, and clinical settings. These limitations raise concerns about the reliability and usefulness of these models in real-world clinical practice.

A scoping review shows that no single existing risk of bias assessment tool considers all sources of bias for cross-sectional studies.

OBJECTIVES: Different tools to assess the potential risk of bias (RoB) for cross-sectional studies have been developed, but it is unclear whether all pertinent bias concepts are addressed. We aimed to identify RoB concepts applicable to cross-sectional research validity and to explore coverage for each in existing appraisal tools. STUDY DESIGN AND SETTING: This scoping review followed the Joanna Briggs Institute methodology. We included records of any study design describing or reporting methods, concepts or tools used to consider RoB in health research reported to be descriptive/prevalence survey or analytic/association (cross-sectional) study designs. Synthesis included quantitative and qualitative analysis. RESULTS: Of the 4556 records screened, 90 were selected for inclusion; 67 (74%) described the development of, or validation process for, appraisal tools, 15 (17%) described methodological content or theory relevant to RoB for cross-sectional studies and 8 (9%) records of methodological systematic reviews. Review of methodological reports identified important RoB concepts for both descriptive/prevalence and analytic/association studies. Tools identified (n = 64 unique tools) were either intended to appraise quality or assess RoB in multiple study designs including cross-sectional studies (n = 21; 33%) or cross-sectional designs alone (n = 43; 67%). Several existing tools were modified (n = 17; 27%) for application to cross-sectional studies. The RoB items most frequently addressed in the RoB tools were validity and reliability of the exposure (53%) or outcome (65%) measurement and representativeness of the study population (59%). Most tools did not consider nonresponse or missingness appropriately or at all. CONCLUSION: Assessing cross-sectional studies involve unique RoB considerations. We identified RoB tools designed for broad applicability across various study designs as well as those specifically tailored for cross-sectional studies. However, none of the identified tools comprehensively address all potential biases pertinent to cross-sectional studies. Our findings indicate a need for continued improvement of RoB tools and suggest that the development of context-specific or more precise tools for this study design may be necessary.

Trial characteristics and treatment effect estimates in randomized controlled trials of Chinese herbal medicine: A meta-epidemiological study.

BACKGROUND: Previously published meta-epidemiological studies focused on Western medicine have identified some trial characteristics that impact the treatment effect of randomized controlled trials (RCTs). Nevertheless, it remains unclear if similar associations exist in RCTs on Chinese herbal medicine (CHM). Further, Chinese medicine-related characteristics have not been explored yet. OBJECTIVE: To investigate trial characteristics related to treatment effect estimates on CHM RCTs. SEARCH STRATEGY: This meta-epidemiological study searched 5 databases for systematic reviews on CHM treatment published between January 2011 and July 2021. INCLUSION CRITERIA: An eligible systematic review should only include RCTs of CHM and conduct at least one meta-analysis. DATA EXTRACTION AND ANALYSIS: Two reviewers independently conducted data extraction on general characteristics of systematic reviews, meta-analyses and included RCTs. They also assessed the risk of bias of RCTs using the Cochrane risk of bias tool. A two-step approach was used for data analyses. The ratio of odds ratios (ROR) and difference in standardized mean differences (dSMD) with 95% confidence interval (CI) were applied to present the difference in effect estimates for binary and continuous outcomes, respectively. RESULTS: Ninety-one systematic reviews, comprising 1338 RCTs were identified. For binary outcomes, RCTs incorporated with syndrome differentiation (ROR: 1.23; 95 % CI: [1.07, 1.39]), adopting Chinese medicine formula (ROR: 1.19; 95% CI: [1.03, 1.34]), with low risk of bias on incomplete outcome data (ROR: 1.29; 95% CI: [1.06, 1.52]) and selective outcome reporting (ROR: 1.12; 95% CI: [1.01, 1.24]), as well as a trial size ≥ 100 (ROR: 1.23; 95% CI: [1.04, 1.42]) preferred to show larger effect estimates. As for continuous outcomes, RCTs with Chinese medicine diagnostic criteria (dSMD: 0.23; 95% CI: [0.06, 0.41]), judged as high/unclear risk of bias on allocation concealment (dSMD: -0.70; 95% CI: [-0.99, -0.42]), with low risk of bias on incomplete outcome data (dSMD: 0.30; 95% CI: [0.18, 0.43]), conducted at a single center (dSMD: -0.33; 95% CI: [-0.61, -0.05]), not using intention-to-treat analysis (dSMD: -0.75; 95% CI: [-1.43, -0.07]), and without funding support (dSMD: -0.22; 95% CI: [-0.41, -0.02]) tended to show larger effect estimates. CONCLUSION: This study provides empirical evidence for the development of a specific critical appraisal tool for risk of bias assessments on CHM RCTs. Please cite this article as: Wang BH, Lin YL, Gao YY, Song JL, Qin L, Li LQ, Liu WQ, Zhong CCW, Jiang MY, Mao C, Yang XB, Chung VCH, Wu IXY. Trial characteristics and treatment effect estimates in randomized controlled trials of Chinese herbal medicine: A meta-epidemiological study. J Integr Med. 2024; 22(3): 223-234.

Predictive models for lymph node metastasis in endometrial cancer: A systematic review and bibliometric analysis.

BACKGROUND: Lymph node metastasis is associated with a poorer prognosis in endometrial cancer. OBJECTIVE: The objective was to synthesize and critically appraise existing predictive models for lymph node metastasis risk stratification in endometrial cancer. DESIGN: This study is a systematic review. DATA SOURCES AND METHODS: We searched the Web of Science for articles reporting models predicting lymph node metastasis in endometrial cancer, with a systematic review and bibliometric analysis conducted based upon which. Risk of bias was assessed by the Prediction model Risk Of BiAS assessment Tool (PROBAST). RESULTS: A total of 64 articles were included in the systematic review, published between 2010 and 2023. The most common articles were "development only." Traditional clinicopathological parameters remained the mainstream in models, for example, serum tumor marker, myometrial invasion and tumor grade. Also, models based upon gene-signatures, radiomics and digital histopathological images exhibited an acceptable self-reported performance. The most frequently validated models were the Mayo criteria, which reached a negative predictive value of 97.1%-98.2%. Substantial variability and inconsistency were observed through PROBAST, indicating significant between-study heterogeneity. A further bibliometric analysis revealed a relatively weak link between authors and organizations on models predicting lymph node metastasis in endometrial cancer. CONCLUSION: A number of predictive models for lymph node metastasis in endometrial cancer have been developed. Although some exhibited promising performance as they demonstrated adequate to good discrimination, few models can currently be recommended for clinical practice due to lack of independent validation, high risk of bias and low consistency in measured predictors. Collaborations between authors, organizations and countries were weak. Model updating, external validation and collaborative research are urgently needed. REGISTRATION: None.

Introduction to predictive models for lymph node metastasis in endometrial cancerLymph node metastasis of endometrial cancer is associated with a poor prognosis. There are currently many predictive models. We summarized and evaluated them in this article.

Diagnostic accuracy of cognitive screening tools validated for older adults in Iran: a systematic review and meta-analysis.

BACKGROUND: This systematic review aims to comprehensively assess the diagnostic accuracy of cognitive screening tools validated for older adults in Iran, providing evidence-based recommendations for clinicians and researchers. METHODS: A comprehensive search in March 2023 across Web of Science, PubMed, Scopus, ScienceDirect, SID, IranMedex, and IranDoc, enhanced by hand-searching references and Google Scholar, identified cross-sectional studies on cognitive screening in Iranian seniors. We assessed diagnostic accuracy, cognitive domains, and test strengths and weaknesses. A bivariate random-effects meta-analysis provided summary estimates and 95% confidence intervals, illustrated in forest plots. RESULTS: Our review, derived from an initial screening of 38 articles, focused on 17 studies involving 14 cognitive screening tools and participant counts from 60 to 350, mostly from specialized clinics. The MMSE was the only tool examined in at least three studies, prompting a meta-analysis revealing its sensitivity at 0.89 and specificity at 0.77 for dementia detection, albeit amidst significant heterogeneity (I^2 > 80%). ACE-III demonstrated the highest diagnostic accuracy for MCI and dementia, while MoCA's performance was deemed adequate for MCI and excellent for dementia. High bias risk in studies limits interpretation. CONCLUSION: This review identifies key cognitive tools for dementia and MCI in Iranian older adults, tailored to educational levels for use in primary and specialized care. It emphasizes the need for further validation to enhance diagnostic precision across diverse settings, within a concise framework prioritizing brevity and accuracy for clinical applicability.

Quantity and quality are increasing but there's room for improvement: A scoping review of physical activity intervention trials.

BACKGROUND: Observing trends in research publications helps to identify the quantity and quality of research produced, as well as reveal evidence gaps. No comprehensive review of the quality and quantity of physical activity intervention trials has been conducted. OBJECTIVE: We aimed to investigate i) the volume and quality (and changes in these over time) of randomized controlled trials evaluating physical activity interventions, and ii) the association between journal ranking and trial quality. METHODS: We searched the Physiotherapy Evidence Database (PEDro) for trials investigating physical activity interventions (no restrictions for population, comparison, or language). Descriptive statistics were used to describe the volume and quality of trials. The association between journal ranking (Journal Impact Factor) and trial quality (PEDro Scale) was examined using Spearman's rho correlation. RESULTS: We identified 1779 trials, of which 40% (n = 710) were published between 2016 and 2020. The mean (SD) total PEDro score was 5.3 (1.5) points out of 10, increasing over time from 2.5 (0.7) points in 1975-1980 to 5.6 (1.4) points in 2016-2020. Quality criteria that were least reported included blinding of intervention deliverers (therapists) (n = 3, 0.2%), participants (n = 21, 1.2%), or assessors (n = 541, 31%); concealed allocation to groups (n = 526, 30%); and intention to treat analysis (n = 764, 43%). There was a small correlation between trial quality and Journal Impact Factor (0.21, p < 0.001). CONCLUSION: A large volume of trials has investigated physical activity interventions. The quality of these trial reports is suboptimal but improving over time. Journal ranking should not be used for selecting high quality trials.

Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies.

OBJECTIVES: To review the findings of studies that have evaluated the design and/or usability of key risk of bias (RoB) tools for the assessment of RoB in primary studies, as categorized by the Library of Assessment Tools and InsTruments Used to assess Data validity in Evidence Synthesis Network (a searchable library of RoB tools for evidence synthesis): Prediction model Risk Of Bias ASessment Tool (PROBAST) , Risk of Bias-2 (RoB2), Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I), Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), Quality Assessment of Diagnostic Accuracy Studies-Comparative (QUADAS-C), Quality Assessment of Prognostic Accuracy Studies (QUAPAS), Risk Of Bias in Non-randomised Studies of Exposures (ROBINS-E), and the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) RoB checklist. STUDY DESIGN AND SETTING: Systematic review of methodological studies. We conducted a forward citation search from the primary report of each tool, to identify primary studies that aimed to evaluate the design and/or usability of the tool. Two reviewers assessed studies for inclusion. We extracted tool features into Microsoft Word and used NVivo for document analysis, comprising a mix of deductive and inductive approaches. We summarized findings within each tool and explored common findings across tools. RESULTS: We identified 13 tool evaluations meeting our inclusion criteria: PROBAST (3), RoB2 (3), ROBINS-I (4), and QUADAS-2 (3). We identified no evaluations for the other tools. Evaluations varied in clinical topic area, methodology, approach to bias assessment, and tool user background. Some had limitations affecting generalizability. We identified common findings across tools for 6/14 themes: (1) challenging items (eg, RoB2/ROBINS-I "deviations from intended interventions" domain), (2) overall RoB judgment (concerns with overall risk calculation in PROBAST/ROBINS-I), (3) tool usability (concerns about complexity), (4) time to complete tool (varying demands on time, eg, depending on number of outcomes assessed), (5) user agreement (varied across tools), and (6) recommendations for future use (eg, piloting) and development (add intermediate domain answer to QUADAS-2/PROBAST; provide clearer guidance for all tools). Of the other eight themes, seven only had findings for the QUADAS-2 tool, limiting comparison across tools, and one ("reorganization of questions") had no findings. CONCLUSION: Evaluations of key RoB tools have posited common challenges and recommendations for tool use and development. These findings may be helpful to people who use or develop RoB tools. Guidance is necessary to support the design and implementation of future RoB tool evaluations.

Reporting Quality and Risk of Bias Analysis of Published RCTs Assessing Anti-CGRP Monoclonal Antibodies in Migraine Prophylaxis: A Systematic Review.

Objective: Phase II/III randomized clinical trials (RCTs) are vulnerable to many types of bias beyond randomization. Insights into the reporting quality of RCTs involving migraine patients treated with monoclonal antibodies targeting the calcitonin gene-related peptide system (anti-CGRP MAbs) are currently lacking. Our aim was to analyze the reporting quality of phase II/III RCTs involving migraine patients treated with anti-CGRP MAbs. Methods: A systematic search was performed on the PubMed and EMBASE databases, according to PRISMA guidelines, for relevant RCTs in either episodic or chronic migraine prevention. Additionally, an adapted version of the 2010 CONSORT statement checklist was utilized. The ROBvis online tool was used to document the risk of bias. Results: From the initially identified 179 articles, we finally found 31 RCTs that were eligible for evaluation. The average CONSORT compliance was 88.7% (69.7-100%), while 93.5% (N = 29) of the articles had a compliance greater than 75%. Twenty-eight CONSORT items were reported in more than 75% of the articles. The average compliance of the analyzed RCTs was 93.9% for Galcanezumab, 91.3% for Fremanezumab, followed by 85.4% for Erenumab and Eptinezumab studies. Implementation of the ROB2 tool showed some concerning "missing information" arising from the inadequate reporting. Specifically, 50% of the studies (N = 16) were categorized as having inadequate information regarding the randomization process. Conclusions: Adequate reporting quality was disclosed in the evaluated RCTs with anti-CGRP MAbs in migraine prevention. However, some methodological issues need to be highlighted to be addressed in future studies assessing the efficacy of new molecules targeting CGRP or other candidate pathways implicated in migraine pathophysiology.

Systematic assessment of the influence of quality of studies on mistletoe in cancer care on the results of a meta-analysis on overall survival.

PURPOSE: Mistletoe treatment in cancer patients is controversial, and a Cochrane review concluded that due to heterogeneity, performing a meta-analysis was not suitable. However, several systematic reviews included meta-analyses in favor of mistletoe. The aim of this work was to assess the influence of the methodological quality of controlled studies on the results of a meta-analysis regarding overall survival. METHODS: Between April and August 2022, Medline, Embase, Cochrane Central Register of Controlled Trials (CENTRAL), PsycINFO, CINAHL and Web of Science were systematically searched. In addition, reference lists of previously published meta-analyses were checked for relevant publications. A random effects meta-analysis with clustering was performed. The risk of bias within the studies was assessed using ROB 2.0 and ROBINS-I. RESULTS: The search identified 4685 hits, and 28 publications reporting on 28 298 patients were included in the quantitative analysis. Overall, the analysis led to a significant result in favor of mistletoe therapy (overall HR = 0.61 with 95% CI [0.53;0.7]). According to our subgroup analysis of randomized studies, studies of higher quality (lower risk of bias) did not lead to a significant result in favor of mistletoe therapy (HR = 0.78; CI = [0.30; 2.00]). CONCLUSIONS: In the case of mistletoe therapy, the results of the meta-analysis strongly depended on the methodological quality of the included studies. Calculating meta-analyses that include low-quality studies may lead to severe misinterpretation of the data.

Evaluation of quality and quantity of randomized controlled trials in hepatobiliary surgery: A scoping/mapping review.

AIM: To evaluate the quantity and quality of randomized controlled trials (RCTs) in hepatobiliary surgery and for identifying gaps in current evidences. METHODS: A systematic search was conducted in MEDLINE (via PubMed), Web of Science, and Cochrane Controlled Register of Trials (CENTRAL) for RCTs of hepatobiliary surgery published from inception until the end of 2023. The quality of each study was assessed using the Cochrane risk-of-bias (RoB) tool. The associations between risk of bias and the region and publication date were also assessed. Evidence mapping was performed to identify research gaps in the field. RESULTS: The study included 1187 records. The number and proportion of published randomized controlled trials (RCTs) in hepatobiliary surgery increased over time, from 13 RCTs (.0005% of publications) in 1970-1979 to 201 RCTs (.003% of publications) in 2020-2023. There was a significant increase in the number of studies with a low risk of bias in RoB domains (p < .01). The proportion of RCTs with low risk of bias improved significantly after the introduction of CONSORT guidelines (p < .001). The evidence mapping revealed a significant research focus on major and minor hepatectomy and cholecystectomy. However, gaps were identified in liver cyst surgery and hepatobiliary vascular surgery. Additionally, there are gaps in the field of perioperative management and nutrition intervention. CONCLUSION: The quantity and quality of RCTs in hepatobiliary surgery have increased over time, but there is still room for improvement. We have identified gaps in current research that can be addressed in future studies.

Triangulation of epidemiological evidence and risk of bias evaluation: A proposed framework and applied example using formaldehyde exposure and risk of myeloid leukemias.

Evidence triangulation may help identify the impact of study design elements on study findings and to tease out biased results when evaluating potential causal relationships; however, methods for triangulating epidemiologic evidence are evolving and have not been standardized. Building upon key principles of epidemiologic evidence triangulation and risk of bias assessment, and responding to the National Academies of Sciences, Engineering, and Medicine (NASEM) call for applied triangulation examples, the objective of this manuscript is to propose a triangulation framework and to apply it as an illustrative example to epidemiologic studies examining the possible relationship between occupational formaldehyde exposure and risk of myeloid leukemias (ML) including acute (AML) and chronic (CML) types. A nine-component triangulation framework for epidemiological evidence was developed incorporating study quality and ROB guidance from various federal health agencies (i.e., US EPA TSCA and NTP OHAT). Several components of the triangulation framework also drew from widely used epidemiological analytic tools such as stratified meta-analysis and sensitivity analysis. Regarding the applied example, fourteen studies were identified and assessed using the following primary study quality domains to explore potential key sources of bias: 1) study design and analysis; 2) study participation; 3) exposure assessment; 4) outcome assessment; and 5) potential confounding. Across studies, methodological limitations possibly contributing to biased results were observed within most domains. Interestingly, results from one study - often providing the largest and least-precise relative risk estimates, likely reflecting study biases, deviated from most primary study findings indicating no such associations. Triangulation of epidemiological evidence appears to be helpful in exploring inconsistent results for the identification of study results possibly reflecting various biases. Nonetheless, triangulation methodologies require additional development and application to real-world examples to enhance objectivity and reproducibility.

A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E).

BACKGROUND: Observational epidemiologic studies provide critical data for the evaluation of the potential effects of environmental, occupational and behavioural exposures on human health. Systematic reviews of these studies play a key role in informing policy and practice. Systematic reviews should incorporate assessments of the risk of bias in results of the included studies. OBJECTIVE: To develop a new tool, Risk Of Bias In Non-randomized Studies - of Exposures (ROBINS-E) to assess risk of bias in estimates from cohort studies of the causal effect of an exposure on an outcome. METHODS AND RESULTS: ROBINS-E was developed by a large group of researchers from diverse research and public health disciplines through a series of working groups, in-person meetings and pilot testing phases. The tool aims to assess the risk of bias in a specific result (exposure effect estimate) from an individual observational study that examines the effect of an exposure on an outcome. A series of preliminary considerations informs the core ROBINS-E assessment, including details of the result being assessed and the causal effect being estimated. The assessment addresses bias within seven domains, through a series of 'signalling questions'. Domain-level judgements about risk of bias are derived from the answers to these questions, then combined to produce an overall risk of bias judgement for the result, together with judgements about the direction of bias. CONCLUSION: ROBINS-E provides a standardized framework for examining potential biases in results from cohort studies. Future work will produce variants of the tool for other epidemiologic study designs (e.g. case-control studies). We believe that ROBINS-E represents an important development in the integration of exposure assessment, evidence synthesis and causal inference.

Assessment of associations between inhaled formaldehyde and lymphohematopoietic cancer through the integration of epidemiological and toxicological evidence with biological plausibility.

Formaldehyde is recognized as carcinogenic for the portal of entry sites, though conclusions are mixed regarding lymphohematopoietic (LHP) cancers. This systematic review assesses the likelihood of a causal relationship between formaldehyde and LHP cancers by integrating components recommended by NASEM. Four experimental rodent bioassays and 16 observational studies in humans were included following the implementation of the a priori protocol. All studies were assessed for risk of bias (RoB), and meta-analyses were conducted on epidemiological studies, followed by a structured assessment of causation based on GRADE and Bradford Hill. RoB analysis identified systemic limitations precluding confidence in the epidemiological evidence due to inadequate characterization of formaldehyde exposure and a failure to adequately adjust for confounders or effect modifiers, thus suggesting that effect estimates are likely to be impacted by systemic bias. Mixed findings were reported in individual studies; meta-analyses did not identify significant associations between formaldehyde inhalation (when measured as ever/never exposure) and LHP outcomes, with meta-SMRs ranging from 0.50 to 1.51, depending on LHP subtype. No associations with LHP-related lesions were reported in reliable animal bioassays. No biologically plausible explanation linking the inhalation of FA and LHP was identified, supported primarily by the lack of systemic distribution and in vivo genotoxicity. In conclusion, the inconsistent associations reported in a subset of the evidence were not considered causal when integrated with the totality of the epidemiological evidence, toxicological data, and considerations of biological plausibility. The impact of systemic biases identified herein could be quantitatively assessed to better inform causality and use in risk assessment.

Risk of bias in cross-sectional studies: Protocol for a scoping review of concepts and tools.

Cross-sectional studies are commonly used to study human health and disease, but are especially susceptible to bias. This scoping review aims to identify and describe available tools to assess the risk of bias (RoB) in cross-sectional studies and to compile the key bias concepts relevant to cross-sectional studies into an item bank. Using the JBI scoping review methodology, the strategy to locate relevant RoB concepts and tools is a combination of database searches, prospective review of PROSPERO registry records; and consultation with knowledge users and content experts. English language records will be included if they describe tools, checklists, or instruments which describe or permit assessment of RoB for cross-sectional studies. Systematic reviews will be included if they consider eligible RoB tools or use RoB tools for RoB of cross-sectional studies. All records will be independently screened, selected, and extracted by one researcher and checked by a second. An analytic framework will be used to structure the extraction of data. Results for the scoping review are pending. Results from this scoping review will be used to inform future selection of RoB tools and to consider whether development of a new RoB tool for cross-sectional studies is needed.