The characteristics, development methods, reporting quality, and evidence basis of diagnostic criteria in China between 1965 and 2023: A cross-sectional study.

OBJECTIVE: As a large and populous country, China releases a high number of diagnostic criteria. However, the published diagnostic criteria have not yet been systematically analyzed. Therefore, the aim of this study is to investigate the characteristics, development methods, reporting quality, and evidence basis of diagnostic criteria published in China. METHODS: We searched five databases for diagnostic criteria from their inception until July 31, 2023. All diagnostic criteria were screened through abstract and full-text reading, and included if satisfying the prespecified criteria. Two researchers independently extracted data on the characteristics, development methods, reporting quality, and evidence basis of diagnostic criteria. RESULTS: A total of 143 diagnostic criteria were included. In terms of development methods, the proportions of diagnostic criteria that involved a systematic literature search (n = 2; 1.4%; 95% confidence interval (CI), 0.4% to 5.0%), adoption of formal consensus methods (n = 4; 2.8%; 95% CI, 1.1% to 7.0%), and criteria validation (n = 9; 6.3%; 95% CI, 3.3% to 11.5%) were relatively low. Regarding reporting quality, the average compliance with the ACCORD checklist was 5.1%; none of the diagnostic criteria reported on registration, expert inclusion criteria, expert recruitment process, or consensus results. A majority (58.7%; 95% CI, 50.6% to 66.5%) of criteria did not cite any research, and only one (0.7%; 95% CI, 0.1% to 3.9%) criterion was derived from a systematic review. Moreover, only 16.1% (95% CI, 11.0% to 23.0%) of diagnostic criteria used evidence from the Chinese population. CONCLUSION: The diagnostic criteria developed in China exhibit serious flaws, particularly in evidence retrieval, formation of expert panels, consensus methods, and validation. Additionally, only few diagnostic criteria used a systematic synthesis of the evidence or evidence from the China. There is an urgent need to enhance the methodology for developing diagnostic criteria.

Adaptive designs were primarily used but inadequately reported in early phase drug trials.

BACKGROUND: Faced with the high cost and limited efficiency of classical randomized controlled trials, researchers are increasingly applying adaptive designs to speed up the development of new drugs. However, the application of adaptive design to drug randomized controlled trials (RCTs) and whether the reporting is adequate are unclear. Thus, this study aimed to summarize the epidemiological characteristics of the relevant trials and assess their reporting quality by the Adaptive designs CONSORT Extension (ACE) checklist. METHODS: We searched MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials (CENTRAL) and ClinicalTrials.gov from inception to January 2020. We included drug RCTs that explicitly claimed to be adaptive trials or used any type of adaptative design. We extracted the epidemiological characteristics of included studies to summarize their adaptive design application. We assessed the reporting quality of the trials by Adaptive designs CONSORT Extension (ACE) checklist. Univariable and multivariable linear regression models were used to the association of four prespecified factors with the quality of reporting. RESULTS: Our survey included 108 adaptive trials. We found that adaptive design has been increasingly applied over the years, and was commonly used in phase II trials (n = 45, 41.7%). The primary reasons for using adaptive design were to speed the trial and facilitate decision-making (n = 24, 22.2%), maximize the benefit of participants (n = 21, 19.4%), and reduce the total sample size (n = 15, 13.9%). Group sequential design (n = 63, 58.3%) was the most frequently applied method, followed by adaptive randomization design (n = 26, 24.1%), and adaptive dose-finding design (n = 24, 22.2%). The proportion of adherence to the ACE checklist of 26 topics ranged from 7.4 to 99.1%, with eight topics being adequately reported (i.e., level of adherence ≥ 80%), and eight others being poorly reported (i.e., level of adherence ≤ 30%). In addition, among the seven items specific for adaptive trials, three were poorly reported: accessibility to statistical analysis plan (n = 8, 7.4%), measures for confidentiality (n = 14, 13.0%), and assessments of similarity between interim stages (n = 25, 23.1%). The mean score of the ACE checklist was 13.9 (standard deviation [SD], 3.5) out of 26. According to our multivariable regression analysis, later published trials (estimated ß = 0.14, p < 0.01) and the multicenter trials (estimated ß = 2.22, p < 0.01) were associated with better reporting. CONCLUSION: Adaptive design has shown an increasing use over the years, and was primarily applied to early phase drug trials. However, the reporting quality of adaptive trials is suboptimal, and substantial efforts are needed to improve the reporting.

Reporting characteristics and quality of randomized controlled trial protocols in traditional Chinese medicine: a cross-sectional study.

Objectives: The impact of the Standard Protocol Items: Recommendations for Interventional Trials of Traditional Chinese Medicine (SPIRIT-TCM) Extension 2018 statement on the reporting quality of randomized controlled trial (RCT) protocols in traditional Chinese medicine (TCM) is not clear. This review aimed to assess the reporting characteristics and quality of RCT protocols involving interventions such as Chinese herbal medicine formulas (CHMF), acupuncture, and moxibustion published in the last 3 years. Methods: We conducted an extensive search among multiple databases, including All EBM Reviews, Allied and Complementary Medicine (AMED), Embase, Ovid MEDLINE(R), PubMed, Web of Science, Google Scholar, and ClinicalTrials.gov for publications in English from 1 January 2020 to 10 August 2023. Two reviewers independently assessed the eligibility of the publications, extracted predetermined information, and evaluated the reporting based on the SPIRIT-TCM Extension 2018 checklist. Results: Of the 420 eligible protocols (comprising 163 studies on CHMF, 239 on acupuncture, and 18 on moxibustion), the average reporting compliance rate was only 35.4%. Approximately half of the assessed items fell into the category of poorly reported, demonstrating a compliance rate below 65%. Notably, reporting compliance in acupuncture and moxibustion interventional studies exhibited higher scores than compliance in CHMF studies. Conclusion: Continued, concerted, and coordinated efforts are required by journals, editors, reviewers, and investigators to improve the application and promotion of the SPIRIT-TCM Extension 2018 reporting guideline.

The quality of reporting in case reports of permanent neonatal diabetes mellitus: a cross-sectional study.

BACKGROUND: Although randomized trials and systematic reviews provide the best evidence to guide medical practice, many permanent neonatal diabetes mellitus (PNDM) studies have been published as case reports. However, the quality of these studies has not been assessed. The purpose of this study was to assess the extent to which the current case reports for PNDM comply with the Case Report (CARE) guidelines and to explore variables associated with the reporting. METHOD: Six English and four Chinese databases were searched from their inception to December 2022 for PNDM case reports. The 23 items CARE checklist was used to measure reporting quality. Primary outcome was the adherence rate of each CARE item and second outcome was total reporting score for each included PNDM case report. Linear and logistic regression analyses were used to examine the connection between five pre-specified predictor variables and the reporting quality. The predictor variables were impact factor of the published journal (<3.4 vs. ≥3.4, categorized according to the median), funding (yes vs. no), language (English vs. other language), published journal type (general vs. special) and year of publication (>2013 vs. ≤ 2013). RESULT: In total, 105 PNDM case reports were included in this study. None of the 105 PNDM case reports fulfilled all 23 items of the CARE checklist. The response rate of 11 items were under 50%, including prognostic characteristics presentation (0%), patient perspective interpretation (0%), diagnostic challenges statement (2.9%), clinical course summary (21.0%), diagnostic reasoning statement (22.9%), title identification (24.8%), case presentation (33.3%), disease history description (34.3%), strengths and limitations explanation (41.0%), informed consent statement (45.7%), and lesson elucidation (47.6%). This study identified that the PNDM case reports published in higher impact factor journals were statistically associated with a higher reporting quality. CONCLUSION: The reporting of case reports for PNDM is generally poor. As a result, this information may be misleading to providers, and the clinical applications may be detrimental to patient care. To improve reporting quality, journals should encourage strict adherence to the CARE guidelines.

Enhancing reporting through structure: a before and after study on the effectiveness of SPIRIT-based templates to improve the completeness of reporting of randomized controlled trial protocols.

BACKGROUND: Despite the improvements in the completeness of reporting of randomized trial protocols after the publication of the Standard Protocol Items: Recommendations for Interventional Trial (SPIRIT) guidelines, many items remain poorly reported. This study aimed to assess the effectiveness of using SPIRIT-tailored templates for trial protocols to improve the completeness of reporting of the protocols that master's students write as part of their master's theses. METHODS: Before and after experimental study performed at the University Master's Degree in Orthopaedic Manual Physiotherapy of the Universitat Internacional de Catalunya (Barcelona, Spain). While students in the post-intervention period were instructed to use a trial protocol template that was tailored to SPIRIT, students in the pre-intervention period did not use the template. PRIMARY OUTCOME: Difference between the pre- and post-intervention periods in the mean number of adequately reported items (0-10 scale). The outcomes were evaluated independently and in duplicate by two blinded assessors. Students and their supervisors were not aware that they were part of a research project. For the statistical analysis, we used a generalized linear regression model (dependent variable: number of adequately reported items in the protocol; independent variables: intervention period, call, language). RESULTS: Thirty-four trial protocols were included (17, pre-intervention; 17, post-intervention). Protocols produced during the post-intervention period (mean: 8.24; SD: 1.52) were more completely reported than those produced during the pre-intervention period (mean: 6.35; SD: 1.80); adjusted difference: 1.79 (95% CI: 0.58 to 3.00). CONCLUSIONS: SPIRIT-based templates could be used to improve the completeness of reporting of randomized trial protocols.

Assessment of description and implementation fidelity of clinical trials involving exercise-based treatment in individuals with rotator cuff tears: a scoping review.

BACKGROUND: For evidence-based practice, clinicians and researchers can rely on well-conducted randomized clinical trials that exhibit good methodological quality, provide adequate intervention descriptions, and implementation fidelity. OBJECTIVE: To assess the description and implementation fidelity of exercise-based interventions in clinical trials for individuals with rotator cuff tears. METHODS: A systematic search was conducted in PubMed, Embase, CINAHL, LILACS, Cochrane Library, Web of Science, SCOPUS and SciELO. Randomized clinical trials that assessed individuals with rotator cuff tears confirmed by imaging exam were included. All individuals must have received an exercise-based treatment. The methodological quality was scored with the Physiotherapy Evidence Database (PEDro) scale. The Template for Intervention Description and Replication (TIDieR) checklist and the National Institutes of Health Behaviour Change Consortium (NIHBCC) were used to assess intervention description and implementation fidelity, respectively. RESULTS: A total of 13 studies were included. Despite their adequate methodological quality, the description of the intervention was poor with TIDieR scores ranging from 6 to 15 out of 24 total points. The TIDieR highest-scoring item was item 1 (brief name) that was reported in all studies. Considering fidelity, only one of the five domains of NIHBCC (i.e., treatment design) reached just over 50%. CONCLUSION: Exercise-based interventions used in studies for individuals with rotator cuff tears are poorly reported. The description and fidelity of the intervention need to be better reported to assist clinical decision-making and support evidence-based practice.

Automatic categorization of self-acknowledged limitations in randomized controlled trial publications.

OBJECTIVE: Acknowledging study limitations in a scientific publication is a crucial element in scientific transparency and progress. However, limitation reporting is often inadequate. Natural language processing (NLP) methods could support automated reporting checks, improving research transparency. In this study, our objective was to develop a dataset and NLP methods to detect and categorize self-acknowledged limitations (e.g., sample size, blinding) reported in randomized controlled trial (RCT) publications. METHODS: We created a data model of limitation types in RCT studies and annotated a corpus of 200 full-text RCT publications using this data model. We fine-tuned BERT-based sentence classification models to recognize the limitation sentences and their types. To address the small size of the annotated corpus, we experimented with data augmentation approaches, including Easy Data Augmentation (EDA) and Prompt-Based Data Augmentation (PromDA). We applied the best-performing model to a set of about 12K RCT publications to characterize self-acknowledged limitations at larger scale. RESULTS: Our data model consists of 15 categories and 24 sub-categories (e.g., Population and its sub-category DiagnosticCriteria). We annotated 1090 instances of limitation types in 952 sentences (4.8 limitation sentences and 5.5 limitation types per article). A fine-tuned PubMedBERT model for limitation sentence classification improved upon our earlier model by about 1.5 absolute percentage points in F1 score (0.821 vs. 0.8) with statistical significance (p<.001). Our best-performing limitation type classification model, PubMedBERT fine-tuning with PromDA (Output View), achieved an F1 score of 0.7, improving upon the vanilla PubMedBERT model by 2.7 percentage points, with statistical significance (p<.001). CONCLUSION: The model could support automated screening tools which can be used by journals to draw the authors' attention to reporting issues. Automatic extraction of limitations from RCT publications could benefit peer review and evidence synthesis, and support advanced methods to search and aggregate the evidence from the clinical trial literature.

Reporting quality of acupuncture overviews: A methodological investigation based on the PRIOR statement.

OBJECTIVE: Acupuncture overviews are increasing rapidly; however, their reporting quality is yet unclear. We aimed to investigate the reporting quality of relevant overviews according to the preferred reporting items for overviews of reviews (PRIOR) statement. METHODS: We systematically searched PubMed from inception to August 16, 2022 for overviews on acupuncture therapies. Reporting quality of included overviews was evaluated using the PRIOR statement, and the results were cross-checked. Multiple linear regression analysis was used to assess the predictors of the reporting completeness. GraphPad 9.4 was utilized to generate an evidence map, Excel 2019 was used to extract and manage data, and R 4.2.3 was used for data analysis. RESULTS: A total of 49 overviews published from 2006 to 2022 were included, of which China ranked first with 38 overviews. The most frequently searched database was PubMed/ Medline (n = 48, 98%), and commonly used methodological quality assessment tool was AMSTAR-2 (n = 14, 29%). The overarching themes centered on acupuncture for obstetrics, gynecology, reproductive diseases, as well as depression, anxiety, and insomnia. Reporting quality needs to be improved involving the definition of systematic reviews (SRs), overlap of primary studies and SRs, methods for managing discrepant data across SRs, risk of bias in primary studies, heterogeneity, and sensitivity analysis of synthesized results, reporting bias assessment, and registration and protocol. Moreover, publication in recent years and receiving funding support were significantly associated with higher overall reporting quality score (P < 0.05). CONCLUSION: Based on the PRIOR statement, this methodological study indicates that the reporting quality of the included acupuncture overviews is poor. In the future, authors of overviews are encouraged to use the PRIOR statement for standardized reporting. Furthermore, it is recommended that journal editors mandate the inclusion of this statement in authors' reports and require a complete PRIOR checklist.

Radiomics Studies on Ischemic Stroke and Carotid Atherosclerotic Disease: A Reporting Quality Assessment.

Objective: To assess the reporting quality of radiomics studies on ischemic stroke, intracranial and carotid atherosclerotic disease using the Image Biomarker Standardization Initiative (IBSI) reporting guidelines with the aim of finding avenues of improvement for future publications. Method: PubMed database was searched to identify relevant radiomics studies. Of 560 articles, 41 original research articles were included in this analysis. Based on IBSI radiomics reporting guidelines, checklists for CT-based and MRI-based studies were created to allow a structured and comprehensive evaluation of each study's adherence to these guidelines. Results: The main topics covered by the included radiomics studies were ischemic stroke, intracranial artery disease, and carotid atherosclerotic disease. The reporting checklist median score was 17/40 for the 20 CT-based radiomics studies and 22.5/50 for the 20 MRI-based studies. Basic items like imaging modality, region of interest, and image biomarker set utilized were included in all studies. However, details regarding image acquisition and reconstruction, post-acquisition image processing, and image biomarkers computation were inconsistently detailed across studies. Conclusion: The overall reporting quality of the included radiomics studies was suboptimal. These findings underscore a pressing need for improved reporting practices in radiomics research, to ensure validation and reproducibility of results. Our study provides insights into current reporting standards and highlights specific areas where adherence to IBSI guidelines could be significantly improved.

The Reporting Quality of Machine Learning Studies on Pediatric Diabetes Mellitus: Systematic Review.

BACKGROUND: Diabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. OBJECTIVE: We aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. METHODS: We searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. RESULTS: After screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. CONCLUSIONS: The reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable.

Definition and reporting of composite outcomes are often inadequate in randomized clinical trials on pharmacological interventions for coronary artery disease.

OBJECTIVES: To estimate the frequency and critically appraise the use and reporting of composite outcomes in randomized clinical trials on pharmacological interventions for coronary artery disease. STUDY DESIGN AND SETTING: A metaresearch study. A search strategy was developed to retrieve references from MEDLINE. We considered articles, published from 1st January 2020, to December 31, 2021, reporting results of clinical primary outcomes from randomized clinical trials which assessed pharmacological interventions, used alone or in combination, for the treatment or secondary prevention (previous coronary event) of coronary artery disease. RESULTS: From the 34 included studies, 28 (82.35%) had a primary composite outcome. Thirteen unique composite primary outcomes were used with the most frequent being "cardiovascular death, myocardial infarction, stroke" (12/28, 42.86%). The term major adverse cardiac events was used for five distinct composite primary outcomes. A combination of 12 different components resulted in the 28 primary composite outcomes, with stroke being the most frequent component present in 96.43% (27/28) of the primary composite outcomes. From the included studies, 60.71% (17/28) reported the estimates for each individual component and the direction of the effect was consistent between all components and the composite outcomes in 58.82% (10/17) of them. Additionally, no included study discussed potential limitations and/or related advantages of the composite outcomes. CONCLUSION: In randomized clinical trials on pharmacological interventions for coronary artery disease, composite outcomes are frequently used, but the definition of their components is very heterogeneous. The estimate for individual components within the composite outcome is often not fully reported, which prevents a complete analysis of their adequacy for clinical practice. The term major adverse cardiac events was used inconsistently and to refer to different set of components, which can also be misleading and confusing.

Quality of reporting of integrative Chinese and Western medicine intervention in randomized controlled trials of ulcerative colitis: a review.

BACKGROUND: Integrative Chinese and Western medicine (ICWM) is commonly used for the treatment of ulcerative colitis (UC) in clinical practice. However, it is unclear whether the details of ICWM interventions, such as selection rationale, implementation design, and potential interactions, were adequately reported. Therefore, this study aimed to assess the quality of reporting in the ICWM interventional randomized controlled trials (RCTs) of UC and to identify the common problems if any. METHODS: Through a search of 10 international electronic databases, we identified RCTs of UC with ICWM interventions published in English or Chinese from the inception date of each database up to 16 June 2023. Literature screening was strictly conducted based on the inclusion and exclusion criteria of the Population, Concept, and Context (PCC) framework. The general characteristics of the included studies were described. The quality of reporting was assessed according to three checklists, including the CONSORT (Consolidated Standards of Reporting Trials) with 36 items (except for one item 1b about abstract), the CONSORT for Abstracts (17 items), and a self-designed ICWM-related checklist (27 items covering design rationale, intervention details, outcome assessments, and analysis). The reporting scores of RCTs published before and after 2010 were compared. RESULTS: A total of 1458 eligible RCTs were included. For the reporting compliance, the median score (interquartile ranges) of the CONSORT (72 score in total), the CONSORT for Abstract (34 score), and ICWM-related (54 score) items was 21 (18-25), 13 (12-15), and 18 (15-21), respectively. Although the time period comparisons showed that reporting quality of included publications improved significantly after the CONSORT 2010 issued (P < 0.01), more than 50% of items were evaluated as poor quality (reporting rate < 65%) among each checklist, especially in the CONSORT for Abstract and ICWM-specific items. CONCLUSION: Although CONSORT appears to have enhanced the reporting of RCTs in UC, the quality of ICWM specifics is variable and in need of improvement. Reporting guidelines of the ICWM recommendations should be developed to improve their quality.

Abstracts of randomized controlled trials in pediatric dentistry: reporting quality and spin.

BACKGROUND: Abstracts provide readers a concise and readily accessible information of the trials. However, poor reporting quality and spin (misrepresentation of research findings) can lead to an overestimation in trial validity. This methodological study aimed to assess the reporting quality and spin among randomized controlled trial (RCT) abstracts in pediatric dentistry. METHODS: We hand-searched RCTs in five leading pediatric dental journals between 2015 and 2021. Reporting quality in each abstract was assessed using the original 16-item CONSORT for abstracts checklist. Linear regression analyses were performed to identify factors associated with reporting quality. We evaluated the presence and characteristics of spin only in abstracts of parallel-group RCTs with nonsignificant primary outcomes according to pre-determined spin strategies. RESULTS: One hundred eighty-two abstracts were included in reporting quality evaluation. The mean overall quality score was 4.57 (SD, 0.103; 95% CI, 4.36-4.77; score range, 1-10). Only interventions, objective, and conclusions were adequately reported. Use of flow diagram (P < 0.001) was the only significant factor of higher reporting quality. Of the 51 RCT abstracts included for spin analysis, spin was identified in 40 abstracts (78.4%), among which 23 abstracts (45.1%) had spin in the Results section and 39 in the Conclusions Sect. (76.5%). CONCLUSIONS: The reporting quality of RCT abstracts in pediatric dentistry is suboptimal and the prevalence of spin is high. Joint efforts are needed to improve reporting quality and minimize spin.

Statistical significance and publication reporting bias in abstracts of reproductive medicine studies.

STUDY QUESTION: What were the frequency and temporal trends of reporting P-values and effect measures in the abstracts of reproductive medicine studies in 1990-2022, how were reported P-values distributed, and what proportion of articles that present with statistical inference reported statistically significant results, i.e. 'positive' results? SUMMARY ANSWER: Around one in six abstracts reported P-values alone without effect measures, while the prevalence of effect measures, whether reported alone or accompanied by P-values, has been increasing, especially in meta-analyses and randomized controlled trials (RCTs); the reported P-values were frequently observed around certain cut-off values, notably at 0.001, 0.01, or 0.05, and among abstracts present with statistical inference (i.e. P-value, CIs, or significant terms), a large majority (77%) reported at least one statistically significant finding. WHAT IS KNOWN ALREADY: Publishing or reporting only results that show a 'positive' finding causes bias in evaluating interventions and risk factors and may incur adverse health outcomes for patients.Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time. STUDY DESIGN, SIZE, DURATION: We studied abstracts of reproductive medicine studies from 1990 to 2022. The reproductive medicine studies were published in 23 first-quartile journals under the category of Obstetrics and Gynaecology and Reproductive Biology in Journal Citation Reports and 5 high-impact general medical journals (The Journal of the American Medical Association, The Lancet, The BMJ, The New England Journal of Medicine, and PLoS Medicine). Articles without abstracts, animal studies, and non-research articles, such as case reports or guidelines, were excluded. PARTICIPANTS/MATERIALS, SETTING, METHODS: Automated text-mining was used to extract three types of statistical significance reporting, including P-values, CIs, and text description. Meanwhile, abstracts were text-mined for the presence of effect size metrics and Bayes factors. Five hundred abstracts were randomly selected and manually checked for the accuracy of automatic text extraction. The extracted statistical significance information was then analysed for temporal trends and distribution in general as well as in subgroups of study designs and journals. MAIN RESULTS AND THE ROLE OF CHANCE: A total of 24 907 eligible reproductive medicine articles were identified from 170 739 screened articles published in 28 journals. The proportion of abstracts not reporting any statistical significance inference halved from 81% (95% CI, 76-84%) in 1990 to 40% (95% CI, 38-44%) in 2021, while reporting P-values alone remained relatively stable, at 15% (95% CI, 12-18%) in 1990 and 19% (95% CI, 16-22%) in 2021. By contrast, the proportion of abstracts reporting effect measures alone increased considerably from 4.1% (95% CI, 2.6-6.3%) in 1990 to 26% (95% CI, 23-29%) in 2021. Similarly, the proportion of abstracts reporting effect measures together with P-values showed substantial growth from 0.8% (95% CI, 0.3-2.2%) to 14% (95% CI, 12-17%) during the same timeframe. Of 30 182 statistical significance inferences, 56% (n = 17 077) conveyed statistical inferences via P-values alone, 30% (n = 8945) via text description alone such as significant or non-significant, 9.3% (n = 2820) via CIs alone, and 4.7% (n = 1340) via both CI and P-values. The reported P-values (n = 18 417), including both a continuum of P-values and dichotomized P-values, were frequently observed around common cut-off values such as 0.001 (20%), 0.05 (16%), and 0.01 (10%). Of the 13 200 reproductive medicine abstracts containing at least one statistical inference, 77% of abstracts made at least one statistically significant statement. Among articles that reported statistical inference, a decline in the proportion of making at least one statistically significant inference was only seen in RCTs, dropping from 71% (95% CI, 48-88%) in 1990 to 59% (95% CI, 42-73%) in 2021, whereas the proportion in the rest of study types remained almost constant over the years. Of abstracts that reported P-value, 87% (95% CI, 86-88%) reported at least one statistically significant P-value; it was 92% (95% CI, 82-97%) in 1990 and reached its peak at 97% (95% CI, 93-99%) in 2001 before declining to 81% (95% CI, 76-85%) in 2021. LIMITATIONS, REASONS FOR CAUTION: First, our analysis focused solely on reporting patterns in abstracts but not full-text papers; however, in principle, abstracts should include condensed impartial information and avoid selective reporting. Second, while we attempted to identify all types of statistical significance reporting, our text mining was not flawless. However, the manual assessment showed that inaccuracies were not frequent. WIDER IMPLICATIONS OF THE FINDINGS: There is a welcome trend that effect measures are increasingly reported in the abstracts of reproductive medicine studies, specifically in RCTs and meta-analyses. Publication reporting bias remains a major concern. Inflated estimates of interventions and risk factors could harm decisions built upon biased evidence, including clinical recommendations and planning of future research. STUDY FUNDING/COMPETING INTEREST(S): No funding was received for this study. B.W.M. is supported by an NHMRC Investigator grant (GNT1176437); B.W.M. reports research grants and travel support from Merck and consultancy from Merch and ObsEva. W.L. is supported by an NHMRC Investigator Grant (GNT2016729). Q.F. reports receiving a PhD scholarship from Merck. The other author has no conflict of interest to declare. TRIAL REGISTRATION NUMBER: N/A.

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia.

OBJECTIVE: We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit. MATERIALS AND METHODS: A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020. RESULTS: Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020. CONCLUSION: The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

Methodological and reporting quality of systematic reviews on health effects of air pollutants were higher than extreme temperatures: a comparative study.

BACKGROUND: An increasing number of systematic reviews (SRs) in the environmental field have been published in recent years as a result of the global concern about the health impacts of air pollution and temperature. However, no study has assessed and compared the methodological and reporting quality of SRs on the health effects of air pollutants and extreme temperatures. This study aims to assess and compare the methodological and reporting quality of SRs on the health effects of ambient air pollutants and extreme temperatures. METHODS: PubMed, Embase, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane Library, Web of Science, and Epistemonikos databases were searched. Two researchers screened the literature and extracted information independently. The methodological quality of the SRs was assessed through A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2). The reporting quality was assessed through Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA). RESULTS: We identified 405 SRs (286 for air pollution, 108 for temperature, and 11 for the synergistic effects). The methodological and reporting quality of the included SRs were suboptimal, with major deficiencies in protocol registration. The methodological quality of SRs of air pollutants was better than that of temperature, especially in terms of satisfactory explanations for any heterogeneity (69.6% v. 45.4%). The reporting quality of SRs of air pollution was better than temperature, however, adherence to the reporting of the assessment results of risk of bias in all SRs (53.5% v. 34.3%) was inadequate. CONCLUSIONS: Methodological and reporting quality of SRs on the health effect of air pollutants were higher than those of temperatures. However, deficiencies in protocol registration and the assessment of risk of bias remain an issue for both pollutants and temperatures. In addition, developing a risk-of-bias assessment tool applicable to the temperature field may improve the quality of SRs.

Low-quality of patient-reported outcome reporting in randomized clinical trials of major depressive disorder-a meta-epidemiological review.

Objective: Our goal was to review current peer-reviewed articles in which the BDI (Beck Depression Inventory), PHQ-9 (Patient Health Questionnaire), or QIDS-SR16 (16-Item Quick Inventory of Depressive Symptomatology) was used as the primary or secondary outcome measure and to evaluate the quality of PRO (Patient-Reported Outcome) reporting in RCTs (Randomized Controlled Trials) according to the 2013 PRO-specific CONSORT (Consolidated Standards of Reporting Trials) extension. Methods: We systematically searched in electronic databases. A study would be included if it included patients diagnosed with major depressive disorder according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders (DSM) or International Classification of Diseases, version 10 (ICD-10) as participants, was a randomized controlled trial, included the BDI, PHQ-9, or QIDS-SR16 as the primary or secondary outcome measure, published between 1990 and 2013, and was in English. Two of the authors evaluated the quality of PRO reporting according to the 2013 CONSORT-PRO. Logistic regression were used to evaluate the association between reporting completeness and trial characteristics. Results: A total of 116 studies were included. These studies were conducted in 25 countries. Sample sizes ranged from 12 to 750. The CONSORT-PRO was not cited in any one of the included studies. Among the 116 studies, 2 (1.72%) studies introduced the rationale for PRO assessment, 60 (51.72%) studies explicitly stated statistical approaches for dealing with missing data, 87 (75.00%) studies reported PRO outcome data at baseline and at subsequent time points. The mean score of reporting completeness was 66.24%. Significantly higher reporting completeness was found for RCTs published after 2013 (OR, 95%CI: 3.81, 1.32-10.99). Studies with a higher sample size were more completely reported than studies with a lower sample size (OR, 95%CI: 1.01, 1.00-1.02). Conclusion: The CONSORT-PRO guidance was rarely cited. The quality of PRO reporting in depression studies requires improvement. This result may be meaningful for the promotion of PRO reporting in RCTs.

Cochrane review abstracts are getting longer, but this has no large impact on the reporting quality.

OBJECTIVE: To evaluate the reporting of abstracts of Cochrane Reviews according to PRISMA-A and to investigate a possible association with the abstract´s length. METHODS: This is a retrospective, observational study based on all Cochrane Reviews indexed in Medline (via PubMed) until November 18, 2022. In the second part, a random sample of 440 abstracts was drawn, in which PRISMA-A adherence was assessed by two independent reviewers. Analyses were stratified by the year of publication and the number of words. RESULTS: Overall, the median number of words of the 15,188 included abstracts was 469 (IQR 389-686 words), steadily increasing from 353 words in 2000 to 838 words in 2022, with less than one percent of the abstracts in 2022 having ≤ 300 words (in 2000: 30.7%). Analyses on PRISMA-A adherence in the random sample showed a mean score of 6.1 out of 12 fully reported items. Stratified by year, PRISMA-A adherence increased, with higher word counts in 2000-2010 and 2011-2015, while there was no difference in PRISMA-A adherence by abstract length in 2016-2022. CONCLUSION: Over the years, abstracts of Cochrane Reviews have become longer, running up to 1000 words. This conflicts with the Cochrane Handbook, which recommends a maximum length of 400 until it was aligned with MECIR in 2019, which has recommended a length of <700 words since 2012 but allows up to 1000 words. It is debatable whether such long abstracts meet the key goals of abstracts of being informative, accurate, appealing, and concise.

Use of guidelines, checklists, frameworks, and recommendations in behavioral intervention preliminary studies and associations with reporting comprehensiveness: a scoping bibliometric review.

BACKGROUND: Guidelines, checklists, frameworks, and recommendations (GCFRs) related to preliminary studies serve as essential resources to assist behavioral intervention researchers in reporting findings from preliminary studies, but their impact on preliminary study reporting comprehensiveness is unknown. The purpose of this study was to conduct a scoping bibliometric review of recently published preliminary behavioral-focused intervention studies to (1) examine the prevalence of GCFR usage and (2) determine the associations between GCFR usage and reporting feasibility-related characteristics. METHODS: A systematic search was conducted for preliminary studies of behavioral-focused interventions published between 2018 and 2020. Studies were limited to the top 25 journals publishing behavioral-focused interventions, text mined to identify usage of GCFRs, and categorized as either not citing GCFRs or citing ≥ 2 GCFRs (Citers). A random sample of non-Citers was text mined to identify studies which cited other preliminary studies that cited GCFRs (Indirect Citers) and those that did not (Never Citers). The presence/absence of feasibility-related characteristics was compared between Citers, Indirect Citers, and Never Citers via univariate logistic regression. RESULTS: Studies (n = 4143) were identified, and 1316 were text mined to identify GCFR usage (n = 167 Citers). A random sample of 200 studies not citing a GCFR were selected and categorized into Indirect Citers (n = 71) and Never Citers (n = 129). Compared to Never Citers, Citers had higher odds of reporting retention, acceptability, adverse events, compliance, cost, data collection feasibility, and treatment fidelity (ORrange = 2.62-14.15, p < 0.005). Citers also had higher odds of mentioning feasibility in purpose statements, providing progression criteria, framing feasibility as the primary outcome, and mentioning feasibility in conclusions (ORrange = 6.31-17.04, p < 0.005) and lower odds of mentioning efficacy in purpose statements, testing for efficacy, mentioning efficacy in conclusions, and suggesting future testing (ORrange = 0.13-0.54, p < 0.05). Indirect Citers had higher odds of reporting acceptability and treatment fidelity (ORrange = 2.12-2.39, p < 0.05) but lower odds of testing for efficacy (OR = 0.36, p < 0.05) compared to Never Citers. CONCLUSION: The citation of GCFRs is associated with greater reporting of feasibility-related characteristics in preliminary studies of behavioral-focused interventions. Researchers are encouraged to use and cite literature that provides guidance on design, implementation, analysis, and reporting to improve the comprehensiveness of reporting for preliminary studies.

Methodological quality and reporting quality of COVID-19 living systematic review: a cross-sectional study.

OBJECTIVES: The main objective of this study is to evaluate the methodological quality and reporting quality of living systematic reviews (LSRs) on Coronavirus disease 2019 (COVID-19), while the secondary objective is to investigate potential factors that may influence the overall quality of COVID-19 LSRs. METHODS: Six representative databases, including Medline, Excerpta Medica Database (Embase), Cochrane Library, China national knowledge infrastructure (CNKI), Wanfang Database, and China Science, Technology Journal Database (VIP) were systematically searched for COVID-19 LSRs. Two authors independently screened articles, extracted data, and then assessed the methodological and reporting quality of COVID-19 LSRs using the "A Measurement Tool to Assess systematic Reviews-2" (AMSTAR-2) tool and "Preferred Reporting Items for Systematic reviews and Meta-Analyses" (PRISMA) 2020 statement, respectively. Univariate linear regression and multivariate linear regression were used to explore eight potential factors that might affect the methodological quality and reporting quality of COVID-19 LSRs. RESULTS: A total of 64 COVID-19 LSRs were included. The AMSTAR-2 evaluation results revealed that the number of "yes" responses for each COVID-19 LSR was 13 ± 2.68 (mean ± standard deviation). Among them, 21.9% COVID-19 LSRs were rated as "high", 4.7% as "moderate", 23.4% as "low", and 50% as "critically low". The evaluation results of the PRISMA 2020 statement showed that the sections with poor adherence were methods, results and other information. The number of "yes" responses for each COVID-19 LSR was 21 ± 4.18 (mean ± standard deviation). The number of included studies and registration are associated with better methodological quality; the number of included studies and funding are associated with better reporting quality. CONCLUSIONS: Improvement is needed in the methodological and reporting quality of COVID-19 LSRs. Researchers conducting COVID-19 LSRs should take note of the quality-related factors identified in this study to generate evidence-based evidence of higher quality.