Physician characteristics associated with proper assessment of overstated conclusions in research abstracts: A secondary analysis of a randomized controlled trial.

OBJECTIVES: Little is known about the physician characteristics associated with appraisal skills of research evidence, especially the assessment of the validity of study methodology. This study aims to explore physician characteristics associated with proper assessment of overstated conclusions in research abstracts. DESIGN: A secondary analysis of a randomized controlled trial. SETTING AND PARTICIPANTS: We recruited 567 volunteers from the Japan Primary Care Association. METHODS: Participants were randomly assigned to read the abstract of a research paper, with or without an overstatement, and to rate its validity. Our primary outcome was proper assessment of the validity of its conclusions. We investigated the association of physician characteristics and proper assessment using logistic regression models and evaluated the interaction between the associated characteristics and overstatement. RESULTS: We found significant associations between proper assessment and post-graduate year (odds ratio [OR] = 0.67, 95% confidence interval [CI] 0.49 to 0.91, for every 10-year increase) and research experience as a primary investigator (PI; OR = 2.97, 95% CI 1.65 to 5.34). Post-graduate year and PI had significant interaction with overstatement (P = 0.015 and < 0.001, respectively). Among participants who read abstracts without an overstatement, post-graduate year was not associated with proper assessment (OR = 1.04, 95% CI 0.82 to 1.33), and PI experience was associated with lower scores of the validity (OR = 0.58, 95% CI 0.35 to 0.96). CONCLUSION: Physicians who have been in practice longer should be trained in distinguishing overstatements in abstract conclusions. Physicians with research experience might be informed that they tend to rate the validity of research lower regardless of the presence or absence of overstatements. TRIAL REGISTRATION: UMIN000026269.

Effect of Recruitment Methods on Response Rate in a Web-Based Study for Primary Care Physicians: Factorial Randomized Controlled Trial.

BACKGROUND: Low participation rates are one of the most serious disadvantages of Web-based studies. It is necessary to develop effective strategies to improve participation rates to obtain sufficient data. OBJECTIVE: The objective of this trial was to investigate the effect of emphasizing the incentive in the subject line of the invitation email and the day of the week of sending the invitation email on the participation rate in a Web-based trial. METHODS: We conducted a 2×2 factorial design randomized controlled trial. We contacted 2000 primary care physicians from members of the Japan Primary Care Association in January 2017 and randomly allocated them to 1 of 4 combinations of 2 subject lines (presence or absence of an emphasis on a lottery for an Amazon gift card worth 3000 yen or approximately US $30) and 2 delivery days (sending the invitation email on Tuesday or Friday). The primary outcome was the response rate defined as the number of participants answering the first page of the questionnaire divided by the number of invitation emails delivered. All outcomes were collected between January 17, 2017, and February 8, 2017. RESULTS: We analyzed data from 1943 out of 2000 participants after excluding those whose email addresses were invalid. The overall response rate was 6.3% (123/1943). There was no significant difference in the response rates between the 2 groups regarding incentive in the subject line: the risk ratio was 1.12 (95% CI 0.80 to 1.58) and the risk difference was 0.7% (95% CI -1.5% to 2.9%). Similarly, there was no significant difference in the response rates between the 2 groups regarding sending the email on Tuesday or Friday: the risk ratio was 0.98 (95% CI 0.70 to 1.38) and the risk difference was -0.1% (95% CI -2.3% to 2.1%). CONCLUSIONS: Neither emphasizing the incentive in the subject line of the invitation email nor varying the day of the week the invitation email was sent led to a meaningful increase in response rates in a Web-based trial with primary care physicians. TRIAL REGISTRATION: University Hospital Medical Information Network Clinical Trials Registry UMIN000025317; https://upload.umin.ac.jp/cgi-open-bin/ctr_e/ctr_view.cgi?recptno=R000029121 (Archived by WebCite at http://www.webcitation. org/6wOo1jl9t).

Influence of overstated abstract conclusions on clinicians: a web-based randomised controlled trial.

OBJECTIVES: To investigate whether overstatements in abstract conclusions influence primary care physicians' evaluations when they read reports of randomised controlled trials (RCTs) DESIGN: RCT setting: This study was a parallel-group randomised controlled survey, conducted online while masking the study hypothesis. PARTICIPANTS: Volunteers were recruited from members of the Japan Primary Care Association in January 2017. We sent email invitations to 7040 primary care physicians. Among the 787 individuals who accessed the website, 622 were eligible and automatically randomised into 'without overstatement' (n=307) and 'with overstatement' (n=315) groups. INTERVENTIONS: We selected five abstracts from published RCTs with at least one non-significant primary outcome and overstatement in the abstract conclusion. To construct a version without overstatement, we rewrote the conclusion sections. The methods and results sections were standardised to provide the necessary information of primary outcome information when it was missing in the original abstract. Participants were randomly assigned to read an abstract either with or without overstatements and asked to evaluate the benefit of the intervention. OUTCOME MEASURES: The primary outcome was the participants' evaluation of the benefit of the intervention discussed in the abstract, on a scale from 0 to 10. A secondary outcome was the validity of the conclusion. RESULTS: There was no significant difference between the groups with respect to their evaluation of the benefit of the intervention (mean difference: 0.07, 95% CI -0.28 to 0.42, p=0.69). Participants in the 'without' group considered the study conclusion to be more valid than those in the 'with' group (mean difference: 0.97, 95% CI 0.59 to 1.36, P<0.001). CONCLUSION: The overstatements in abstract conclusions did not significantly influence the primary care physicians' evaluations of the intervention effect when necessary information about the primary outcomes was distinctly reported. TRIAL REGISTRATION NUMBER: UMIN000025317; Pre-results.

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation.

OBJECTIVE: Abstracts of scientific reports are sometimes criticized for exaggerating significant results when compared to the corresponding full texts. Such abstracts can mislead the readers. We aimed to conduct a systematic review of overstatements in abstract conclusions in psychiatry trials. METHODS: We searched for randomized controlled trials published in 2014 that explicitly claimed effectiveness of any intervention for mental disorders in their abstract conclusion, using the Cochrane Register of Controlled Trials. Claims of effectiveness in abstract conclusion were categorized into three types: superiority (stating superiority of intervention to control), limited superiority (intervention has limited superiority), and equal efficactiveness (claiming equal effectiveness of intervention with standard treatment control), and full text results into three types: significant (all primary outcomes were statistically significant in favor of the intervention), mixed (primary outcomes included both significant and non-significant results), or all results non-significant. By comparing these classifications, we assessed whether each abstract was overstated. Our primary outcome was the proportion of overstated abstract conclusions. RESULTS: We identified and included 60 relevant trials. 20 out of 60 studies (33.3%) showed overstatements. Nine reports reported only significant results although none of their primary outcomes were significant. Large sample size (>300) and publication in high impact factor (IF>10) journals were associated with low occurrence of overstatements. CONCLUSIONS: We found that one in three psychiatry studies claiming effectiveness in their abstract conclusion, either superior to control or equal to standard treatment, for any mental disorders were overstated in comparison with the full text results. Readers of the psychiatry literature are advised to scrutinize the full text results regardless of the claims in the abstract. TRIAL REGISTRATION: University hospital Medical Information Network (UMIN) Clinical Trials Registry (UMIN000018668).

Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: a study protocol for a meta-epidemiological investigation.

INTRODUCTION: Abstracts are the major and often the most important source of information for readers of the medical literature. However, there is mounting criticism that abstracts often exaggerate the positive findings and emphasise the beneficial effects of intervention beyond the actual findings mentioned in the corresponding full texts. In order to examine the magnitude of this problem, we will introduce a systematic approach to detect overstated abstracts and to quantify the extent of their prevalence in published randomised controlled trials (RCTs) in the field of psychiatry. METHODS AND ANALYSIS: We will source RCTs published in 2014 from the Cochrane Register of Controlled Trials (CENTRAL) that claim effectiveness of any intervention for mental disorders. The abstract conclusions will be categorised into three types: superior (only stating significant superiority of intervention to control), limited (suggesting that intervention has limited superiority to control) and equal (claiming equal effectiveness of intervention as control). The full texts will also be classified as one of the following based on the primary outcome results: significant (all primary outcomes were statistically significant in favour of the intervention), mixed (primary outcomes included both significant and non-significant results) or all non-significant results. By comparing the abstract conclusion classification and that of the corresponding full text, we will assess whether each study exhibited overstatements in its abstract conclusion. ETHICS AND DISSEMINATION: This trial requires no ethical approval. We will publish our findings in a peer-reviewed journal. TRIAL REGISTRATION NUMBER: UMIN000018668; Pre-results.