ABSTRACT
The company Altmetric is often used to collect mentions of research in online news stories, yet there have been concerns about the quality of this data. This study investigates these concerns. Using a manual content analysis of 400 news stories as a comparison method, we analyzed the precision and recall with which Altmetric identified mentions of research in 8 news outlets. We also used logistic regression to identify the characteristics of research mentions that influence their likelihood of being successfully identified. We find that, for a predefined set of outlets, Altmetric's news mention data were relatively accurate (F-score = 0.80), with very high precision (0.95) and acceptable recall (0.70), although recall is below 0.50 for some news outlets. Altmetric is more likely to successfully identify mentions of research that include a hyperlink to the research item, an author name, and/or the title of a publication venue. This data source appears to be less reliable for mentions of research that provide little or no bibliometric information, as well as for identifying mentions of scholarly monographs, conference presentations, dissertations, and non-English research articles. Our findings suggest that, with caveats, scholars can use Altmetric news mention data as a relatively reliable source to identify research mentions across a range of outlets with high precision and acceptable recall, offering scholars the potential to conserve resources during data collection. Our study does not, however, offer an assessment of completeness or accuracy of Altmetric news data overall. Supplementary Information: The online version contains supplementary material available at 10.1007/s11192-022-04510-7.
ABSTRACT
OBJECTIVE: To characterise how online media coverage of journal articles on cancer funded by the US government varies by cancer type and stage of the cancer control continuum and to compare the disease prevalence rates with the amount of funded research published for each cancer type and with the amount of media attention each receives. DESIGN: A cross-sectional study. SETTING: The United States. PARTICIPANTS: The subject of analysis was 11 436 journal articles on cancer funded by the US government published in 2016. These articles were identified via PubMed and characterised as receiving online media attention based on data provided by Altmetric. RESULTS: 16.8% (n=1925) of articles published on US government-funded research were covered in the media. Published journal articles addressed all common cancers. Frequency of journal articles differed substantially across the common cancers, with breast cancer (n=1284), lung cancer (n=630) and prostate cancer (n=586) being the subject of the most journal articles. Roughly one-fifth to one-fourth of journal articles within each cancer category received online media attention. Media mentions were disproportionate to actual burden of each cancer type (ie, incidence and mortality), with breast cancer articles receiving the most media mentions. Scientific articles also covered the stages of the cancer continuum to varying degrees. Across the 13 most common cancer types, 4.4% (n=206) of articles focused on prevention and control, 11.7% (n=550) on diagnosis and 10.7% (n=502) on therapy. CONCLUSIONS: Findings revealed a mismatch between prevalent cancers and cancers highlighted in online media. Further, journal articles on cancer control and prevention received less media attention than other cancer continuum stages. Media mentions were not proportional to actual public cancer burden nor volume of scientific publications in each cancer category. Results highlight a need for continued research on the role of media, especially online media, in research dissemination.
