Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices.

Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the 'best' decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful grant application, is associated with motivating applicants career decisions to persist (reapply for funding at T1), or to switch (not to reapply, or else leave academia). This study combined data from interviews with unsuccessful ECR applicants (n = 19) to The Wellcome Trust 2009-19, and manual coding of reviewer comments received by applicants (n = 81). All applicants received feedback on their application at T0 with a large proportion of unsuccessful applicants reapplying for funding at T1. Here, peer-review-comments-as-feedback sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. Feedback associated by unsuccessful applicants as motivating their decision to resubmit had three characteristics: actionable; targeted; and fair. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The provision of quality reviewer-reports-as-feedback to applicants, ensures that peer review acts as a participatory research governance tool focused on supporting the development of individuals and their future research plans.

Award rate inequities in biomedical research.

PURPOSE: The analysis of existing institutional research proposal databases can provide novel insights into science funding parity. The purpose of this study was to analyze the relationship between race/ethnicity and extramural research proposal and award rates across a medical school faculty and to determine whether there was evidence that researchers changed their submission strategies because of differential inequities across submission categories. METHOD: The authors performed an analysis of 14,263 biomedical research proposals with proposed start dates between 2010-2022 from the University of Michigan Medical School, measuring the proposal submission and award rates for each racial/ethnic group across 4 possible submission categories (R01 & Equivalent programs, other federal, industry, and non-profit). RESULTS: Researchers from each self-identified racial/ethnic group (Asian, Black/African American, Hispanic/Latino) pursued a different proposal submission strategy than the majority group (White). The authors found that Black/African American researchers experienced negative award rate differentials across all submission categories, which resulted in the lowest R01 & Equivalent and Other Federal submission rates of any racial/ethnic group and the highest submission rate to non-profit sources. The authors did not find support for the hypothesis that researchers changed submission strategies in response to award rate inequalities across submission categories. CONCLUSIONS: Biomedical researchers from different racial/ethnic groups follow markedly different proposal submission strategies within the University of Michigan Medical School. There is also a clear relationship between race/ethnicity and rates of proposal award. Black/African American and Asian researchers appear disadvantaged across all submission categories relative to White researchers. This study can be easily replicated by other academic research institutions, revealing opportunities for positive intervention.

AI Research Funding Portfolios and Extreme Growth.

Our work analyzes the artificial intelligence and machine learning (AI/ML) research portfolios of six large research funding organizations from the United States [National Institutes of Health (NIH) and National Science Foundation (NSF)]; Europe [European Commission (EC) and European Research Council (ERC)]; China [National Natural Science Foundation of China (NNSFC)]; and Japan [Japan Society for the Promotion of Science (JSPS)]. The data for this analysis is based on 127,000 research clusters (RCs) that are derived from 1.4 billion citation links between 104.8 million documents from four databases (Dimensions, Microsoft Academic Graph, Web of Science, and the Chinese National Knowledge Infrastructure). Of these RCs, 600 large clusters are associated with AI/ML topics, and 161 of these AI/ML RCs are expected to experience extreme growth between May 2020 and May 2023. Funding acknowledgments (in the corpus of the 104.9 million documents) are used to characterize the overall AI/ML research portfolios of each organization. NNSFC is the largest funder of AI/ML research and disproportionately funds computer vision. The EC, RC, and JSPS focus more efforts on natural language processing and robotics. The NSF and ERC are more focused on fundamental advancement of AI/ML rather than on applications. They are more likely to participate in the RCs that are expected to have extreme growth. NIH funds the largest relative share of general AI/ML research papers (meaning in areas other than computer vision, natural language processing, and robotics). We briefly describe how insights such as these could be applied to portfolio management decision-making.

A detailed open access model of the PubMed literature.

Portfolio analysis is a fundamental practice of organizational leadership and is a necessary precursor of strategic planning. Successful application requires a highly detailed model of research options. We have constructed a model, the first of its kind, that accurately characterizes these options for the biomedical literature. The model comprises over 18 million PubMed documents from 1996-2019. Document relatedness was measured using a hybrid citation analysis + text similarity approach. The resulting 606.6 million document-to-document links were used to create 28,743 document clusters and an associated visual map. Clusters are characterized using metadata (e.g., phrases, MeSH) and over 20 indicators (e.g., funding, patent activity). The map and cluster-level data are embedded in Tableau to provide an interactive model enabling in-depth exploration of a research portfolio. Two example usage cases are provided, one to identify specific research opportunities related to coronavirus, and the second to identify research strengths of a large cohort of African American and Native American researchers at the University of Michigan Medical School.

A novel approach to predicting exceptional growth in research.

The prediction of exceptional or surprising growth in research is an issue with deep roots and few practical solutions. In this study, we develop and validate a novel approach to forecasting growth in highly specific research communities. Each research community is represented by a cluster of papers. Multiple indicators were tested, and a composite indicator was created that predicts which research communities will experience exceptional growth over the next three years. The accuracy of this predictor was tested using hundreds of thousands of community-level forecasts and was found to exceed the performance benchmarks established in Intelligence Advanced Research Projects Activity's (IARPA) Foresight Using Scientific Exposition (FUSE) program in six of nine major fields in science. Furthermore, 10 of 11 disciplines within the Computing Technologies field met the benchmarks. Specific detailed forecast examples are given and evaluated, and a critical evaluation of the forecasting approach is also provided.

A standardized citation metrics author database annotated for scientific field.

Citation metrics are widely used and misused. We have created a publicly available database of 100,000 top scientists that provides standardized information on citations, h-index, coauthorship-adjusted hm-index, citations to papers in different authorship positions, and a composite indicator. Separate data are shown for career-long and single-year impact. Metrics with and without self-citations and ratio of citations to citing papers are given. Scientists are classified into 22 scientific fields and 176 subfields. Field- and subfield-specific percentiles are also provided for all scientists who have published at least five papers. Career-long data are updated to end of 2017 and to end of 2018 for comparison.

The Research Focus of Nations: Economic vs. Altruistic Motivations.

What motivates the research strategies of nations and institutions? We suggest that research primarily serves two masters-altruism and economic growth. Some nations focus more research in altruistic (or non-economic) fields while others focus more research in fields associated with economic growth. What causes this difference? Are there characteristics that would suggest why a nation is more aligned with altruism or economic growth? To answer this question, we have identified nine major fields of research by analyzing the publication activity of 4429 institutions using Scopus data. Two fields of research are clearly altruistic (there is relatively little involvement by industry) and two fields are clearly aligned with economic growth. The altruistic vs. economic nature of nations based on their publication profiles across these fields is correlated with national indicators on wealth, education, capitalism, individualism, power, religion, and language. While previous research has suggested that national research strategy is aligned with national wealth, our analysis shows that national wealth is not highly correlated with the tradeoff between altruistic and economic motives. Instead, the tradeoff is largely captured by a culture of individualism. Accordingly, implications for national research strategies are discussed.

Correction: Multiple Citation Indicators and Their Composite across Scientific Disciplines.

[This corrects the article DOI: 10.1371/journal.pbio.1002501.].

Multiple Citation Indicators and Their Composite across Scientific Disciplines.

Many fields face an increasing prevalence of multi-authorship, and this poses challenges in assessing citation metrics. Here, we explore multiple citation indicators that address total impact (number of citations, Hirsch H index [H]), co-authorship adjustment (Schreiber Hm index [Hm]), and author order (total citations to papers as single; single or first; or single, first, or last author). We demonstrate the correlation patterns between these indicators across 84,116 scientists (those among the top 30,000 for impact in a single year [2013] in at least one of these indicators) and separately across 12 scientific fields. Correlation patterns vary across these 12 fields. In physics, total citations are highly negatively correlated with indicators of co-authorship adjustment and of author order, while in other sciences the negative correlation is seen only for total citation impact and citations to papers as single author. We propose a composite score that sums standardized values of these six log-transformed indicators. Of the 1,000 top-ranked scientists with the composite score, only 322 are in the top 1,000 based on total citations. Many Nobel laureates and other extremely influential scientists rank among the top-1,000 with the composite indicator, but would rank much lower based on total citations. Conversely, many of the top 1,000 authors on total citations have had no single/first/last-authored cited paper. More Nobel laureates of 2011-2015 are among the top authors when authors are ranked by the composite score than by total citations, H index, or Hm index; 40/47 of these laureates are among the top 30,000 by at least one of the six indicators. We also explore the sensitivity of indicators to self-citation and alphabetic ordering of authors in papers across different scientific fields. Multiple indicators and their composite may give a more comprehensive picture of impact, although no citation indicator, single or composite, can be expected to select all the best scientists.

Estimates of the continuously publishing core in the scientific workforce.

BACKGROUND: The ability of a scientist to maintain a continuous stream of publication may be important, because research requires continuity of effort. However, there is no data on what proportion of scientists manages to publish each and every year over long periods of time. METHODOLOGY/PRINCIPAL FINDINGS: Using the entire Scopus database, we estimated that there are 15,153,100 publishing scientists (distinct author identifiers) in the period 1996-2011. However, only 150,608 (<1%) of them have published something in each and every year in this 16-year period (uninterrupted, continuous presence [UCP] in the literature). This small core of scientists with UCP are far more cited than others, and they account for 41.7% of all papers in the same period and 87.1% of all papers with >1000 citations in the same period. Skipping even a single year substantially affected the average citation impact. We also studied the birth and death dynamics of membership in this influential UCP core, by imputing and estimating UCP-births and UCP-deaths. We estimated that 16,877 scientists would qualify for UCP-birth in 1997 (no publication in 1996, UCP in 1997-2012) and 9,673 scientists had their UCP-death in 2010. The relative representation of authors with UCP was enriched in Medical Research, in the academic sector and in Europe/North America, while the relative representation of authors without UCP was enriched in the Social Sciences and Humanities, in industry, and in other continents. CONCLUSIONS: The proportion of the scientific workforce that maintains a continuous uninterrupted stream of publications each and every year over many years is very limited, but it accounts for the lion's share of researchers with high citation impact. This finding may have implications for the structure, stability and vulnerability of the scientific workforce.

A list of highly influential biomedical researchers, 1996-2011.

We have generated a list of highly influential biomedical researchers based on Scopus citation data from the period 1996-2011. Of the 15,153,100 author identifiers in Scopus, approximately 1% (n=149,655) have an h-index >=20. Of those, we selected 532 authors who belonged to the 400 with highest total citation count (>=25,142 citations) and/or the 400 with highest h-index (>=76). Of those, we selected the top-400 living core biomedical researchers based on a normalized score combining total citations and h-index. Another 62 authors whose focus is outside biomedicine had a normalized score that was at least as high as the score of the 400th core biomedical researcher. We provide information on the profile of these most influential authors, including the most common Medical Subject Heading terms in their articles that are also specific to their work, most common journals where they publish, number of papers with over 100 citations that they have published as first/single, last, or middle authors, and impact score adjusted for authorship positions, given that crude citation indices and authorship positions are almost totally orthogonal. We also show for each researcher the distribution of their papers across 4 main levels (basic-to-applied) of research. We discuss technical issues, limitations and caveats, comparisons against other lists of highly-cited researchers, and potential uses of this resource.

Design and update of a classification system: the UCSD map of science.

Global maps of science can be used as a reference system to chart career trajectories, the location of emerging research frontiers, or the expertise profiles of institutes or nations. This paper details data preparation, analysis, and layout performed when designing and subsequently updating the UCSD map of science and classification system. The original classification and map use 7.2 million papers and their references from Elsevier's Scopus (about 15,000 source titles, 2001-2005) and Thomson Reuters' Web of Science (WoS) Science, Social Science, Arts & Humanities Citation Indexes (about 9,000 source titles, 2001-2004)-about 16,000 unique source titles. The updated map and classification adds six years (2005-2010) of WoS data and three years (2006-2008) from Scopus to the existing category structure-increasing the number of source titles to about 25,000. To our knowledge, this is the first time that a widely used map of science was updated. A comparison of the original 5-year and the new 10-year maps and classification system show (i) an increase in the total number of journals that can be mapped by 9,409 journals (social sciences had a 80% increase, humanities a 119% increase, medical (32%) and natural science (74%)), (ii) a simplification of the map by assigning all but five highly interdisciplinary journals to exactly one discipline, (iii) a more even distribution of journals over the 554 subdisciplines and 13 disciplines when calculating the coefficient of variation, and (iv) a better reflection of journal clusters when compared with paper-level citation data. When evaluating the map with a listing of desirable features for maps of science, the updated map is shown to have higher mapping accuracy, easier understandability as fewer journals are multiply classified, and higher usability for the generation of data overlays, among others.

Clustering more than two million biomedical publications: comparing the accuracies of nine text-based similarity approaches.

BACKGROUND: We investigate the accuracy of different similarity approaches for clustering over two million biomedical documents. Clustering large sets of text documents is important for a variety of information needs and applications such as collection management and navigation, summary and analysis. The few comparisons of clustering results from different similarity approaches have focused on small literature sets and have given conflicting results. Our study was designed to seek a robust answer to the question of which similarity approach would generate the most coherent clusters of a biomedical literature set of over two million documents. METHODOLOGY: We used a corpus of 2.15 million recent (2004-2008) records from MEDLINE, and generated nine different document-document similarity matrices from information extracted from their bibliographic records, including titles, abstracts and subject headings. The nine approaches were comprised of five different analytical techniques with two data sources. The five analytical techniques are cosine similarity using term frequency-inverse document frequency vectors (tf-idf cosine), latent semantic analysis (LSA), topic modeling, and two Poisson-based language models--BM25 and PMRA (PubMed Related Articles). The two data sources were a) MeSH subject headings, and b) words from titles and abstracts. Each similarity matrix was filtered to keep the top-n highest similarities per document and then clustered using a combination of graph layout and average-link clustering. Cluster results from the nine similarity approaches were compared using (1) within-cluster textual coherence based on the Jensen-Shannon divergence, and (2) two concentration measures based on grant-to-article linkages indexed in MEDLINE. CONCLUSIONS: PubMed's own related article approach (PMRA) generated the most coherent and most concentrated cluster solution of the nine text-based similarity approaches tested, followed closely by the BM25 approach using titles and abstracts. Approaches using only MeSH subject headings were not competitive with those based on titles and abstracts.