Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study.

BACKGROUND: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. OBJECTIVE: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. METHODS: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. RESULTS: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. CONCLUSIONS: ChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.

Literature citation and writer identity construction in linguistic academic discourse.

This study revealed the distributional features of literature citation and writer identity in linguistic academic discourse, and further explored how writer identity is constructed through literature citation. Based on Petric's citation typology and Ivanic's writer identity framework, the study investigated various types of citations and identities in thirty journal articles published in the Journal of English for Academic Purposes from 2017 to 2021. The results showed that there are a total of 1637 citation instances, among which the dominant type is attribution citation (36.59 %). Besides, the study revealed that 80.45 % of the total citations construct discoursal self only and 19.55 % reveal both discoursal self and authorial self. In terms of discoursal self, writers present themselves as a member of academic community, a contributor to a field of knowledge, a knowledgeable and professional scholar, and a reliable and credible writer. As for authorial self, they would like to position themselves to be a writer with authority and an evaluator. It is expected that the current study can help novice writers use citations strategically and establish their desired identity accordingly.

Dynamics and landscape of academic discourse on environmental attitudes and behaviors since the 1970s.

Introduction: This study addresses the lack of systematic review and analysis of the academic discourse on environmental attitudes and behaviors. Despite the wealth of knowledge published in academic journals, there is a need to understand the order and content of this discourse, including the employed theoretical approaches and empirical evidence. Methods: A combination of systematic literature review and quantitative content analysis methods was employed. Articles for analysis were identified through Web of Science and SCOPUS, followed by a detailed analysis of 200 papers from the journal Environment and Behavior. The study aimed to explore the historical stages, theoretical diversity, and the empirical evidence brought forward in the academic discourse on environmental attitudes and behaviors. Results: The findings reveal distinct historical stages within the academic field of environmental attitudes and behaviors. There is a notable growth in theoretical diversity and intensity of the discourse, particularly after 2000. The dominance of socio-psychological explanatory models is evident. Furthermore, the empirical evidence base is geographically limited, mostly coming from the US. Discussion: The study discusses the limitations of the academic discourse on environmental attitudes and behaviors and provides guidelines for future research. It emphasizes the need to address the identified shortcomings, such as expanding theoretical perspectives and increasing the geographical diversity of empirical evidence. The study's findings contribute to understanding the development and characteristics of the academic field, while also identifying avenues for further exploration.

AI ethics with Chinese characteristics? Concerns and preferred solutions in Chinese academia.

Since Chinese scholars are playing an increasingly important role in shaping the national landscape of discussion on AI ethics, understanding their ethical concerns and preferred solutions is essential for global cooperation on governance of AI. This article, therefore, provides the first elaborated analysis on the discourse on AI ethics in Chinese academia, via a systematic literature review. This article has three main objectives. (1) to identify the most discussed ethical issues of AI in Chinese academia and those being left out (the question of "what"); (2) to analyze the solutions proposed and preferred by Chinese scholars (the question of "how"); and (3) to map out whose voices are dominating and whose are in the marginal (the question of "who"). Findings suggest that in terms of short-term implications, Chinese scholars' concerns over AI resemble predominantly the content of international ethical guidelines. Yet in terms of long-term implications, there are some significant differences needed to be further addressed in a cultural context. Further, among a wide range of solution proposals, Chinese scholars seem to prefer strong-binding regulations to those weak ethical guidelines. In addition, this article also found that the Chinese academic discourse was dominated by male scholars and those who are from elite universities, which arguably is not a unique phenomenon in China. Supplementary Information: The online version contains supplementary material available at 10.1007/s00146-022-01578-w.

A corpus-based quantitative study on the interpersonal functions of hedges in Chinese and German academic discourse.

Hedge is one of the most essential ways of accomplishing interpersonal function. With its help, the speakers are able to say as much information as they could in the fewest possible words and better promote the interpersonal interactionChen & Li., 1994 while communicating, such as expressing politeness, providing more opportunities for further discussion, avoiding subjectivity, and boosting credibility. Therefore, hedge has become a commonly used interpersonal strategy for researchers. In this study, the corpus consists of 170 Chinese and 170 German academic articles from renowned linguistic empirical studies that were published between 2010 and 2019. Based on the theoretical research of prior studies, the interpersonal function of hedges and the similarities/differences of its pragmatic realization between Chinese and German academic written discourses are investigated from a quantitative perspective. It is concluded that the Chinese researchers are more cautious in using strong-modality-expressing-hedges. They communicate with readers indirectly and implicitly, focus on avoiding their own responsibilities, demonstrate a conservative and cautious attitude, and provide more room for further discussion, while the German researchers are considered more active in using those types of hedging strategies, which make them more direct while interacting with readers, concentrating on emphasizing the correctness of their own academic views and highlighting their positive attitude towards their own opinions.

What Still Needs to be Noted: Pseudo-Clefts in the Academic Discourse of Applied Linguistics.

Pseudo-clefts are the building blocks of coherent discourse progression and serve as a rhetorical toolkit to construct an authorial stance in the academic discourse. Despite an increasing interest in grammatical constructions in the academic discourse, researchers have not treated pseudo-clefts in much detail. This paper explores the features of pseudo-clefts in the corpus of academic discourse in the field of applied linguistics. Here, we take the textual and the interpersonal perspectives, focusing on the use of pseudo-clefts in terms of their distribution in generic structure, discourse functions with reference to clefted constituents, and evaluative meaning. The results show that pseudo-clefts were more frequently used in "Results and Discussion" and Literature Review, performing the functions such as the specification of key terms, generalization of the literature, the description and explanation of findings, etc. They are facilitative in creating information gaps and establishing a logic-semantic expansive relationship between the clauses. The findings also suggest that the pseudo-clefts are evaluative devices and are involved in the construction of authorial identities.

Food security and food insecurity in Europe: An analysis of the academic discourse (1975-2013).

In this paper we address the academic discourse on food insecurity and food security in Europe as expressed in articles published in scientific journals in the period 1975 to 2013. The analysis indicates that little knowledge has been produced on this subject, and that the limited research that has been produced tends to focus on the production of food rather than on people's access to food. The lack of knowledge about European food insecurity is particularly alarming in these times, which are characterised by increasing social inequalities and poverty, as well as shifting policy regimes. More empirical, comparative and longitudinal research is needed to survey the extent of food security problems across European countries over time. There is also a need to identify groups at risk of food insecurity as well as legal, economic, practical, social, and psychological constraints hindering access to appropriate and sufficient food.

História, nação e raça no contexto da Exposição do Centenário em 1922 / History, nation and race in the context of the Centennial Exhibition of 1922

Tem como tema os debates públicos e acadêmicos sobre a história do Brasil que ocorreram no contexto da Exposição do Centenário, em 1922, na cidade do Rio de Janeiro. Pretende-se destacar a maneira como certos conceitos sobre ‘nação’ e ‘raça’ foram debatidos em conferências acadêmicas sobre história, antropologia e arqueologia, como foram conectados à história nacional e como se tornaram, finalmente, parte da autoimagem nacional. Além disso, o artigo indaga como e por quem esses ideais foram contestados.

The subjects addressed in this article are the public and academic debates about the history of Brazil that took place in the context of the Centennial Exhibition held in 1922 in the city of Rio de Janeiro. The aim is to highlight how certain concepts of nation and race were debated in academic discussions about history, anthropology and archaeology, how they were related to the nation’s history, and how they ultimately became a part of the nation’s self-image. Furthermore, the article investigates how and by whom these ideals were contested.

Some linguistic and pragmatic considerations affecting science reporting in English by non-native speakers of the language.

Approximately 50% of publications in English peer reviewed journals are contributed by non-native speakers (NNS) of the language. Basic thought processes are considered to be universal yet there are differences in thought patterns and particularly in discourse management of writers with different linguistic and cultural backgrounds. The study highlights some areas of potential incompatibility in native and NNS processing of English scientific papers. Principles and conventions in generating academic discourse are considered in terms of frequently occurring failures of NNS to meet expectations of editors, reviewers, and readers. Major problem areas concern organization and flow of information, principles of cohesion and clarity, cultural constraints, especially those of politeness and negotiability of ideas, and the complicated area of English modality pragmatics. The aim of the paper is to sensitize NN authors of English academic reports to problem areas of discourse processing which are stumbling blocks, often affecting acceptance of manuscripts. The problems discussed are essential for acquiring pragmalinguistic and sociocultural competence in producing effective communication.