Is ChatGPT a good popular science disseminator in cosmetology? A linguistic study on popular science texts

Abstract

The science popularisation texts are fundamental for disseminating scientific knowledge in an accessible and understandable way to a non-specialised audience and have a different structure and characteristics from scientific articles (e.g. Garces-Conejos & Sanchez-Macarro, 1998; Zamboni, 1998). Studies on the linguistic properties of science popularisation texts in European Portuguese are not abundant, the exception being the Promoção da Literacia Científica project (Gonçalves & Jorge, 2018). On the other hand, within the realm of producing content, the large language models (LLMs), namely OpenAI's GPT models, have gained widespread public attention in a short period of time. Since they are recent, there is still very little assessment of the linguistic quality of the texts produced. Bearing these premises in mind, the aim of this study is to assess the linguistic quality of the responses generated by ChatGPT (GPT-3.5) in the field of cosmetology, with regard to cosmetic products, ingredients, safety and efficacy and regulation categories, with the objective of identifying patterns that allow an understanding of the differences and/or similarities between the content generated by LLM and that produced by human experts on the Portal infoCosméticos. For this, twenty questions previously answered and published on the portal were selected and subsequently four different prompts with different degrees of complexity were created, which resulted in eighty answers generated by ChatGPT. The answers were then analysed according to the results of a linguistic evaluation grid consisting of 11 questions. The analysis produced different types of results: overall, the answers written by the experts produced slightly better results than those from ChatGPT; in terms of interphrasal cohesion, it was found that the texts produced by the experts use a reduced number of connectors, in contrast to the recurrent use of discourse markers in the ChatGPT texts; there is the use of non-explained scientific jargon and a macrostructure with the absence of a conclusion paragraph in the texts published on the portal; the texts generated by ChatGPT have a high frequency of repetitions and/or tautologies.