How do AI tools and Large Language models fit into the future of Peer Review?
by adya misra
It would be tough to imagine discussing the future of peer review without discussing large language models (LLMs) or AI tools. To inspire this piece, I couldn’t help but ask ChatGPT about how it perceived its own capabilities in peer review.
“How do you see ChatGPT helping academic researchers with peer review?”
ChatGPT was quick to rattle off a long list of editorial and related tasks it could assist with, some very interesting and some probably unlikely to be adopted by journal editors. I was pleasantly surprised to see the following at the end of its list:
“However, it's crucial to emphasize that ChatGPT should not replace human expertise and critical judgment. Its role should be supportive and complementary to human reviewers, helping streamline and enhance the peer review process rather than replacing the vital role of human evaluation.”
Fast improving technologies associated with artificial intelligence and large language models are creating opportunities for innovation and efficiency. Thus far, it has created polarising debates on whether these tools are glorified chatbots or paradigm-shifting technology.
“Should they be used to do data-analyses?”
“Can we use these tools to improve language?”
“Is it ethical to generate images with AI tools?”
We received some of these queries in our recent “How to be a Peer Reviewer Webinar on Research Integrity and Ethics” (Watch the Chinese-Language Webinar here). Earlier this year we saw ChatGPT listed as an author on some publications. There were several opinion pieces and editorials published on the use or misuse of large language models and how. and we continue to look at technology to detect computer-generated text.
It remains to be seen how we may use these technologies to complement our current workflows in scholarly publishing, but there are several reasons to be cautious. Reviewers, editors, and publishers have rightly been concerned about the potential misuse of technology to generate unreliable content in the literature that may pass filters of peer review. I created brief guidance for our editors and reviewers on how they should approach the potential use of these tools in submissions. Despite what ChatGPT may believe about its capabilities, we believe that it is not an appropriate tool to assist with peer review due to issues around confidentiality and similarly may be unsuitable for summarizing reviewer comments for decision letters.
Sage policy on AI tools and LLMs is straightforward in that we ask that authors declare their use in the appropriate section (s) of the manuscript. We have already seen several published articles with ChatGPT hallmarks such as “regenerate response” or “As an AI model” with the undeclared use of assisted technology. The reality is that authors may be wary of declaring the use of these tools if they believe that the editor is likely to reject their submission based on this declaration. As we learn more about these technologies, how they change researcher behaviour, and how they affect publishing, we will continue to adapt our guidance and policies in this area to ensure we are meeting the needs of our communities.
Sage is a proud partner and contributor to Peer Review Week. Browse more content here.
About the Author
