As we recognise Peer Review Week 2025 (15-19 September), Head of Editorial Matt Hodgkinson speaks to this year’s theme, ‘Rethinking Peer Review in the AI Era’ and how artificial intelligence is beginning to reshape peer review and the wider scholarly publishing landscape.
There’s something about automated tools, and in particular generative artificial intelligence (AI), that makes people turn off their critical thinking. When ChatGPT was first released in late 2022, there was a flurry of scholarly articles published with the tool listed as an author despite tools clearly lacking the ability to take responsibility for any of their outputs. Thankfully, sanity was restored and a rapid consensus formed that automated tools cannot be authors.
However, people still keep finding new ways to misuse tools. Earlier this year, DOAJ rejected an application from a journal that uses AI to select its peer reviewers. This wasn’t the only concern we identified, but it stood out as a new concern in publication ethics. The use of automated algorithms to suggest potential peer reviewers dates back many years, both within publisher databases and using external sources such as Publons (now part of Web of Science). However, what struck us as different about this journal was that the proprietary algorithm used was not described in any way, making it a ‘black box’, doubly important in light of the other factor: there is no human oversight of the reviewer invites.
Our reluctance to welcome our robot overlords is supported by a recent study, which found that “AI shows promise in enhancing reviewer selection efficiency and broadening the reviewer pool, it requires human oversight to address limitations in understanding nuanced disciplinary contexts”. The particular AI tool tested in that research, based on GPT-4, performed worse in the arts, humanities, and social sciences than in STEM, and even suggested some fictional reviewers.
Validating peer reviewer suggestions is hard: there is no gold standard of how to match a researcher’s expertise to an article. Agreement rates might help, but many off-topic reviewers agree out of interest and naivety. Author ratings might favour more lenient reviewers. Still, any automated tool for matching people to papers needs to be thoroughly tested, declared, described, and monitored. Tools need to be able to be adjusted if reviewers complain that they receive off-topic invites. Editors need to check whether the suggestions are appropriate. I have seen several journal editors admit on social media, without apparent shame, that they invite dozens of reviewers suggested by automated tools at a time – and they respond with surprise to the suggestion that they should screen them. How could one manually check so many invites? Here is the point: one would not check so many, one would check fewer. The more the merrier does not apply to peer review invites when so many academics are already suffering ‘reviewer fatigue’. Just as spamming out your CV/resumé does not get you many job interviews, clicking ‘Invite’ on an automatically generated list of names is not being a diligent editor.
In light of these and other concerns, at the end of the month we are updating our guide to applying to add a section on artificial intelligence and other automated tools. Briefly, we will require journals to have a policy on their use, which should include:
- Authors must disclose use besides spelling, grammar checking, or the like;
- Authors must take responsibility for the output of tools;
- Tools cannot be authors;
- Generative AI must not be cited;
- Reviewers should not use generative AI to write their reports.
- The journal should disclose its use of tools, validate them, and have people check their results.
These are not new suggestions: COPE, the STM Association, WAME, and others have got there before us and inspired many of these requirements. We hope these updates capture the current consensus in journal publishing and offer guardrails without overly restricting experimentation with automation.
Peer Review Week 15-19 September 2025 – https://peerreviewweek.net/theme.php
