A key science publishing platform is cracking down on AI slop

The pre-print website arXiv has announced that researchers who put their names to papers which included errors clearly generated by artificial intelligence (AI) will face a year-long ban and ongoing restrictions.

The move is a response to a growing influx of AI-generated papers faced by scholarly journals as well as sites such as arXiv, which serve as unofficial platforms for research publication ahead of peer review.

However, not everyone agrees that arXiv’s response to the problem is appropriate – and the solution to the flood of AI slop research may involve more AI, not less.

The rise of bot-assisted writing

AI-generated text is on the rise everywhere. A study released last week suggests half of new articles published online are now “primarily AI-generated”.

Science is not immune to this trend. Last month, the journal Organization Science published a study of how the rise of AI has affected submissions and peer reviews since the release of ChatGPT in 2022. Reporting a dramatic rise in submitted papers and a drop in quality, the authors conclude that “the current state of AI tools, amplified by existing publish-or-perish incentives, appears to be pushing the system toward an equilibrium of more rather than better research”.

A common problem in AI-generated research writing is hallucinated citations: references to other research that does not exist.

The traditional safeguard against poor quality in scholarly publishing is peer review: another expert in the subject at hand reads the research paper and interrogates the work behind it before it can be published.

However, the peer review system was already struggling before AI. Pressured researchers often have little time or incentive to do the unpaid work of peer review.

And on arXiv, which publishes preprints – articles which have most often not been peer-reviewed – even this system is not available. Last year, flooded with AI-generated submissions, the site stopped accepting certain types of article.

A study published in January (itself a preprint) estimated around 1 in 8 papers in biomedical science now contain AI-generated text.

Most researchers would agree that AI-generated text is not a problem in itself. The problem is the lower-quality work that AI can make easy to produce.

Does the punishment fit the crime?

The ArXiv announcement doesn’t come out against AI use, but rather says

If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation, this means we can’t trust anything in the paper.

This may be true as far as it goes. But the penalty – a year-long ban for all authors listed on a paper – may be out of keeping with current research practices.

In the past, research was often carried out by people working alone or in groups of two or three. In these circumstances, it seems reasonable to expect each author to take responsibility for the whole.

But research is now more collaborative than ever before. Many papers have four or five authors, and in a growing number of extreme cases papers may be credited to groups of hundreds of scientists working together, each working on their own speciality and trusting their colleagues to be doing the same.

In a case where one author of dozens or hundreds included an AI-hallucinated reference in their part of the paper, banning the lot seems harsh.

And there are no equivalent sanctions for publishing other problematic material. There’s no ban for pushing fringe or discredited theories, or using poor quality evidence and illogical arguments, for example.

Can AI help fight slop?

The rise of AI produces problems for publishers and quality assurance. And the idea of some kind of sanctions for reckless use of AI, such as included hallucinated references, is a good one.

But ArXiv’s particular choice seems drastic. If the goal is to improve peer review and quality assurance, AI systems themselves can play a role.

Modern AI systems are quite capable of taking a list of references and checking everything on it is a real paper available on the internet. Any references flagged as suspect can then be checked by a human.

AI can even be useful for carrying out quick sense-checks of things like a paper’s statistical analysis.

Perhaps this is the way forward, rather than harsh sanctions for relatively minor AI-related infractions.