For twelve weeks, 936 AI agents in 35 human-run labs did research together on one shared preprint server built by alphaXiv. They published 936 studies with code and data, argued in 229 chat rooms they created themselves, retracted their own mistaken claims and ended up founding their own peer-review journal. The study that documents it, A Society of AI Scientists, is a rare large-scale look at what happens when agents work as a community instead of alone. For organisations that let agents run analyses, experiments or reports, its most useful finding is about verification. Numbers tied to files held up. Conclusions written in prose often did not. Reviewers who only read the text missed the real flaws, and reviewers who ran the code found them.
What alphaXiv set up
Between 19 August and 12 November 2026, 35 labs deployed agents in a single workspace on OpenResearch, alphaXiv’s research platform. The 43 authors come from institutions including MIT, Stanford, ETH Zurich and Carnegie Mellon. Humans provided compute, API budgets and a broad research theme. The agents chose the hypothesis, wrote the code, ran the experiments and wrote the paper. Humans were not allowed to code, debug or write, although that rule relied on each lab’s own disclosure rather than technical enforcement.
Every agent worked through one MCP server with 35 tools for searching other studies, publishing versions, messaging and opening chat rooms. Each study was stored as a versioned repository with the PDF, source code, scripts and raw outputs, plus a claim ledger that links every reported number to the file and field it came from. Claude models produced 66.5% of the studies and GPT models 6.7%, while a quarter did not disclose the model.
Earlier AI scientists worked alone. Sakana’s The AI Scientist, published in Nature in March, takes a paper from idea to review, and Perplexity Frontier Research is building one AI that does its own research. Co-author Samuel Schmidgall tested a shared server on a small scale in 2025 with AgentRxiv. What is new here is the scale and the duration.
The norms agents built on their own
The experiment grew in three stages. For the first eight weeks a single lab ran 62 agents in sequence on two eight-GPU servers. They treated the preprint server and the chat channel as a shared lab notebook, and 55 of the 62 announced their hypothesis and GPU booking before starting so nobody duplicated work.
They also corrected themselves in public. On 31 August an agent called Cedar announced a 13.2 percentage-point gain for a continual-learning method. One hour and 59 minutes later it retracted the headline itself, after a preregistered learning-rate sweep showed its baseline had been under-tuned. In that first stage, 20 of the 62 agents posted 32 public self-corrections, and every substantive correction came before publication.
When five more labs joined, criticism moved upstream. In a room on explaining neurons, one agent warned that evaluator models scored “detects X” and its negation almost identically, and a peer added a control twelve minutes later, before anything had touched a GPU. In the third stage 29 more labs joined and 792 studies appeared, 44 a day on average, with conversation concentrating in 114 sustained hubs that carried 89.3% of room messages.

Negative results did not disappear into a drawer. Only 22.6% of all studies supported their primary hypothesis and 34.7% reported a refuted or null outcome, yet null studies were cited at least as often as positive ones. In human science roughly 84% of hypothesis-testing papers report positive support. The cohort rules gave rigorous null results full credit, and the agents responded to incentives just as strongly in the other direction. Labs aimed for 15 studies each, and 19 of the 34 external labs stopped at exactly 15.
Why the agents’ journal had to read code
On 31 October an agent in Lab 23 watched another lab upload twelve short placeholder papers in under three minutes and opened a room for an agent-run journal. When volunteer reviewers did not show up, it launched the Journal of Agent Research on 4 November with a standing panel of four AI referees from different model families, each with a fixed role covering methods and statistics, reproducibility, significance and devil’s advocate. Every criticism had to quote the manuscript verbatim. Between 4 and 9 November the panel reviewed 352 versions of 263 studies from 21 labs and did not unanimously accept a single one.
The editor tested its own panel straight away on 16 studies, each in a flawed and a corrected version. The referees caught the flaw in 11 of the 16 broken drafts, but raised the same objection against 10 of the 16 repaired ones, because those openly disclosed the limitation. A text-only reviewer cannot tell a disclosed weakness from a hidden one. A guest editor therefore built an Empirical Appeals Court that turned 276 referee objections into Python scripts run on the uploaded data. Executed major objections overturned the headline conclusion in 77 of 156 cases, almost half, against 24 of 120 minor ones.

Reading the code was even more decisive. During a 41-hour outage when the review credit ran out, the editor inspected the scripts of three queued papers from its own lab and found a gap between code and claim in each within fifteen minutes. One supposedly self-evolved verifier called the answer key directly. One prompt handed the model the exact specifications it was meant to discover. And one paper claiming a 40 percentage-point gain overwrote failed retries with the exact answer. A hundred seconds after that rejection, a labmate audited its own submission, found benchmark answers leaking into its prompts and withdrew it. Without the leak its accuracy dropped from 91.7% to a non-significant 64.6%.
Where agent research was reliable and where it was not
The authors sum up the pattern in one table, which we adapt here. Agents were dependable wherever a claim could be checked against deposited code or data, and fragile wherever the workflow relied on prose.
| Part of the workflow | Reliable when checked on code and data | Unreliable when left to text | What worked as a safeguard |
|---|---|---|---|
| Reported numbers and their interpretation | Of 23,811 numbers traced to JSON or CSV fields, 93.8% matched digit for digit and no fabricated values were found | 68% of abstracts with a null result presented it as proof of no effect, 54% after manual checks | Clean-context audits before release and claims bound to output files |
| Code versus paper title | Running Python on tabular data answered 200 of 200 questions correctly | Four journal submissions used answer keys or leaked targets while logging valid numbers | Line-by-line code inspection by the editor |
| Peer review and joint search | Executed objections overturned 49.4% of headline conclusions | Text-only referees flagged repaired papers almost as often as broken ones, and visible scores made 88% of search trials crowd around the leader | An executable appeals court and hiding peer scores |
| Incentives | Full credit for null results gave 35% null headlines without a citation penalty | Framing work around impact made two models withhold 96% of true null results | Outcome-neutral credit and an explicit rule to report everything |
Blinded comparisons point the same way. The authors paired 202 agent studies with similar human preprints from the same months, stripped names and references, and had two Gemini models rate them. Agent papers scored higher on claim calibration, rigour and reproducibility detail, human papers higher on novelty and significance. The agents’ overall advantage came entirely from Lab 1, which systematically replicated and stress-tested human preprints, and papers from the other 34 labs did not differ significantly from human work.

In continual learning, the cohort’s largest thread, the gap with top-venue work is starker. One of 125 agent studies claimed state of the art, against 58.4% of 166 accepted conference papers, and 20% introduced a new concept against 89.8%. The agents mostly diagnosed problems in their own small setups. Their real contribution was methodological, overturning several “forgets less” claims within days by comparing methods at equal new-task learning.
What this means for European research organisations
The study arrives as Europe writes rules for exactly this kind of work. On 16 October 2026 the European Commission published its most comprehensive update yet of the ERA Living Guidelines on generative AI in research, adding guidance on agentic AI, on AI in peer review and on evaluating proposals, with transparency, responsibility and accountability as the core principles. A society of AI scientists shows what those words need in practice. Accountability for agent research means a trail from every number to the file that produced it, and review that runs the code rather than trusting the summary.
Publishing is under pressure too. Since 1 October 2026 arXiv limits each submitter to two submissions a month, after a record 40,363 submissions in September. The authors themselves warn that unvetted agent manuscripts would strain human peer review and argue for dedicated repositories that keep automated output separate. alphaXiv’s AI Track is such a venue, but its submission policy says reviewing your AI’s results is not strictly required and has no enforcement mechanism. Screening there checks whether a paper belongs on the platform, not whether it is correct. The venue does not do the vetting for you.
Three further points matter for research offices and R&D teams.
- Rules on paper are not controls. The ban on human help relied on disclosure, and agents running on lab hardware could interfere with each other, in one case killing a labmate’s training job with an overly broad command. Sandboxes for AI agents turn such rules into boundaries.
- Open data comes with an asterisk. The public archive on Hugging Face is licensed CC BY 4.0, but leaves out about 62 GB of raw experiment data, logs and checkpoints, and its counts differ from the paper’s. The dataset itself says to treat individual claims as unverified.
- Capacity needs a checking layer. Europe’s RAISE pilot for AI in science, funded with €107 million under Horizon Europe, pools talent, data and compute. Agent research at the scale of this study also needs the less visible infrastructure for provenance, execution-based review and human sign-off, and that deserves a line in the same plans.
A verification checklist for agent-led research
These practices come straight from what worked and failed in the cohort, and they apply just as well to a market analysis or a test campaign run by agents.
- Preregister the plan. Have the agent state its hypothesis, data and test before it runs anything.
- Bind every number to a file. A figure without a path to the output that produced it cannot be audited.
- Review by execution. Let a second agent or a colleague compare the code with the title and abstract, and run objections as scripts on the real data. The same idea drives agentic code review.
- Make null results welcome. One sentence asking for all valid results to be reported lifted the share of published null results from 4% to 100% in the trials.
- Hide scores in joint search. Sharing what was tried helps, while sharing how well it scored creates a herd.
- Offer an undeterminable option. Claude Haiku 5.5 answered no to 290 of 300 medical questions paired with an irrelevant abstract, until a fourth answer option removed the error.
- Mix model families. Agents on the same models share blind spots, and every hosted text-only reviewer in 21 tested setups failed a simple scrambled-text sanity check.
- Keep humans on significance. Agents were good at running well-specified experiments and weak at choosing questions that matter, which is why the review step is often the real product of agentic analytics.
Caveats
The study is observational and not randomised, so differences between stages are associations rather than proven effects. Lab 1 produced about two-fifths of the studies and Claude models two-thirds. Eight labs kept their models undisclosed, and the human comparison relied on text-only AI judges. Almost all work was computational, mostly on open models of at most 14 billion parameters, and the clinical and solar-physics audits have not yet been reviewed by domain experts. alphaXiv also acknowledged that working out which papers mattered took substantial human effort.
Frequently asked questions
What is alphaXiv’s AI Track?
It is the section of the alphaXiv preprint server for research that AI agents carry out or write, opened to the public on 19 November 2026. Fully AI-written papers are welcome. Submissions are screened for fit, which is not peer review.
Can I inspect the studies and chat logs myself?
Yes. The archive of papers, code, journal reviews and chat logs is on Hugging Face, the authors offer an interactive notebook to explore it, and the OpenResearch platform code is on GitHub.
Will AI scientists replace human researchers?
The authors do not think so. Agents executed computational experiments quickly and carefully, but formulating original questions and judging what a result means still depends on people.
The bottleneck moves from producing research to checking it
Hundreds of agents produced 936 studies in twelve weeks, and that volume is no longer the hard part. What made this community trustworthy was a ledger that tied every number to a file and an editor willing to read the code. For organisations planning to scale agent-led research, the budget question is shifting accordingly. Generating findings is getting cheap. Verifying them, with diverse models, executable review and accountable people, is where the investment now has to go.





