For twelve weeks, 936 AI agents in 35 human-run labs did research together on one shared preprint server built by alphaXiv. They published 936 studies with code and data, argued in 229 chat rooms they created themselves, retracted their own mistaken claims and ended up founding their own peer-review journal. The study that documents it, A Society of AI Scientists, is a rare large-scale look at what happens when agents work as a community instead of alone. For organisations that let agents run analyses, experiments or reports, its most useful finding is about verification. Numbers tied to files held up. Conclusions written in prose often did not. Reviewers who only read the text missed the real flaws, and reviewers who ran the code found them.

What alphaXiv set up

Between 19 August and 12 November 2026, 35 labs deployed agents in a single workspace on OpenResearch, alphaXiv’s research platform. The 43 authors come from institutions including MIT, Stanford, ETH Zurich and Carnegie Mellon. Humans provided compute, API budgets and a broad research theme. The agents chose the hypothesis, wrote the code, ran the experiments and wrote the paper. Humans were not allowed to code, debug or write, although that rule relied on each lab’s own disclosure rather than technical enforcement.

Every agent worked through one MCP server with 35 tools for searching other studies, publishing versions, messaging and opening chat rooms. Each study was stored as a versioned repository with the PDF, source code, scripts and raw outputs, plus a claim ledger that links every reported number to the file and field it came from. Claude models produced 66.5% of the studies and GPT models 6.7%, while a quarter did not disclose the model.

Earlier AI scientists worked alone. Sakana’s The AI Scientist, published in Nature in March, takes a paper from idea to review, and Perplexity Frontier Research is building one AI that does its own research. Co-author Samuel Schmidgall tested a shared server on a small scale in 2025 with AgentRxiv. What is new here is the scale and the duration.

The norms agents built on their own

The experiment grew in three stages. For the first eight weeks a single lab ran 62 agents in sequence on two eight-GPU servers. They treated the preprint server and the chat channel as a shared lab notebook, and 55 of the 62 announced their hypothesis and GPU booking before starting so nobody duplicated work.

They also corrected themselves in public. On 31 August an agent called Cedar announced a 13.2 percentage-point gain for a continual-learning method. One hour and 59 minutes later it retracted the headline itself, after a preregistered learning-rate sweep showed its baseline had been under-tuned. In that first stage, 20 of the 62 agents posted 32 public self-corrections, and every substantive correction came before publication.

When five more labs joined, criticism moved upstream. In a room on explaining neurons, one agent warned that evaluator models scored “detects X” and its negation almost identically, and a peer added a control twelve minutes later, before anything had touched a GPU. In the third stage 29 more labs joined and 792 studies appeared, 44 a day on average, with conversation concentrating in 114 sustained hubs that carried 89.3% of room messages.

Timeline from 19 August to 12 November in which labs grow from 1 to 6 to 35 and published studies reach 936
Each stage added a new habit, from public self-correction to an agent-run journal. Source: A Society of AI Scientists, table 1.

Negative results did not disappear into a drawer. Only 22.6% of all studies supported their primary hypothesis and 34.7% reported a refuted or null outcome, yet null studies were cited at least as often as positive ones. In human science roughly 84% of hypothesis-testing papers report positive support. The cohort rules gave rigorous null results full credit, and the agents responded to incentives just as strongly in the other direction. Labs aimed for 15 studies each, and 19 of the 34 external labs stopped at exactly 15.

Why the agents’ journal had to read code

On 31 October an agent in Lab 23 watched another lab upload twelve short placeholder papers in under three minutes and opened a room for an agent-run journal. When volunteer reviewers did not show up, it launched the Journal of Agent Research on 4 November with a standing panel of four AI referees from different model families, each with a fixed role covering methods and statistics, reproducibility, significance and devil’s advocate. Every criticism had to quote the manuscript verbatim. Between 4 and 9 November the panel reviewed 352 versions of 263 studies from 21 labs and did not unanimously accept a single one.

The editor tested its own panel straight away on 16 studies, each in a flawed and a corrected version. The referees caught the flaw in 11 of the 16 broken drafts, but raised the same objection against 10 of the 16 repaired ones, because those openly disclosed the limitation. A text-only reviewer cannot tell a disclosed weakness from a hidden one. A guest editor therefore built an Empirical Appeals Court that turned 276 referee objections into Python scripts run on the uploaded data. Executed major objections overturned the headline conclusion in 77 of 156 cases, almost half, against 24 of 120 minor ones.

Comparison of text-only referees flagging 11 and 10 of 16 papers with executed objections overturning 77 of 156 headline conclusions
Text-only referees objected almost as often to repaired papers as to broken ones. Executed objections hit the core in half of the cases. Source: A Society of AI Scientists, figure 11.

Reading the code was even more decisive. During a 41-hour outage when the review credit ran out, the editor inspected the scripts of three queued papers from its own lab and found a gap between code and claim in each within fifteen minutes. One supposedly self-evolved verifier called the answer key directly. One prompt handed the model the exact specifications it was meant to discover. And one paper claiming a 40 percentage-point gain overwrote failed retries with the exact answer. A hundred seconds after that rejection, a labmate audited its own submission, found benchmark answers leaking into its prompts and withdrew it. Without the leak its accuracy dropped from 91.7% to a non-significant 64.6%.

Where agent research was reliable and where it was not

The authors sum up the pattern in one table, which we adapt here. Agents were dependable wherever a claim could be checked against deposited code or data, and fragile wherever the workflow relied on prose.

Part of the workflow Reliable when checked on code and data Unreliable when left to text What worked as a safeguard
Reported numbers and their interpretation Of 23,811 numbers traced to JSON or CSV fields, 93.8% matched digit for digit and no fabricated values were found 68% of abstracts with a null result presented it as proof of no effect, 54% after manual checks Clean-context audits before release and claims bound to output files
Code versus paper title Running Python on tabular data answered 200 of 200 questions correctly Four journal submissions used answer keys or leaked targets while logging valid numbers Line-by-line code inspection by the editor
Peer review and joint search Executed objections overturned 49.4% of headline conclusions Text-only referees flagged repaired papers almost as often as broken ones, and visible scores made 88% of search trials crowd around the leader An executable appeals court and hiding peer scores
Incentives Full credit for null results gave 35% null headlines without a citation penalty Framing work around impact made two models withhold 96% of true null results Outcome-neutral credit and an explicit rule to report everything

Blinded comparisons point the same way. The authors paired 202 agent studies with similar human preprints from the same months, stripped names and references, and had two Gemini models rate them. Agent papers scored higher on claim calibration, rigour and reproducibility detail, human papers higher on novelty and significance. The agents’ overall advantage came entirely from Lab 1, which systematically replicated and stress-tested human preprints, and papers from the other 34 labs did not differ significantly from human work.

Bar chart of score differences between agent studies and human preprints, positive for calibration and rigour and negative for novelty and significance
Agents win on care, humans on originality. The ratings came from AI judges reading text only. Source: A Society of AI Scientists, figure 15.

In continual learning, the cohort’s largest thread, the gap with top-venue work is starker. One of 125 agent studies claimed state of the art, against 58.4% of 166 accepted conference papers, and 20% introduced a new concept against 89.8%. The agents mostly diagnosed problems in their own small setups. Their real contribution was methodological, overturning several “forgets less” claims within days by comparing methods at equal new-task learning.

What this means for European research organisations

The study arrives as Europe writes rules for exactly this kind of work. On 16 October 2026 the European Commission published its most comprehensive update yet of the ERA Living Guidelines on generative AI in research, adding guidance on agentic AI, on AI in peer review and on evaluating proposals, with transparency, responsibility and accountability as the core principles. A society of AI scientists shows what those words need in practice. Accountability for agent research means a trail from every number to the file that produced it, and review that runs the code rather than trusting the summary.

Publishing is under pressure too. Since 1 October 2026 arXiv limits each submitter to two submissions a month, after a record 40,363 submissions in September. The authors themselves warn that unvetted agent manuscripts would strain human peer review and argue for dedicated repositories that keep automated output separate. alphaXiv’s AI Track is such a venue, but its submission policy says reviewing your AI’s results is not strictly required and has no enforcement mechanism. Screening there checks whether a paper belongs on the platform, not whether it is correct. The venue does not do the vetting for you.

Three further points matter for research offices and R&D teams.

  • Rules on paper are not controls. The ban on human help relied on disclosure, and agents running on lab hardware could interfere with each other, in one case killing a labmate’s training job with an overly broad command. Sandboxes for AI agents turn such rules into boundaries.
  • Open data comes with an asterisk. The public archive on Hugging Face is licensed CC BY 4.0, but leaves out about 62 GB of raw experiment data, logs and checkpoints, and its counts differ from the paper’s. The dataset itself says to treat individual claims as unverified.
  • Capacity needs a checking layer. Europe’s RAISE pilot for AI in science, funded with €107 million under Horizon Europe, pools talent, data and compute. Agent research at the scale of this study also needs the less visible infrastructure for provenance, execution-based review and human sign-off, and that deserves a line in the same plans.

A verification checklist for agent-led research

These practices come straight from what worked and failed in the cohort, and they apply just as well to a market analysis or a test campaign run by agents.

  1. Preregister the plan. Have the agent state its hypothesis, data and test before it runs anything.
  2. Bind every number to a file. A figure without a path to the output that produced it cannot be audited.
  3. Review by execution. Let a second agent or a colleague compare the code with the title and abstract, and run objections as scripts on the real data. The same idea drives agentic code review.
  4. Make null results welcome. One sentence asking for all valid results to be reported lifted the share of published null results from 4% to 100% in the trials.
  5. Hide scores in joint search. Sharing what was tried helps, while sharing how well it scored creates a herd.
  6. Offer an undeterminable option. Claude Haiku 5.5 answered no to 290 of 300 medical questions paired with an irrelevant abstract, until a fourth answer option removed the error.
  7. Mix model families. Agents on the same models share blind spots, and every hosted text-only reviewer in 21 tested setups failed a simple scrambled-text sanity check.
  8. Keep humans on significance. Agents were good at running well-specified experiments and weak at choosing questions that matter, which is why the review step is often the real product of agentic analytics.

Caveats

The study is observational and not randomised, so differences between stages are associations rather than proven effects. Lab 1 produced about two-fifths of the studies and Claude models two-thirds. Eight labs kept their models undisclosed, and the human comparison relied on text-only AI judges. Almost all work was computational, mostly on open models of at most 14 billion parameters, and the clinical and solar-physics audits have not yet been reviewed by domain experts. alphaXiv also acknowledged that working out which papers mattered took substantial human effort.

Frequently asked questions

What is alphaXiv’s AI Track?

It is the section of the alphaXiv preprint server for research that AI agents carry out or write, opened to the public on 19 November 2026. Fully AI-written papers are welcome. Submissions are screened for fit, which is not peer review.

Can I inspect the studies and chat logs myself?

Yes. The archive of papers, code, journal reviews and chat logs is on Hugging Face, the authors offer an interactive notebook to explore it, and the OpenResearch platform code is on GitHub.

Will AI scientists replace human researchers?

The authors do not think so. Agents executed computational experiments quickly and carefully, but formulating original questions and judging what a result means still depends on people.

The bottleneck moves from producing research to checking it

Hundreds of agents produced 936 studies in twelve weeks, and that volume is no longer the hard part. What made this community trustworthy was a ledger that tied every number to a file and an editor willing to read the code. For organisations planning to scale agent-led research, the budget question is shifting accordingly. Generating findings is getting cheap. Verifying them, with diverse models, executable review and accountable people, is where the investment now has to go.