Multi-Agent Intelligence Systems

Two research programmes share this heading, and merging them produces most of the confusion in the area. The first is multi-agent reinforcement learning, where a population of agents is trained against itself and the interaction is part of the optimisation. That programme produced OpenAI Five, AlphaStar and the hide-and-seek autocurriculum; its results are superhuman, replicated in competition conditions, and not seriously disputed. The second is multi-agent inference over large language models: debate protocols, role-play scaffolds, and orchestration frameworks in which a planner, a coder, a tester and a reviewer pass messages to one another. The second programme attracts the funding, the frameworks and the headlines. It is also the one whose evidence base is thin, and the two are routinely cited as though they were one body of work.

This brief is unusual for the Institute in that the deflationary position is not something argued here against the field. It is the field’s own. The most-cited failure analysis of multi-agent LLM systems opens by stating that performance gains on popular benchmarks are often minimal. The cleanest single-versus-multi comparison in the literature found that a single agent with a strong prompt matches the best discussion method. Debate does not reliably beat self-consistency. Performance is non-monotone in the number of model calls. The flagship collaboration-scaling paper reports logistic — that is, saturating — growth, which is close to the opposite of what a scaling law is usually taken to mean.

Underneath all of that sit two measurements nobody has published. The first is a compute-matched comparison: the same model, the same total token budget, the same wall clock, a multi-agent configuration against the best single-agent configuration, with error bars. The second is an error-correlation matrix for an LLM ensemble — the pairwise probability that two agents are wrong on the same item. Ensemble theory says the second quantity is what determines whether a collective can exceed its best member at all. Without it, the exotic question at the far end of this subject — can a designed collective provably outperform its strongest individual? — is not merely unanswered. It is not yet well-posed. Both experiments are cheap. Neither has been run. That is the finding this brief is built around.

Established The field’s own most-cited failure analysis opens with a deflationary sentence, and it is worth quoting exactly. Cemri and colleagues at Berkeley, in “Why Do Multi-Agent LLM Systems Fail?” (arXiv:2503.13657, March 2025), begin: “Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal.” Established The protocol behind that sentence is substantial rather than rhetorical: MAST-Data collects more than 1,600 annotated execution traces across seven popular multi-agent frameworks, of which 150 traces were used to build the taxonomy, with inter-annotator agreement at kappa = 0.88, over GPT-4, Claude 3, Qwen2.5 and CodeLlama on coding, mathematics and general agent tasks. Established The output is the Multi-Agent System Failure Taxonomy, MAST: fourteen failure modes in three categories — system design issues, inter-agent misalignment, and task verification. Frontier One caveat belongs with the citation rather than in a footnote: the per-category failure rates and the head-to-head accuracy comparisons live in the paper body, which the research pack behind this brief could not extract, so no numeric failure rate is stated here. The opening sentence and the taxonomy structure are what this brief vouches for.

Established The cleanest single-agent-versus-multi-agent comparison in the literature is negative. Wang, Wang, Su, Tong and Song, “Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?” (arXiv:2402.18272, February 2024), report that a single-agent LLM with strong prompts achieves almost the same performance as the best existing discussion approach across reasoning tasks. Established Their qualification is the more interesting half: multi-agent discussion outperforms a single agent only when there is no demonstration in the prompt. Frontier Read structurally, that says the discussion is supplying something a well-written prompt already supplies. It is a substitute for prompt engineering, not an addition to it, and once the prompt is good the substitute stops paying.

Established Debate does not reliably beat the single-agent baseline it is usually compared against unfavourably. Smit, Duckworth, Grinsztajn, Barrett and Pretorius, “Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs” (arXiv:2311.17371, November 2023), state that multi-agent debating systems in their current form do not reliably outperform other proposed prompting strategies such as self-consistency and ensembling over multiple reasoning paths. Established With hyperparameter optimisation — and specifically by tuning agent agreement levels — some debate variants do exceed non-debate baselines. Frontier That is a more damaging result than it first reads: if the gain arrives through a tuned agreement threshold, the gain is a hyperparameter, not an architecture. Established The baseline in question is not weak. Self-consistency, from Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery and Zhou (arXiv:2203.11171, March 2022), samples diverse chains of thought and marginalises over them, and reports gains of +17.9 points on GSM8K, +11.0 on SVAMP, +12.2 on AQuA, +6.4 on StrategyQA and +3.9 on ARC-challenge. Any multi-agent claim that does not report self-consistency at a matched sample count has not reported a baseline.

Established More model calls is not monotonically better, and there is a mechanism for why. Chen, Davis, Hanin, Bailis, Stoica, Zaharia and Zou, “Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems” (arXiv:2403.02419, March 2024), find that the performance of both Vote and Filter-Vote strategies can first increase and then decrease as a function of the number of language-model calls. Established The mechanism they give is that easy queries benefit from additional calls while hard queries degrade, so the aggregate curve turns over. Frontier They fit a scaling model that predicts the optimum from a small number of samples, which makes the turnover a design parameter rather than a surprise. Established This is the theoretical reason that “add more agents” cannot be a general strategy, and it is a result from the same laboratory tradition that produced MAST.

Established The two strongest pro-scaling results say less than their titles do. Li, Zhang, Yu, Fu and Ye, “More Agents Is All You Need” (arXiv:2402.05120, February 2024, Tencent), report that performance scales with the number of agents instantiated, via a method they call Agent Forest, described as orthogonal to existing complicated methods, with the enhancement correlated to task difficulty; the method is sampling-and-voting. Frontier Sampling-and-voting is what self-consistency is, so the paper is best read as a restatement of an inference-time-compute result in agent vocabulary rather than as evidence about agent specialisation or communication. Established Qian and colleagues at Tsinghua and OpenBMB, “Scaling Large Language Model-based Multi-Agent Collaboration” (MacNet, arXiv:2406.07155, June 2024), organise agents as a directed acyclic graph, demonstrate collaboration among over a thousand agents, and report that irregular topologies outperform regular ones. Established Their headline scaling claim is that overall performance follows a logistic growth pattern as agents scale, with collaborative emergence occurring earlier than traditional neural emergence, and logistic growth saturates by definition. Frontier The most enthusiastic scaling paper in the subfield is therefore also the paper whose own fitted curve says that adding agents stops helping, and this brief regards that as the single most under-read result in the area.

Established Where multi-agent architectures demonstrably do win, the wins are in three specific places, and none of them is raw problem-solving. The first is judgment. Khan, Hughes, Valentine, Ruis, Sachan, Radhakrishnan, Grefenstette, Bowman, Rocktaeschel and Perez, “Debating with More Persuasive LLMs Leads to More Truthful Answers” (arXiv:2402.06782, February 2024), have two expert models argue opposite answers while a non-expert judge chooses: non-expert models reach 76% with debate against 48% naive, and human judges reach 88% against 60% naive — 28-point improvements in both cases. Frontier That is the strongest positive multi-agent result this brief could source, and it is a scalable-oversight result about judging, not a capability result about solving. Established The second is context. Zhang, Sun, Chen, Pfister, Zhang and Arik, “Chain of Agents” (arXiv:2406.02818, June 2024, Google), report improvements of up to 10% over retrieval-augmented generation, full-context and multi-agent baselines on long-context tasks. Frontier Sequential workers plus a manager partition a context window; the agents divide the text, not the problem, which makes this a memory-architecture result. Established The third is cost. Wang, Wang, Athiwaratkun, Zhang and Zou, “Mixture-of-Agents” (arXiv:2406.04692, June 2024), report 65.1% on AlpacaEval 2.0 using only open-source models against 57.5% for GPT-4 Omni. Frontier Two caveats travel with that number and must not be dropped: AlpacaEval 2.0 is an LLM-judged preference benchmark rather than a correctness benchmark, and the comparison is not compute-matched.

Established The orchestration frameworks are engineering artefacts with framework papers, and their abstracts do not carry controlled single-agent comparisons. AutoGen (Wu and colleagues, Microsoft, arXiv:2308.08155, August 2023) claims effectiveness across mathematics, coding, question answering, operations research and entertainment. Established MetaGPT (Hong, Zhuge, Chen and colleagues including Schmidhuber, arXiv:2308.00352, first version August 2023 through a seventh in November 2024) encodes standardised operating procedures into prompt sequences and assigns roles assembly-line fashion, claiming superior coherence on software engineering benchmarks against chat-based alternatives. Established ChatDev (Qian and colleagues, arXiv:2307.07924) adds a chat chain and a mechanism it calls communicative dehallucination, reporting that natural language suits system design while programming language helps in debugging. Established Self-Organized Agents (Ishibashi and Nishimura, arXiv:2404.02183) scale agent count with task complexity and report +5% accuracy on HumanEval. Frontier Generative Agents (Park, O’Brien, Cai, Morris, Liang and Bernstein, arXiv:2304.03442) put 25 agents in a sandbox and produced the Valentine’s-party demonstration, with an ablation showing that observation, planning and reflection each contribute critically — to believability, which is what the ablation measured. Established None of these five is a compute-matched comparison and none claims to be.

Established The benchmarks that compare coordination strategies report small effects, and the ceiling on the whole enterprise is measured. Zhu and colleagues, “MultiAgentBench” (arXiv:2503.01935, March 2025), score milestone-based key performance indicators rather than task completion alone, compare star, chain, tree and graph topologies, and find that graph structure performs best among coordination protocols in the research scenario, with cognitive planning improving milestone achievement rates by 3%. Frontier A 3% milestone improvement, reported without an interval, is the honest magnitude of a coordination-strategy effect as the field currently measures it. Established Zhuge and colleagues, “Agent-as-a-Judge” (arXiv:2410.10934, October 2024), introduce DevAI — 55 automated AI development tasks with 365 hierarchical user requirements — and report that an agentic judge dramatically outperforms LLM-as-a-Judge and is as reliable as their human evaluation baseline. Frontier That result matters here mostly as a circularity warning: multi-agent systems are increasingly evaluated by other agent systems. Established TheAgentCompany (arXiv:2412.14161, version 3 dated September 2025) simulates a software company and reports that the most competitive agent completes 30% of tasks autonomously. Frontier That is the environment in which multi-agent architectures ought to shine, and it is where the ceiling is currently drawn.

Established The previous generation of multi-agent work produced superhuman results, and forgetting this is the commonest error on the skeptical side. OpenAI Five (Berner and colleagues, arXiv:1912.06680, December 2019) became, on 13 April 2019, the first AI system to defeat the world champions at an esports game, after ten months of continual training consuming roughly two million frames every two seconds. Established AlphaStar (Vinyals and colleagues, Nature 575:350–354, 2019) reached Grandmaster level for all three StarCraft II races through a league of continually adapting strategies and counter-strategies, placing above 99.8% of officially ranked human players. Established Baker, Kanitscheider, Markov, Wu, Powell, McGrew and Mordatch (arXiv:1909.07528, 2019, revised 2020) report clear evidence of six emergent phases of agent strategy from hide-and-seek self-play. Frontier CICERO, Meta’s Diplomacy agent (Science 378:1067–1074, 2022), belongs in this list, but its performance figures could not be obtained for this brief — the publisher was gated during the research pass — so the bibliographic record is cited and no number from it is stated. Established The distinction that matters: multi-agent training has a strong track record; multi-agent LLM inference is the thing with the weak one.

Established Infrastructure that makes a fair comparison possible arrived only recently, which is the most important dating fact in this brief. Ye, Huang, Wu, Chen and colleagues released MASLab (arXiv:2505.16988, May 2025, revised June 2026), motivated explicitly by redundant re-implementation efforts, unfair comparisons and high entry barriers. Established It integrates more than twenty methods validated against their official implementations, with standardised benchmarking across more than ten datasets and eight models. Frontier The implication is uncomfortable and should be stated plainly: until MASLab existed, essentially no multi-agent comparison in the published literature was controlled in the sense a reader would assume. Speculative Everything in the workback plan below is now runnable for the first time, and the reason to expect it to run soon is that the substrate is public, cheap and already validated.

Established The most striking thing about the frontier here is that the consensus has already turned deflationary, and it turned from inside. Four independent sources between 2024 and 2026 point the same way and none of them comes from a skeptic camp: MAST opens by calling gains minimal; MASLab exists because prior comparisons were unfair; MultiAgentBench measures a 3% coordination effect; MacNet reports logistic rather than power-law scaling. Frontier A field’s own builders converging on a deflationary reading is a stronger signal than an outside critique, and it is the reason this brief treats the negative position as the current best estimate rather than as a contrarian option.

Frontier Three claims are habitually merged and should be separated. One: multiple agents solve better. Two: multiple agents judge better. Three: multiple agents let a weaker supervisor oversee a stronger system. Established The evidence for the second and third is far stronger than for the first, since Khan and colleagues’ 28-point gains are gains for weak judges. Speculative If the correct framing of this subject is scalable oversight rather than capability, the capability literature of the last three years is a detour and the useful programme is much narrower than the one being funded.

Frontier The quantity that actually determines whether an ensemble helps is error correlation, and it is not being measured. Classical ensemble theory is unambiguous: a majority vote over voters whose errors are independent, each individually better than chance, converges toward certainty as the population grows, and a sufficiently decorrelated ensemble can exceed the accuracy of its best single member. Established That is the Condorcet result and its modern ensemble-learning descendants, and it is a theorem rather than an empirical finding. Speculative The mechanistic explanation for the negative LLM results above is that agents built on the same base model, prompted from the same template, trained on overlapping data, have highly correlated errors — so the ensemble bound collapses toward the individual accuracy. Frontier The testable prediction that falls out is sharp: multi-agent gains should scale with the diversity of the base models, not with the number of agents. Frontier Mixture-of-Agents, in which heterogeneous open-source models beat a single frontier model on a preference benchmark, is weak evidence for exactly this. Handwave Nobody has run the controlled version, and no error-correlation matrix for an LLM multi-agent ensemble appears to have been published anywhere.

Frontier The bottleneck may be verification rather than coordination. MAST’s third failure category is task verification, and the parallel finding in Artificial Scientists is that AI systems produce durable results exactly where a cheap formal verifier exists and fail where the verifier is another language model. Speculative Multi-agent systems multiply generated content without multiplying verification capacity, which would predict that systems with strong external verifiers — unit tests, proof checkers, simulators — show much larger multi-agent gains than systems without. Frontier That experiment is cheap, decisive and, as far as this brief could establish, unrun.

Speculative A serious minority position holds that this is an organisational-design problem and that the relevant literature is not machine learning. MAST’s three categories map almost exactly onto classical organisational failure modes: unclear role boundaries, missing escalation paths, absent acceptance criteria. Frontier MetaGPT’s encoding of standardised operating procedures is the existing instance of the hypothesis, and it has never been ablated against an unstructured baseline at matched compute. Handwave The stronger version — that the right theory of multi-agent AI is a theory of firms, contracts and incentives rather than of architectures — is coherent, has advocates, and has produced no measurement.

Frontier Emergence claims in this area rest on believability, not on capability. Generative Agents demonstrated emergent socialisation in a sandbox, and the ablation supporting it measured whether the behaviour was believable to human raters, which is a legitimate and well-executed result about simulation. Speculative A capability claim would require the emergent coordination to solve a problem no individual agent could solve, with the individual baseline measured, and that comparison has not been run for any LLM agent society. Frontier The hide-and-seek autocurriculum is the closest thing to a genuine emergence result here, and it comes from the reinforcement-learning side, where the interaction sits inside the optimisation loop rather than inside the prompt.

Speculative Debate may make systems worse under adversarial pressure, and nobody has measured the curve. If one agent is persuasive and wrong, a debate protocol amplifies rather than corrects. Frontier The suggestive evidence is Smit and colleagues’ finding that agent agreement level was the decisive hyperparameter — which is to say that the system’s truthfulness is a function of how easily its agents capitulate. Handwave A measured sycophancy-versus-accuracy curve for multi-agent debate would settle a great deal and does not exist.

Frontier The economically decisive question is cost per solved task, and it is almost never reported. Kapoor, Stroebl, Siegel, Nadgir and Narayanan (arXiv:2407.01502, July 2024) document that agent benchmarks score accuracy while ignoring cost, that many have no holdout set at all, and that evaluation practice is not standardised. Established Multi-agent systems multiply calls by construction, so the critique applies with double force. Speculative Adoption does not require an accuracy improvement at unbounded cost; it requires a Pareto improvement on the cost-accuracy frontier, and this brief could find no multi-agent result presented that way.

Established The binding constraint in this subject is not a technology. It is an unrun control. Every claim about multi-agent superiority is provisional on a comparison that has not been published: the same base model, the same total token budget, the same wall clock and the same dollar cost, multi-agent configuration against best single-agent configuration, on a held-out task suite, with confidence intervals. Frontier The research pack behind this brief searched for such a comparison and found none. Speculative That is an absence of evidence rather than evidence of absence — discovery in that pass was constrained, with the arXiv search API and OpenAlex unavailable — but the absence is corroborated by MASLab’s own stated motivation, which is that comparisons in this literature have been unfair.

Frontier The second missing measurement is the error-correlation matrix, and it is the one that makes the exotic question well-posed. Pairwise error correlation between agents, measured as a function of shared base model, shared prompt and shared training data, is the input that ensemble theory requires. Established Without it, a claim that a collective provably outperforms its best member has no quantity to be a proof about. Handwave Building the matrix requires nothing beyond per-agent per-item response logging on a benchmark that already exists, which is why its absence is the most surprising fact in this brief.

Established Three secondary bottlenecks sit underneath both. First, statistical practice: Miller’s “Adding Error Bars to Evals” (arXiv:2411.00640, November 2024) makes the case that evaluations are experiments and supplies the paired-difference machinery, and the overwhelming majority of published tables in this area report point estimates with no interval. Established Second, holdouts: agent benchmarks frequently have none, so improvements may be overfitting to the benchmark. Frontier Third, the harness noise floor: a 3% topology effect is not interpretable until somebody reports how much a benchmark score moves under reseeding alone, and this brief could not locate that figure for any multi-agent harness.

Frontier A fourth bottleneck is that failure is now catalogued but not yet quantified. MAST supplies fourteen modes at kappa = 0.88, which is the first reliability-checked taxonomy in the area, and that is a real advance over anecdote. Speculative A taxonomy becomes an engineering instrument only when each mode carries a measured frequency and a measured cost, and the numbers that would do that sit in the paper body rather than in any summary this brief could verify.

Established This subject waits, more than on anything else, on the measurement discipline developed in Intelligence Measurement. Every experiment proposed below is a paired comparison between two systems on a shared item set, and paired comparison is exactly what benchmark practice in this field has not been doing. Established Confidence intervals, holdout sets, cost reporting, contamination checks and a stated attempt count are prerequisites rather than refinements here, because the effects at stake are of the order of a few percentage points.

Frontier A second dependency is a settled operational definition of compute matching. Matched on what — total tokens, total dollars, wall clock, peak concurrency? Speculative The four give different answers, and a multi-agent system that loses on tokens can win on wall clock through parallelism, which is a real engineering advantage and a different claim from a capability advantage. Frontier Until the field agrees which quantity is being held constant, results will not compose.

Frontier A third dependency runs through verification. The finding in Artificial Scientists that generative systems produce durable results only where a cheap exact verifier exists is, if it generalises, the dominant term in multi-agent performance. Speculative That makes progress here partly dependent on progress in automated verification rather than in coordination, and it predicts that the subfield’s next real gains will come from better checkers rather than better conversations.

Frontier The fourth dependency is the one the field’s founding agenda named and then nobody built. The second objective in Dafoe and colleagues’ cooperative-AI programme is cooperation in mixed human-machine populations, and the LLM era has produced almost no empirical literature on it. Established TheAgentCompany, the most realistic organisational environment available, simulates colleagues with language models rather than employing humans. Speculative Benchmarks with real humans inside the agent loop are a prerequisite for any claim about multi-agent systems in deployed organisations, and building them is a research-infrastructure problem rather than a modelling one.

Established Experiment one, and the binding link: the compute-matched control. For each method in MASLab, hold total tokens, wall clock and dollars constant and compare against three single-agent controls — greedy decoding, self-consistency at matched sample count, and best-of-N with a reward model at matched N. Frontier Report paired differences with confidence intervals. Speculative The prediction worth pre-registering is that most published multi-agent gains will not survive the self-consistency control. Established This is the experiment the subfield is missing, it is now cheap because the substrate is public, and every claim above it is provisional until it runs.

Frontier Experiment two: the error-decorrelation measurement. Log per-agent, per-item correctness across a benchmark and compute the pairwise error correlation as a function of shared base model, shared prompt template and shared training corpus. Speculative Report it as a matrix. Handwave That matrix, plus each agent’s marginal accuracy, is sufficient to compute a Condorcet-style bound on collective accuracy from measured rather than assumed quantities — which is what “provably outperforms its best member” would have to mean.

Frontier Experiment three: heterogeneity scaling. Measure gains as a function of base-model diversity at fixed agent count, against agent count at fixed diversity. Speculative The prediction from ensemble theory is that diversity dominates; Mixture-of-Agents is the suggestive uncontrolled result.

Frontier Experiment four: verification-capacity scaling. Run the same multi-agent configuration with and without an external formal verifier — unit tests, a proof checker, a simulator — and attribute the gain to verification versus coordination. Speculative The prediction is that verification explains most of it.

Frontier Experiment five: adversarial robustness of the aggregation rule. Inject one confidently wrong agent and measure collective accuracy as a function of that agent’s persuasiveness and of the agreement-level hyperparameter Smit and colleagues identified. Speculative This produces the sycophancy-versus-accuracy curve the field lacks and is a precondition for deploying debate in any adversarial setting.

Frontier Experiment six: the organisational ablation. Strip MetaGPT’s standardised operating procedures out and compare at matched compute, which tests whether encoded process structure does any work at all, and repeat the design with the interventions organisational theory would suggest — explicit role boundaries, defined escalation, stated acceptance criteria — scored against measured MAST failure rates rather than against task accuracy alone. Speculative The ordering matters: experiment one determines whether the subject has a subject, and experiment two determines whether its central exotic claim can be stated at all. Handwave No component of this chain past the MASLab substrate has been published, and the whole of it could be executed by one well-funded group inside two years.

Established The engineering required is unglamorous and almost entirely instrumentation. A harness that can run any of these experiments needs token accounting per agent and per turn, wall-clock and dollar accounting alongside it, deterministic seeding, a shared model endpoint so that all arms of a comparison hit the same weights, and full trace logging at the granularity MAST annotation requires. Frontier MASLab supplies the method re-implementations; it does not by itself supply the cost ledger that a compute-matched claim needs.

Frontier The error-correlation matrix imposes one specific logging requirement that current harnesses generally do not meet. Aggregate accuracy is not enough: the matrix needs per-agent, per-item correctness before aggregation, which means the harness must retain each agent’s individual answer rather than only the collective output. Established That is a schema change, not a research problem.

Frontier Parallelism is the one genuine engineering advantage of multi-agent designs and it is systematically under-claimed. A configuration that loses on total tokens may still win on latency because its agents run concurrently, and for interactive systems latency is the binding constraint. Speculative A brief honest accounting would report the cost-accuracy frontier and the latency-accuracy frontier separately, because a system can sit on one and not the other.

Frontier Failure containment is the remaining engineering theme. The overthinking pathologies documented by Cuadron and colleagues across 4,018 trajectories — analysis paralysis, rogue actions, premature disengagement — compound in multi-agent settings, because one agent’s pathological trajectory becomes another agent’s input. Speculative Circuit breakers, turn budgets and explicit escalation to a human are the obvious mitigations and none has a published effect size in a multi-agent harness. Frontier The mitigation Cuadron and colleagues report for the single-agent case improves performance while reducing computational cost, which is the shape of result a multi-agent harness ought to be looking for and is not.

Speculative One design requirement follows from the error-correlation framing and is currently ignored. If decorrelation is the quantity that buys collective accuracy, then a production system should be assembled from deliberately different base models, prompt templates and decoding settings, and should measure its own realised correlation in flight rather than assuming it. Handwave A system that reports the decorrelation it is actually achieving, alongside its accuracy and its cost, would be the first honest multi-agent product specification.

Established The nearest technology is not another agent architecture; it is inference-time sampling. Brown and colleagues’ “Large Language Monkeys” (arXiv:2407.21787, July 2024) show that coverage — the fraction of problems solved by any sample — scales log-linearly with sample count over four orders of magnitude, taking DeepSeek-Coder-V2-Instruct from 15.9% at one sample to 56% at 250 samples on SWE-bench Lite. Established Crucially, in domains without automatic verifiers, majority voting and reward models plateau beyond several hundred samples. Frontier Most reported multi-agent gains are consumed by this confound, because a multi-agent system is a sampling budget wearing an organisational chart.

Frontier Three further adjacencies matter. LLM-as-judge and its agentic successor are the evaluation substrate the whole subfield now depends on, with the circularity that implies. Established Long-context architectures compete directly with the chain-of-agents approach, since a model that holds the document does not need workers to partition it. Speculative And market-based coordination — auction and contract-net mechanisms for allocating tasks among self-interested agents, the classical multi-agent-systems tradition that predates all of this by decades — is the natural design language for agent economies, though this brief must record that its source pack could not reach that literature, which is largely in venues that were unavailable during the research pass.

Frontier Multi-agent systems that produced verified scientific output are the strongest applied adjacency. The Virtual Lab (Swanson, Wu, Bulaong, Pak and Zou, Nature 646:716–723, 2025) had an LLM principal-investigator agent direct LLM scientist agents through research meetings with human high-level feedback, designing 92 nanobodies of which two showed improved binding to the JN.1 or KP.3 variants while retaining ancestral-spike binding. Frontier Robin (arXiv:2505.13400, May 2025) produced ripasudil as a candidate for dry age-related macular degeneration with follow-up RNA-seq showing ABCA1 upregulation. Speculative Both are in-house results with explicit and unquantified human involvement, and both are treated in full in Artificial Scientists.

Frontier The reason the decisive experiment has not been run is institutional, not technical. Publication incentives reward a new architecture with a positive number; they do not reward a negative control that retires forty papers. Speculative A compute-matched null result on the best twenty published methods would be one of the most valuable papers in the subfield and is close to unpublishable in the venues where those methods appeared, which is a description of a market failure rather than of a scientific one.

Established Reporting standards are the cheapest available intervention. A conference or journal requiring that any multi-agent result state its total token count, its dollar cost, its single-agent baseline configuration, its attempt count, and a self-consistency comparison at matched samples would change the evidence base within one cycle. Frontier Kapoor and colleagues made precisely this argument for agent benchmarks generally and it has not been adopted as policy anywhere this brief could confirm.

Frontier Vendor claims are the weakest link in the public record, and this brief declines to repeat the most-cited one. A widely circulated engineering write-up reports a large percentage improvement for a multi-agent research system over a single-agent baseline on an internal evaluation, together with a several-fold token-cost multiple. Established The source was inaccessible during the research pass for this brief, so no number from it is stated here. Frontier The general rule follows: a vendor multi-agent number requires the evaluation set, whether it was internal, the single-agent baseline’s exact configuration, the token-cost multiple and the query count, and a claim missing any of those is not a fact about the world.

Speculative Shared infrastructure is a public good that one funder could buy outright. MASLab exists; a funded programme to run the compute-matched comparison across its methods and publish the resulting cost-accuracy frontier would cost a small fraction of what is currently spent building new frameworks. Handwave The Institute’s view is that this is the highest expected-value grant available in machine intelligence at present, precisely because the result is decisive in either direction. Speculative A pre-registration requirement would sharpen it further: state the predicted direction and effect size before running, so that a null result is a completed experiment rather than a failed one.

Established The first ethical consideration is the plainest: multiplying model calls multiplies energy and cost for a gain that may be zero. When the accuracy improvement is a few percentage points and the token multiple is severalfold, the environmental and financial cost is being incurred against an unmeasured benefit. Frontier That is an ordinary consumer-protection question dressed as an architecture debate.

Frontier Accountability fragments across agents. MAST’s middle category, inter-agent misalignment, describes failures that belong to no single component, and a system whose errors are emergent properties of message passing is harder to audit than a single model with a single trace. Speculative Regulatory regimes built around model-level evaluation, discussed in AI Governance, do not currently have a unit of analysis for a system of this shape.

Frontier Debate cuts both ways and the positive side is genuinely important. Khan and colleagues’ result is a scalable-oversight finding: a non-expert judge, human or model, becomes substantially more accurate when two stronger systems argue opposite sides. Speculative If that effect scales with the capability gap between debaters and judge, it is one of the few concrete mechanisms available for supervising systems more capable than their supervisors, and the scaling has not been tested.

Speculative The adversarial case is unaddressed. A multi-agent system in which persuasiveness rather than correctness drives consensus is a system that can be captured by whichever party can make one agent confident. Handwave In an agent economy where agents transact on their principals’ behalf, that failure mode has a market-manipulation analogue, and no published work measures it.

Frontier Evaluation by other agents creates a conflict that is not yet named as one. Agent-as-a-Judge reports an agentic evaluator as reliable as its human baseline, and the field is adopting agentic evaluation because human evaluation is expensive. Speculative A system evaluated by machinery drawn from the same model family it is built on can be right for reasons that will not generalise, and the correlated-error argument this brief makes about solving applies equally to judging. Speculative The obligation that follows is modest and unmet: report the base model of the judge alongside the base model of the system.

Speculative The stakes of the unrun experiment are larger than the subfield. If a designed collective can be shown to exceed its best member from measured error correlations, then organisation becomes a computable engineering discipline rather than a craft, and the immediate application is not software but institutions — committees, juries, review panels, forecasting groups, all of which are ensembles whose error correlations nobody measures either.

Frontier If instead the negative result holds, the consequence is equally consequential and much cheaper to act on. It would mean that the organisational-chart metaphor imported into AI systems design is a cost multiplier, and that the correct architecture for hard problems is one strong model with a good prompt and a large sampling budget. Speculative A great deal of current agent-platform investment is a bet against that outcome.

Speculative Agent economies are the version of this question that arrives whether or not the research resolves. Once agents transact, negotiate and contract on behalf of principals, the coordination problem stops being an architecture choice and becomes market design, with the classical apparatus of mechanism design, reputation and settlement. Handwave Whether such an economy is more efficient than the human one it partially replaces is the kind of question that will be answered by deployment rather than by experiment, which is exactly the wrong order.

Speculative There is a longer civilizational reading, and it belongs to the training side rather than the inference side. League training, autocurricula and self-play produced the only genuinely superhuman multi-agent systems anyone has built, and they did so by putting the interaction inside the optimisation rather than inside the conversation. Handwave If a general capability jump ever comes from multiplicity, the evidence assembled here suggests it comes from populations that train against each other over months, not from committees of frozen models talking for a few thousand tokens — and that is a claim about where to point a very large compute budget.

These horizons track the measurements rather than the products, because the products already exist and the measurements do not.

• 10 yr: Frontier The compute-matched comparison is published across the MASLab method set, with confidence intervals and a cost-accuracy frontier. This is a one-to-three-year piece of work that has been available since 2025, so a ten-year horizon is a statement about incentives rather than about difficulty. Speculative The most likely outcome, on the evidence assembled here, is that most published multi-agent gains do not survive a matched self-consistency control, and that the surviving cases cluster in long-context partitioning, weak-judge oversight and heterogeneous-model aggregation.
• 25 yr: Speculative Error-correlation matrices are routine reporting for any ensemble system, and multi-agent design becomes an exercise in maximising decorrelation subject to a compute budget rather than in choosing a topology. Speculative Heterogeneity, not agent count, is the design variable; systems deliberately mix base models, training corpora and prompting styles to purchase independence. Speculative Scalable oversight through structured debate is either a validated supervision protocol with a measured scaling relation between capability gap and judge accuracy, or it has failed a specific test and been retired, and either outcome is more useful than the present state.
• 50 yr: Speculative Collective bounds are computed from measured quantities before deployment, so a system can carry a stated guarantee that it exceeds its best member on a defined distribution, in the way a structural component carries a load rating. Handwave The same apparatus is turned on human institutions, and committee composition is chosen to minimise measured error correlation rather than to balance constituencies, which would be the first genuinely new tool for institutional design in a very long time.
• 100 / 250+ yr: Handwave A designed collective whose provable capability exceeds any of its members by a wide and predictable margin becomes the standard unit of cognitive work, and the individual model matters less than the decorrelation structure of the population it sits in. Handwave Nothing in the current evidence base supports this and nothing in it forbids it; the honest statement is that the question is not yet formulated well enough to have a timeline.

• Depends on This brief depends most directly on Intelligence Measurement, and the dependency is unusually concrete. Every claim assessed here is a difference between two systems of a few percentage points, measured on benchmarks that frequently lack holdout sets, reported without intervals, at attempt counts that are often unstated. The compute-matched comparison that is this subject’s binding constraint is a paired-difference experiment, and paired-difference machinery, contamination control, cost reporting and error bars are all products of that brief rather than this one. Until the measurement discipline lands, a multi-agent result cannot be distinguished from harness noise, and the entire literature assessed above inherits that limitation.
• Enables What this subject would enable, if the two missing measurements were made, is a genuine engineering discipline of designed collectives: systems assembled to a stated decorrelation target with a computed bound on collective accuracy, rather than assembled by analogy to a company org chart. The nearer-term enablement is narrower and more certain — a validated scalable-oversight protocol in which weaker judges supervise stronger systems through structured debate, which is the one multi-agent result with a large measured effect behind it. Both routes run through the same instrumentation: per-agent, per-item logging and honest cost accounting. A negative outcome enables something too, and it is not nothing. A published compute-matched null across the twenty best methods would redirect a substantial fraction of current agent-platform engineering toward single strong models with large sampling budgets and good verifiers, which is a cheaper architecture that the evidence already mildly favours. This brief regards that as the most likely outcome and does not regard it as a disappointment.
• Adjacent The adjacent territory is wide. Artificial Scientists supplies the verifier finding that may be the dominant term in multi-agent performance, and its strongest systems are themselves multi-agent. Collective Intelligence is the human-side literature on the same question, and the error-correlation framing transfers directly to it. Cognitive Architectures covers the single-agent designs that any honest multi-agent comparison must beat, and Artificial General Intelligence is where the claim that scaffolding substitutes for capability is adjudicated. AI Governance inherits the accountability problem that arises when a failure belongs to no single component. One adjacency is missing from this map rather than from the world: the classical multi-agent-systems tradition of contract nets, auctions and mechanism design, which is the natural language for agent economies and which the research pass behind this brief could not reach, because the venues holding it were unavailable. That gap is recorded here rather than papered over.

Handwave “Multi-agent systems outperform single agents.” This is the field’s central marketing claim and its own literature does not support it. The most-cited failure analysis opens by saying gains on popular benchmarks are often minimal; the cleanest controlled comparison found that a strong-prompt single agent matches the best discussion method; and multi-agent discussion beat single-agent only where the prompt contained no demonstration. Established The correct statement is that multi-agent configurations sometimes win on specific benchmarks against baselines that were not compute-matched, and that when a strong single-agent baseline is used the advantage largely disappears.

Speculative “More agents is all you need.” The paper of that name reports sampling-and-voting, which is what self-consistency does under another name, and it ties the gain explicitly to task difficulty. Established Meanwhile Chen and colleagues show that performance can decrease with more calls, because easy queries improve while hard queries degrade. Frontier A title that generalises a sampling result into an architectural principle is how a field acquires a misconception in one publication cycle.

Frontier “Agent count scales like model parameters.” MacNet’s own reported growth pattern is logistic, not power-law, and the finding comes from the paper that most enthusiastically presents itself as establishing a collaboration scaling law. Established Logistic means saturating: the curve flattens, and adding agents past the inflection buys progressively less. Frontier A reader who takes “scaling law” to imply the unbounded returns of the Kaplan and Chinchilla results has drawn the opposite conclusion from the one the paper reports.

Established “Debate makes models more truthful.” The well-supported result is that debate makes weak judges more accurate: 76% against 48% for non-expert models, 88% against 60% for humans. Frontier That is a claim about the judge, not about the debaters, and it belongs to the scalable-oversight literature rather than to the capability literature. Speculative Whether the effect survives when the debaters are adversarial, or grows with the capability gap, is untested.

Frontier “Generative Agents demonstrated emergent intelligence.” It demonstrated emergent believability, which is what the ablation on observation, planning and reflection actually measured, in a sandbox of 25 agents. Established The Valentine’s-party sequence is a qualitative demonstration and an excellent one. Speculative It is not a capability result, and no measurement in that paper compares the society against an individual agent on a task.

Established “Mixture-of-Agents beats GPT-4o.” On AlpacaEval 2.0, at 65.1% against 57.5%, aggregating multiple open-source models, without compute matching. Frontier AlpacaEval 2.0 is an LLM-judged preference benchmark, so the quantity being beaten is a preference score rather than a correctness rate, and the aggregate system consumes several models’ worth of inference to do it. Speculative The result is genuinely interesting for a different reason than the one it is usually cited for: it is the best available hint that heterogeneity, not agent count, is where the gain lives.

Frontier “Vendor X reported an N% improvement from multi-agent.” The most widely cited such claim could not be verified for this brief because the source was inaccessible during the research pass, and no number from it is printed here. Established The general requirement stands regardless of vendor: the evaluation set, whether it was internal, the baseline’s configuration, the token-cost multiple, the query count, and whether the evaluation existed before the system was built. Speculative An internal evaluation against an unspecified single-agent baseline is a product claim, and treating it as a research finding is how this subject acquired its reputation.

Frontier “Coordination topology is a major lever.” The measured effect in the one benchmark that compares topologies systematically is 3% on milestone achievement, reported without an interval, alongside a finding that graph structure beats star, chain and tree in one scenario. Speculative MacNet’s companion claim that irregular topologies outperform regular ones is interesting and may be real. Handwave Neither result is interpretable until somebody publishes the noise floor of the harness they were measured on.

Established The skeptic-side error: “multi-agent approaches never produced anything.” OpenAI Five defeated the Dota 2 world champions in April 2019 after ten months of training; AlphaStar reached Grandmaster in StarCraft II above 99.8% of ranked players through league training; hide-and-seek self-play produced six distinct emergent strategy phases. Established Multi-agent training has an excellent track record. Frontier It is multi-agent LLM inference at deployment time that has the weak one, and conflating the two produces bad arguments in both directions.

Frontier The version-drift misconception, which is specific to this literature. Papers in this area are living documents: MetaGPT ran to a seventh version, ChatDev to a fifth, MASLab was revised a year after release, and in the neighbouring literature Google’s co-scientist changed both its title and which validations its abstract carried between versions. Established A citation to an identifier without a version may not describe the paper the reader fetches. Speculative For a multi-agent framework, where the system is the paper, that is not a bibliographic inconvenience but a reproducibility problem.

Speculative The last and most important one is a misconception this brief could itself create. Nothing above proves that multi-agent systems cannot outperform single agents. It establishes that the comparison which would show it has not been made public, that the field’s own builders describe the gains as minimal, and that the mechanism which would make a collective provably superior — error decorrelation — is unmeasured. Frontier Discovery for the research pass behind this brief was constrained: two major literature-search interfaces were unavailable and several publisher domains refused to serve, which biases coverage toward well-known work. Handwave So the honest terminal position is a declared tie with a strong prior: on published evidence the deflationary reading wins, and it will keep winning until somebody runs two cheap experiments that nobody has any career incentive to run.