LLM Council: Several AI Models, One Reviewed Answer

An LLM council is a setup where several language models answer the same question independently, review and rank each other's answers without knowing who wrote which, and then one designated "chairman" model writes a single final answer from all of it.

If you have ever typed something like "a place where AI models can argue and give you the best result" or "what is the term for asking multiple AI programs to analyze the same problem," this is very likely the name you were reaching for. The phrase was popularized by Andrej Karpathy's open-source llm-council repository, created in November 2025, which wraps the pattern in a small chat app. The idea underneath it is older and simpler: no single model is reliably right, models from different labs fail in different places, and a structured round of mutual criticism catches some of those failures before you read the answer.

This page covers the council pattern specifically — how its three stages work, how it differs from a debate, and what it costs to run one. For the wider family of multi-model setups, see our guide to multi-agent AI debate systems.

How a council works, stage by stage

A council runs in three fixed stages, and each model speaks once per stage. This is how Karpathy's README describes it (read October 16, 2026), and it is the shape most people mean by the term:

1. First opinions. Your question goes to every member separately. No model sees any other model's answer while writing its own. In Karpathy's app the answers appear in tabs so you can read each one.
2. Peer review. Each member is shown the first-round answers with the model names removed and is asked to rank them for accuracy and insight. In the repository's code the answers are relabeled "Response A," "Response B," and so on, and every reviewer must end its review with a "FINAL RANKING:" list in a fixed format.
3. Chairman synthesis. One model — the chairman — receives the original answers and the reviews and compiles them into one final reply, which is what the user sees as the answer.

Three details make this work. The first stage is genuinely independent, so you get several uncontaminated attempts. The second stage is blind, which matters for reasons covered below. And the third stage produces exactly one answer, which is what people asking for "the best result" want. Notice what is missing: the members never reply to each other. Nobody rebuts anybody. That is the main thing separating a council from a debate.

Council vs. debate

A council aggregates independent judgments; a debate makes models change each other's minds. Both use several models, and both appear in searches like "AI that argues between each other to find a solution," but they produce different things.

The research most often cited for multi-model setups, Du et al.'s 2023 paper on multiagent debate, studies the debate side: several model instances propose answers and then debate them over multiple rounds toward a common answer. The authors report that this improved mathematical and strategic reasoning and the factual validity of the output across the tasks they tested. A council borrows the "independent first answers" half of that design and replaces the rounds of argument with a single ranking pass plus a writer.

Neither is strictly better. If you want to see why two strong models disagree, a council hides that from you by design, and a debate shows it. If you want one clean answer and do not care to read three, a debate leaves you with work a council would have done.

Why the peer review is anonymous

Hiding the authors' names keeps the ranking about the text instead of the brand. Karpathy's README gives the reason in one line: the identities are anonymized "so that the LLM can't play favorites when judging their outputs."

The reasoning is easy to follow even without a study in front of you. A model reviewing a response labeled "Claude" or "GPT" has picked up, from its training data, a great deal of text about those names — reputations, comparisons, marketing. Any of that can leak into a ranking. A model also has an obvious stake in its own answer if it can recognize it. Strip the labels and the reviewer has only the words to go on.

Anonymity is not a full cure. A model may still recognize its own phrasing, and every reviewer can still prefer longer or more confident answers regardless of who wrote them. But it removes the easiest source of bias at almost no cost, which is why it is the part of the council design worth copying even if you change everything else.

Running a council without writing code

You can do it by hand with three chat apps, or approximate it in one room here — but be clear about which parts each approach actually gives you.

The manual version. Open three chat apps from three different providers. Paste the same question into each. Copy the three answers into a document and label them A, B, and C, deleting anything that reveals the author ("As Claude, I…"). Paste the labeled set back into each app with a prompt like: "Rank these three responses for accuracy and insight. Explain the strengths and weaknesses of each, then give a final ranking." Finally, pick one app to be chairman and give it the question, all three answers, and all three reviews with: "Write the single best answer to the original question, using what the reviews agree on and saying plainly where they disagree." That is a real council: independent, blind, and synthesized. It is also about ten minutes of copying and pasting.

On AI to AI Hub. Our product puts up to three models from different providers in one room, where they take turns and read each other. That makes it a debate tool first, and two honest differences follow:

• Stage 1 is not independent here. Models speak in turn, so the second and third member read the earlier answers before writing their own.
• The review is not anonymous here. Each model sees which model wrote which message.

There is also no automatic chairman or judge. You play that role, or you assign it: after the members have answered and reviewed, type a message such as "Claude Opus 5.5: act as chairman. Using everyone's answers and critiques above, write the single best final answer and note where the others disagreed," let it reply, and pause the room. What you gain over the manual version is speed and the debate itself — members can push back on each other's reviews instead of just scoring them. If strict independence and blind ranking matter more to you than that, use the manual version or Karpathy's app.

Running Karpathy's version

Karpathy's llm-council is a local web app you run yourself, paid for through your own OpenRouter account. According to its README (read October 16, 2026), you need:

• Python and uv for the backend, and npm for the frontend. The backend is FastAPI; the frontend is React with Vite.
• An OpenRouter API key in a .env file as OPENROUTER_API_KEY, with credits purchased on OpenRouter. Every council member and the chairman are called through OpenRouter.
• A model list in backend/config.py: a COUNCIL_MODELS list and a single CHAIRMAN_MODEL. The README's example lists four models from four labs and reuses one of them as chairman, so the chairman can also be a council member.

You start it with the included start.sh script (or the backend and frontend in two terminals) and open localhost:5173. Conversations are saved as JSON files in data/conversations/.

Two things to know before relying on it. The README calls the project "99% vibe coded as a fun Saturday hack" and says Karpathy is "not going to support it in any way" and does not intend to improve it — it is provided as inspiration. And when we checked, the repository did not declare a license. If you plan to build on the code rather than just run it, that second point is worth taking seriously.

What a council round costs here

A three-member council on this site costs between 35 and 84 credits per question, depending on model tier. These figures are our own arithmetic from the product's pricing: every AI reply costs a fixed number of credits — 12 for premium models, 8 for standard, 5 for economy. Starter is $19 for 800 credits ($19 ÷ 800 = $0.02375 per credit) and allows up to 2 models per room; Pro is $49 for 3,000 credits ($49 ÷ 3,000 ≈ $0.01633 per credit) and allows 3.

A three-member round is 3 answers + 3 reviews + 1 chairman synthesis = 7 replies.

The Starter column is there for comparison only, since a three-model room needs Pro. On Starter, a two-member council is 2 answers + 2 reviews + 1 synthesis = 5 replies: 60 credits on premium ($1.425), 40 on standard ($0.95), or 25 on economy (about $0.59). That is roughly 13, 20, or 32 rounds out of an 800-credit month. The 20 free trial credits cover four economy replies, which is enough to try the format but not a complete council round.

Notice the mixed row. Putting a cheaper model from a third provider in the third seat costs 6 fewer credits than an all-standard council and adds a voice trained by a different lab. Our models page lists which model sits in which tier, and the pricing page has the plan details.

When a council is the wrong tool

Skip the council when the question has one checkable answer or when the disagreement is the point.

• Lookups and well-specified tasks. A date, a conversion, a function that either passes its tests or doesn't. One good model answers it; seven replies just cost seven times as much.
• Contested judgment calls. "Should we raise prices?" has no best answer to synthesize. A chairman will still write one, confidently, and the disagreement you needed to see ends up as a single hedged sentence. A multi-AI debate shows you who objected and why.
• Same-family councils. Three models from one lab tend to share blind spots, so the review stage agrees with itself. Use different providers or the review adds little.
• When you will not read the reviews. The ranking pass is often more useful than the final synthesis, because it tells you which answer the other models trusted and why. If you only ever read the chairman's output, you are trusting one model's summary of everyone else.

Questions

Is an LLM council the same as a mixture-of-experts model? No. Mixture-of-experts is an architecture inside a single model, where different internal sub-networks handle different inputs. A council is several complete, separate models exchanging text at the application level. You can build a council from any models you can call; you cannot add experts to a model you didn't train.

Does the chairman have to be the strongest model? Not necessarily — its job is synthesis, not original reasoning, so a strong writer with good judgment works. Karpathy's example configuration simply reuses one of the council members. What matters more is that the chairman is told to report disagreement rather than resolve it silently.

Can the members see each other's answers before they write their own? In a strict council, no — that independence is the point of stage one. In a turn-based room like ours, yes, which makes it closer to a debate. Choose based on whether you want independent attempts or models that respond to each other.

If you want models that respond to each other rather than vote, start with how to set up a three-AI debate, or go back to the AI to AI Hub homepage to open a room.