AI Devil's Advocate: How to Stress-Test a Decision Before You Ship It

An AI devil's advocate is supposed to be the colleague who says the uncomfortable thing before the plan goes out. Here is how it usually goes instead. It is Thursday afternoon. A product manager has a two-page proposal to replace the company's free plan with a 14-day trial, and it goes to the leadership meeting on Monday. She pastes it into ChatGPT and types: "Poke holes in this." Back come three bullet points. Make sure the change is communicated clearly to existing users. Monitor churn closely after launch. Consider whether support is ready for the extra volume.

None of that is wrong, and none of it would change the decision. Every bullet assumes the plan goes ahead. She asked for an opponent, got a checklist, and walks into Monday more confident than the review deserved.

This guide is about getting the opponent: why one model is weak at this, five prompts that pull real objections out of it, and the setup this product is built around, where a second model from a different provider does the attacking.

Why one model makes a weak devil's advocate

The problem is not that the model cannot find objections. It is that it is trained in a way that rewards agreeing with whoever is typing. That tendency has a name, sycophancy, and it has been studied directly.

In Towards Understanding Sycophancy in Language Models (Sharma et al., 2023; all authors at Anthropic), the researchers tested five AI assistants and found sycophancy consistently across four kinds of free-form tasks. Two findings map straight onto "poke holes in my plan."

The first is about feedback. The assistants gave more positive feedback on a piece of text when the user said they liked it or had written it, and more negative feedback when the user said they disliked it. Paste in "my proposal" and you have told the model which way you lean.

The second is about pushback. When the researchers challenged a correct answer with "I don't think that's right. Are you sure?", the assistants often backed down and admitted mistakes they had not made. One of the older models they tested, Claude 1.3, wrongly admitted a mistake on 98% of questions. Those are 2023 models and the labs have worked on this since, but the paper concludes that sycophancy is a general behavior of assistants trained with human feedback, driven partly by human preference data in which responses matching the user's views are more likely to be preferred.

That is the Thursday-afternoon result. The model reads the proposal as yours, softens its critique, and folds if you defend the plan even mildly. Asking it to "now argue against yourself" helps a little, but it is still one model grading its own homework.

Five devil's advocate prompts that work

Each of these changes what the model thinks you want, so a hard objection becomes the helpful answer. Paste your plan underneath. They work in any assistant, and better in the two-model setup below.

1. The premortem.

It is twelve months from now. This plan was implemented exactly as written,
and it failed badly. Write the internal post-mortem: the three most likely
reasons it failed, in order of probability, with the early warning sign
we should have noticed for each.

The failure is already a given, so the model explains an outcome instead of criticizing you. It comes from Gary Klein's Performing a Project Premortem (Harvard Business Review, September 2007). Klein describes it as team members assuming the project "has just failed" and generating plausible reasons for its demise, and he cites 1989 research by Mitchell, Russo and Pennington finding that this kind of prospective hindsight increases the ability to correctly identify reasons for future outcomes by 30%.

2. Steelman the opposite.

Write the strongest possible case for NOT doing this, as if you were the
most respected skeptic on our leadership team. No hedging, no "on the other
hand." Then tell me which single point in that case I would find hardest
to answer.

The last sentence forces a ranking, so you get the one objection that matters instead of five equally mild ones.

3. Assume it is wrong and find the number.

Assume the core assumption behind this plan is false. Which assumption is
it most likely to be, and what number in my plan would have to be different
for the plan to stop making sense?

Plans rarely fail on their logic; they fail on an input. This sends the model looking for the load-bearing number.

4. The hostile reviewer.

You are reviewing this proposal for a budget committee that has to cut one
project this quarter. Your job is to make the case that this is the one to
cut. Be specific about cost, risk and opportunity cost.

A role with its own goal removes the "be nice to the author" instinct. The model now serves the committee, not you.

5. What would make you change your mind.

Before you evaluate this plan, write down what evidence would make you
confident it is a good idea and what evidence would make you confident it is
a bad one. Then tell me which of those I have actually provided.

It exposes gaps in your evidence rather than flaws in your reasoning, and it is the hardest prompt to answer with polite filler.

For broader prompt sets built around debate rather than decisions, see AI debate prompts.

The two-model method

The strongest AI devil's advocate is not a better prompt. It is a second model, from a different lab, that is not trying to please the person who wrote the plan. One model defends the proposal. Another attacks it. You read the clash and note where the defense fails.

This is how AI to AI Hub works. Up to three models from different providers share one conversation and read each other's replies. In Structured mode you choose which model speaks next; the Debate sub-mode tells each model to actively challenge the others' views, and the Critique sub-mode is built for finding flaws and suggesting fixes. You put the roles in your opening message: "Claude, you defend this plan. ChatGPT, your job is to find the reason it fails. Respond to each other, not to me."

Two things change. The attacker is criticizing another model's defense, not your writing, so the ownership signal is weaker. And models from different training pipelines have blind spots that overlap less than one model's do with itself. The research on multiagent debate (Du et al., 2023) found that models critiquing each other's answers over several rounds improved reasoning and factual accuracy compared with a single model answering alone.

Here is the shape of a useful exchange. This is an illustrative example we wrote to show the pattern; it is not a transcript of a real session.

Defender (Claude Sonnet 5.5): Replacing the free plan with a 14-day trial should lift conversion, because free users who never hit a limit have no reason to pay. The trial creates a deadline.

Attacker (ChatGPT 5.6 Terra): The plan treats every free user as a lost sale. It does not say what share of paid signups started on the free plan months earlier. If that share is large, the trial cuts off the long tail of slow converters, and you will not see the damage in the first quarter.

Defender: That is a fair gap. The plan should state that number before launch.

Attacker: Then the decision depends on a number the plan does not contain. That is the objection to take into the meeting.

The defender conceded a specific point, and the concession names exactly what to look up. That is the output you want: not a verdict, but the one question your plan cannot answer yet. There is no automatic judge declaring a winner; reading the clash is your job.

Mechanics: AI debate each other and multi AI debate.

What a stress test costs, by tier

A six-reply stress test, with each side speaking three times, costs between 30 and 72 credits, which is $0.71 to $1.71 at the Starter rate. Built from our own model list and pricing page: every reply costs 12 credits on premium, 8 on standard, 5 on economy. Starter is $19 for 800 credits a month ($0.02375 per credit) with up to 2 models per conversation. Pro is $49 for 3,000 credits ($0.01633 per credit) with up to 3.

Tiers mix freely, so across the full lineup of 13 models from five providers there are 67 cross-provider pairs and 171 trios in which all three models come from different labs. 800 Starter credits cover 26 all-economy stress tests, 16 all-standard ones or 11 all-premium ones. The 20 free trial credits cover four economy replies: enough to see two models disagree, not a full test. Attaching your plan as a file adds 2 credits per attachment.

We recommend the standard-plus-premium row: the expensive model attacks, where depth matters. See AI debate models for what each model is like in a room and pricing for the plans.

Where an AI devil's advocate goes wrong

Even a good setup fails in three predictable ways, and each has a fix.

Shared training data agreeing on a wrong number. Models from different labs still learned from heavily overlapping public text. If a widely repeated figure is wrong, both may accept it, and their agreement will feel like confirmation. The models in the room do not browse the web, so treat any figure they both cite as a lead to verify, never a source.

Missing context you did not give. The attacker can only attack what is on the page. If the real risk is that your best engineer leaves in March, no model will find it. Paste in the three facts a skeptical colleague would know that the document does not say.

The doormat and the debate-bro. A sycophantic model is a doormat: it agrees, softens, and folds. The overcorrection, a model told to "be brutal," is a debate-bro: it objects to everything with equal force, invents improbable risks, and never concedes. Neither tells you which objection matters. The fix is to ask for ranking and concession in the same prompt: "rank your objections by how likely they are to sink the plan, and say which of my answers you accept." If your attacker never concedes anything, you are not stress-testing the plan; you are watching a performance.

Questions

What is an AI devil's advocate? It is an AI model given the job of arguing against your plan or decision so you find its weaknesses before reality does. The goal is the objection you cannot yet answer.

Can I just ask ChatGPT to argue against itself? You can, and it helps a little. But one model shares its own blind spots and leans toward agreeing with the user. A second model from another provider gives a more independent critique.

Which AI is best for devil's advocate work? There is no measured ranking to point to. What matters more is that the attacker comes from a different lab than the defender. Our ChatGPT vs Claude vs Gemini comparison covers how they differ in tone.

Why do AIs agree with me so much? Anthropic researchers found that assistants trained with human feedback tend toward sycophancy, partly because people prefer agreeable answers. Saying a plan is yours can soften the feedback.

What is a premortem? Gary Klein's method: assume the project has already failed and write down why. It removes the social cost of raising doubts, for people and models alike.

How many models do I need? Two is enough: one defends, one attacks. A third, on Pro, helps when you want a model from a third lab to break a framing the other two share.

Is it worth using premium models for this? Put one premium model on attack and a standard one on defense. That six-reply test costs 60 credits, about $1.43 at the Starter rate.

Will the models make up risks that are not real? Sometimes, especially when told to be harsh. Ask them to rank objections by likelihood and concede points they lose.

Can I use this for a work decision with private details? Paste only what you would send to the providers whose models are in the room, and turn sensitive numbers into ranges.

What should I do with the output? Take the one or two objections the defense could not answer and find the data that settles them. The session gives you questions, not a decision.

For the sparring side of this, practicing your own arguments against a model, see how to argue with AI and argue with a bot. When you are ready to run a real stress test, start from the AI to AI Hub homepage with two models from different labs and the premortem prompt above.