Nasr Khan

17 AI

AI advice check

What changed in my thinking after the AI's recommendation, and which of those changes have new evidence behind them?

For CEOs, investors and strategy leads using AI on judgment calls. Takes about ten minutes, split across before and after you read the AI's answer. Opens with an example.

Your view, before and after

1Before you read the AI's answer

Picking a type fills example options and criteria. Edit them to match yours.

Options
What matters, and how much

Weight 0 means it doesn't count. 5 means it matters most.

Score each option, 0 to 5
2After you read it
What you changed, and why

One line per change to a weight or a score. Say where it came from. If it's new evidence, name it.

The answer

Check first

Each option, before and after

What changed, and what each change did

The note for the board or investment committee

Before you sign it off

    Take the result with you

    A PDF report with your inputs, results, charts and method. Or the same as plain text for an email.

    How this tool works

    What it does

    In an Anthropic study, analysts working with an AI agent went with the agent's lean on 86% of judgment calls. Only 15% credited the agent for the arguments they used.

    This tool keeps a record you can check. You lock your view before you read the AI's answer. Then you log each change and where it came from. The tool shows which changes decided the outcome, and whether new evidence backs them.

    What you enter

    • Before. The decision, the options, what matters and how much, a score for each option, your pick, and what would change your mind. Then lock it.
    • After. The AI's recommendation, each change you made to a weight or a score with its source, and your decision.

    What you get

    • Each option, before and after, on one chart.
    • The changes that decided it, and which have no new evidence.
    • A short note for a board or investment committee, and a PDF.

    Try it: change the source of "AI features" to new evidence and name it. The verdict changes.

    The maths, assumptions and sources

    Each option's score = the sum of weight × score across what matters, ÷ the sum of weights, on a 0 to 5 scale. Changes apply in order. A change "decided it" when undoing it alone, with the others kept, gives a different winner. The evidence-only view keeps only changes marked as new evidence with the evidence named. The lock records the time in your browser; it isn't a signed record.

    This tool can't tell whether a change of mind was right. It shows what changed, what decided it, and what's left to judgment.

    • Anthropic, How AI agents shape experts' judgment calls (27 October 2026): 50 analysts matched the agent's randomly assigned lean 86% of the time across four judgment calls; 77% repeated its arguments and 15% credited it. The authors note one task, one field and deliberately strengthened leans.
    See answer