Lesson 0.4.40.4.4 Reasoning traps, and using AI to check AI

In everyday life. You visit a city during the one sunny week of its year. You tell everyone the weather there is wonderful. Your sample was real. But it was not typical.

What you will learn. Common reasoning mistakes, called traps, the ones that AI makes worse, and how to use AI to find the weaknesses in its own answer.

Why it matters. Good writing makes old mistakes harder to see. In the memo, several mistakes combined into one very large error.

Traps in any reasoning

Which traps should you learn first?

A trap is a reasoning mistake that is easy to make and hard to notice. Learn these five first. The memo contains four of them.

Which other traps are common?

• Two things happen together, so one caused the other. This is also called correlation as cause. Late payments increased after a new system arrived. Did the system cause it? Ask: what else changed at the same time?
• Looking only for agreement. This is also called confirmation bias. You search only for sources that support what you already believe. Ask: what would a person who disagrees show me?

Which traps are in the memo?

Brightline's invoice export shows that September includes one invoice for $148,500, from a vendor that gives no discount. That one invoice is most of the month. The treasury note says it is a one-time payment to buy leased equipment, and that no similar payment is planned.

Figure 0.4.5. The memo multiplied September by 12. But 84% of September is one invoice that will not happen again.

• Trap 1. The yearly number is built on an invoice that will not happen again.
• Trap 3. The memo applies 2% to every vendor. Only four of the seven vendors give any discount, and two of those give 1%.
• Trap 4. The memo gives a monthly saving to the cent, from a rough guess.
• Trap 5. The memo says paying early costs nothing. But paying early can mean borrowing at 9%.

In the lab you will find the real number from the invoice data. It is a small part of the memo's number.

Traps AI makes worse

Why does AI make some traps worse?

Knowing the names helps you notice a trap. Only checking protects you.

Using AI to check AI

How can AI help you find what is wrong?

Ana pasted the memo into a new chat. She asked: "Give the three strongest reasons this recommendation could be wrong. For each one, say what data would answer the question." The reply named the one-month sample, the different discount rates and the cost of cash.

These requests work well:

• Ask for the strongest objection. An objection is a reason that something may be wrong. Asking for objections deliberately is called red-teaming.
• Ask what would change the conclusion. This gives you checkpoint question 4.
• Start a new chat. A new chat does not contain your earlier wording, so the AI does not stay with its first answer.
• Ask what it cannot know. "What could you not know from what I gave you?" The reply often names the data you need to check.
• Ask for both sides. Ask it to argue for the recommendation, then against it. Then ask which points are true in both arguments.
• Compare two models. Ask Claude and ChatGPT the same neutral question. Then read the two answers three ways. Where they agree, look there first. Where they disagree, you have found something to check. Then ask what is missing from both. That gap is where your own knowledge matters most.¹

What are the limits?

• When two models agree, that is not evidence. They learned from much of the same text, so they can share a mistake.
• An objection from AI is a point to check. It is not yet a fact. Check it against the data.
• AI can argue well for any side. If both sides sound strong, the question is still open. It does not tell you who is right.
• If you ask the same chat "Are you sure?", you usually get the same answer again.

Later in this book. In Part III, AI workers check each other's work automatically. One agent writes, and a second agent looks for what is wrong. The same limits apply there. The second agent can share the first agent's mistake. Its objections are still points to check against data. And a person still owns the decision.

Key words

• Trap: a reasoning mistake that is easy to make and hard to notice.
• Objection: a reason that something may be wrong.
• Red-teaming: asking for objections deliberately.
• Anchoring: staying with the first answer you saw.