Unsupported claims

Symptom

The output states facts or conclusions that the supplied or retrieved evidence does not support.

Signals to look for

• A precise figure, name, or date appears in the output but in none of the supplied or retrieved passages.
• A citation is real, but the cited passage does not say what the claim says.
• The output answers confidently when retrieval returned nothing relevant.

Reproduce and diagnose

1. Keep the failing input, output, and the passages the workflow supplied or retrieved.
2. List each material claim and mark it supported, partly supported, or unsupported against those passages.
3. Check the retrieval log: was the right passage never retrieved, retrieved but ignored, or absent from the source set?

Never retrieved points to retrieval or source coverage. Retrieved but exceeded points to the generation instructions or a missing claim check. Absent from the source set means the task needs abstention, not a better prompt. A more detailed prompt is not a substitute for repairing the missing condition.

Correct the workflow

• Separate claims from evidence.
• Add traceable grounding.
• Require abstention when support is absent.

Confirm the fix

• Pass: a question whose answer is not in the source set produces an abstention. Fail: any substantive answer.
• Pass: a seeded passage containing a figure yields that exact figure with its citation.
• Pass: a real but irrelevant citation is rejected by the claim-to-passage check.
• Pass: the original failing input now produces a supported answer or an abstention.

Record what changed, which tests prove it, who reviewed it, and what would reopen the issue. One passing example is not enough.