Why we render counter-evidence by default

The principle

Counter-evidence is a feature, not an afterthought. Disconfirming voices are surfaced alongside confirming ones, on every finding card, on every slide, in every section of the report.

The discipline is structural. The tool will not render a finding with a populated supporting-quotes block and an empty counter- quotes block unless the empty state is explicitly labelled “tested against the contrary view; none surfaced in this dataset.” A finding without that label and without counter quotes is not a shippable finding.

Why most tools don't

Two reasons. First, it is harder. Counter-evidence requires a second pass that explicitly looks for what the synthesis would prefer to forget. Most pipelines stop after the first synthesis pass and render whatever it produces.

Second, the social pressure inside a research-to-deliverable workflow rewards a clean answer. A finding card with three supporting quotes and no counter quotes reads as confident. The same card with three supporting quotes and two counter quotes reads as nuanced — which a hurried stakeholder will experience as “mixed” or “weak.” The path of least resistance is to omit the counter quotes.

We refuse to take the path of least resistance, because the omitted counter quote is the one that bites in stakeholder review six months later.

What it looks like in practice

A worked example, anonymised. A subscription-services discovery study, members cohort, 12 participants. The research question: “How does the annual price-change notification email affect members' trust in the provider?”

The synthesis stage drafts this finding: members trust the provider more after the price-change notification, because the email reads as transparent and well-justified. Prevalence 4 out of 5; intensity 3 out of 5; tier recurring. Eight participants of twelve described the email in positive terms.

Supporting quotes are attached: P03 (“they actually explained why, and that meant a lot”), P07 (“I appreciated the heads-up — most companies just send a new invoice”), P11 (“it felt like they respected my time”).

The counter-example-search pass then runs. It surfaces two quotes the synthesis missed. P04: “I felt like I was being managed.” P12: “the email made me question whether they actually care, or whether this was a script.” Both go on the finding card. Both go on the slide. Both go in the report.

The finding stands. Most members do trust the provider more after the notification — that is what the prevalence and tier say. But the counter quotes are visible, and any reader who wants to ask “what about the dissenting voices?” has the answer on the page.

What an empty counter-evidence list means

Sometimes the search returns nothing. Twelve participants, all twelve consistent, no contradicting quote anywhere in the corpus. That is a real outcome, and the tool says so explicitly.

The card renders an empty counter-evidence block with the label: “tested against the contrary view; none surfaced in this dataset.” The reader sees the search ran, sees what the search was looking for, and sees that nothing was found. The absence is named.

The label is not optional. A counter-evidence block left silently empty is indistinguishable from a finding where no search ran. The label distinguishes them.

How the tool surfaces counter-evidence

Two passes. The synthesis pass asks the model for both supporting and disconfirming quotes against the draft answer. That is the cheap pass — the model already has the relevant themes loaded — but it is also the biased pass, because the model is anchored on the answer it just drafted.

The dedicated counter-example-search pass is the discipline. It runs separately, with a prompt that explicitly looks for quotes that contradict the finding. It scans the full corpus, not only the quotes the synthesis cited. Anything it surfaces that the synthesis missed gets attached to the finding card.

Both passes are versioned, both passes write a stage_run row, and both passes are part of the back-traceability chain. The auditor sees not only the finding but the search that tested it.

Read more

Every claim links to a quote in a moment in time — the structural chain that makes findings reconstructible.