Retrieval recall in production use (PAINT campaign, n=91)

The other cohorts in this project evaluate what Affinage says — whether its GO
grounding matches the curated function, whether its narrative recovers what the
grounding drops. This one evaluates what it finds. Across 91 human genes with
a committed Affinage deep-research report, how much of the literature the
finished review depends on did Affinage actually supply?

Numbers here are generated by retrieval_recall.py;
see paint-campaign/summary.md for the per-gene
table and paint-campaign/per-gene.json for raw
counts. The cohort is pinned in
paint-campaign/campaign-genes.txt,
reconstructed from the originally committed per-gene.json. The earlier
instruction to re-run with --all no longer reproduces it: --all scores every
committed report, which was 139 genes on 2026-09-27 (recall 47%). Regenerate
this cohort with:

cd projects/AFFINAGE_EVALUATION
uv run python retrieval_recall.py --genes-file results/paint-campaign/campaign-genes.txt \
    --split-file fa-cohort-genes.txt --split-name FA

The figures below were regenerated at commit 943b98815 (2026-10-01). They differ
slightly from the first run (1626 → 1634 cited PMIDs, 718 → 726 novel, 52% → 51%
pooled) because some reviews have been edited since.

The 91 genes are two cohorts

The 91 were not all reviewed the same way. 22 are the Fanconi-anemia cohort
(fa-cohort.md): reviews that were revised specifically to fold in
the papers Affinage's narrative surfaced. Scoring Affinage's recall against those
reference lists partly measures that editing step. The other 69 are
PAINT-backlog genes where Affinage was simply the deep-research provider. The
first version of this page pooled them, which inflated every headline:

17 of the 22 FA genes score 100%. The non-FA figure is the one to quote.

Bottom line

On the non-FA genes, Affinage supplies about half the references a review has
to go find (48%), and its trust gates cannot tell you which half is missing.

The denominator matters. Counting every PMID in a finished review makes recall
look like 32%, but 908 of the 1634 references arrive prepackaged in the GOA
file — the reviewer is handed them and no retrieval is involved. Scoring a
retrieval provider against references it was never asked to retrieve measures
nothing. Restricted to the references the reviewer genuinely had to locate,
recall is 48% on the non-FA genes.

The complement — Affinage references the review never cited (41% on the non-FA
genes) — is the ordinary cost of a literature sweep and not by itself a defect.
The FA reports are long and dense (only 21% cited), which is why the pooled used
fraction looks lower.

Precision is gated; recall is not

gates_passed: True certifies that the citations in a report are real,
resolvable, and quoted correctly. That is a precision guarantee, and it holds up.
What it cannot certify is that the report found the papers that decide the
review, because nothing in the pipeline measures recall.

The gap is not hypothetical. Two verified cases from this campaign, both on genes
whose reports were otherwise clean:

• ADAMTSL1 — PMID:22242013 carries the only molecular-function experiment
  ever performed on the human protein, a measured SPR negative against both
  fibrillin-1 and ADAMTS-10. The finished review cites it seven times. It appears
  zero times in the Affinage report, and it is absent from GOA. It surfaced only
  because a sibling agent reviewing ADAMTSL3 read a paper titled for the partner.
• ACTG2 — PMID:38820162 (2024 cryo-EM of ACTG2 filaments) supplied four of
  five proposed new annotations and is cited 29 times in the review. Zero
  mentions in the Affinage report. It carries no GO annotations at all.

Both are the same shape: the decisive paper for a sparsely-annotated gene is
usually titled for something else — a partner, a complex, a paralogue, or the
well-characterised family member. A symbol-keyed search does not reach it, and a
measured negative, which is the most decision-relevant evidence available on a
dark gene, is almost never in the title.

Does recall depend on how well-studied the gene is? Not established

The first version of this page concluded that recall is flat across curation
depth (52% / 50% / 56%). It also argued that the well-characterised genes at the
100% end of the sorted table (RAD51C, RFWD3, SLX4, UBE2T, XRCC2) were a
small-denominator artifact. Both arguments missed that those five are FA genes,
as are 17 of the 27 genes at 100% and 17 of the 24 genes in the well-studied band.
Split by cohort:

Without FA, the well-studied band falls to 27%. That band is seven genes. One of
them, ACTR8, is an empty report carrying 15 of its 62 novel references; without
ACTR8 the band is 17/47 (36%). That is too few genes to claim recall falls with
curation depth, and equally too few to claim it is flat. What the data do support
is narrower: on dark and medium non-FA genes, recall is about half (51%, 49%).
The small-denominator point still applies to the remaining 100% genes: the ten
non-FA ones had one to seven novel references each.

The 0% end is not only dark genes either: five of the seven at the 0% end
(AADACL2/3/4, ACP7, ACTL10) have no GOA references, but by the script's own
banding ACTR8 has 12 GOA references (well-studied) and ACTR1B has 8 (medium).

What differs across the bands is at least the consequence of a miss. On
a well-studied gene, half the literature still leaves several independent papers
establishing the same function, and the review survives. On a dark gene, half the
literature can mean the difference between one functional experiment and none —
which is exactly what happened with ADAMTSL1 and ACTG2.

Silent empty returns

Six reports returned zero PMIDs: AADACL2, AADACL3, AADACL4, ACP7, ACTL10, ACTR8.
Their reviews went on to cite 6, 3, 4, 1, 2 and 27 references respectively, all
found by other means.

Two features make this worse than a low score. The AADACL cluster is an entire
paralogue family returning nothing, suggesting the failure attaches to a
neighbourhood of sequence space rather than to individual queries. And an empty
report is not flagged: ACTL10's carries gates_passed: True, which is
technically correct — a report with no citations has no false citations — but
reads as success. A gate that passes vacuously on empty input is a gate worth
changing.

Only one further gene, ACTR1B, received a non-empty report that supplied none of
its novel references — so outside the six empty returns, a report that contains
citations at all almost always contributes at least one.

Recommendations

Treat Affinage as a strong first pass, not as the literature search. It front-loads
the obvious references cheaply. 57 of the 69 non-FA reviews cite it as a source,
so the reports are used rather than filed and ignored. None of the 22 FA reviews
cites its report as a file: source; their Affinage provenance is recorded in
fa-cohort/ instead.

For any gene where the review turns on a small number of experiments, search
independently on partners, complexes, family members and paralogues, and run
QuickGO by reference on load-bearing PMIDs to see what else they annotate. This
is already standing practice for the per-gene review agents in this campaign and
is what surfaced both documented misses above.

Two changes would make the provider's own output more honest without touching
retrieval: report an explicit zero-citation warning rather than passing gates
vacuously, and record in the gene notes when the reviewer found a load-bearing
reference the provider missed. Recall cannot be measured at all unless the misses
are written down somewhere.

Limits of this analysis

48% (non-FA) is an upper bound, for two reasons, and the second is the larger one.
The FA cohort's 85% is not a recall estimate at all (see above).

The reviews were written with the Affinage report in hand — 57 of the 69 non-FA
reviews cite it as a source. The reference set being scored is therefore partly caused by the
thing being scored: a paper Affinage surfaced is more likely to end up in the
review than an equally relevant paper it did not, so the numerator is enriched by
construction. Measuring recall against a reviewer who read the report is not the
same as measuring it against an independent literature search, and the honest
reading of 48% is "of the references that ended up mattering, Affinage had already
supplied about half" — not "Affinage finds half of what is findable." A clean
estimate would need reviews written blind to the report.

Recall is also measured against a single finished review per gene, which is an
imperfect standard in the other direction: a reference neither Affinage nor the
reviewer found is invisible here.

Reference sets are compared by PMID string matching, so a paper cited in the
report by DOI or title alone would score as a miss. In this cohort that caveat
never bites — no Affinage report cites by DOI anywhere.

The 91 scored genes are those that had a committed Affinage report when this
analysis was first run (now pinned in campaign-genes.txt), not a random sample
of the campaign. Genes where the provider errored and the reviewer moved
on without committing a report are absent, which biases the sample toward
successful runs.

Gate statistics are thin: only 33 of 91 reports carry a gates_passed field at
all (29 True, 4 False), the remaining 58 predating the field.