Review Quality Audit

Bottom line: some gene reviews were produced by a generation pass that filled every
annotation's reasoning from a handful of templated strings and attached the same
placeholder evidence to every row, so the actions looked plausible but carried no real
curation. We wrote scan_boilerplate.py, which scans every review and ranks this
defect in three tiers (placeholder evidence; templated summary and reason; templated
reason only), after the pattern was first found and fixed on mouse Fyn. We did this
because a review whose reasoning is boilerplate cannot be audited, and the problem
concentrates in large hub genes where it does the most damage. The first run flagged
51 of 2,801 files: 4 Tier 1 (mouse/Egfr, mouse/Grb2, mouse/Cbl, mouse/Egf) and 13 genuine Tier 2
reworks, all of which have since been re-reviewed, plus 34 low-severity Tier 3 files.
A 2026-10-05 re-run over 5,625 files still finds Tier 1: 0, but now reports 4
Tier 2 candidates and 31 Tier 3 files. The 4 Tier 2 hits are dominated by repeated
GO:0005515 protein binding or localization rows whose duplicate rationales may be
legitimate; detector tuning and CI gating are tracked in
#4010.

The defect

A genuine annotation review needs term-specific reasoning and supporting_text
that actually supports that term. Some reviews instead carry, on every
annotation:

• a review.reason drawn from a tiny templated set (e.g. "Supported secondary
  or context-specific role", "Directly supported core function, process, or
  location"), and
• a supported_by whose supporting_text is a generic placeholder — literally
  "… deep research report reviewed for GO term specificity, core-function
  context, and evidence synthesis." — that says nothing about the specific
  claim, plus the same UniProt FUNCTION blob repeated verbatim regardless of the
  term.

The action labels (ACCEPT / KEEP_AS_NON_CORE / …) may be roughly sensible, but
the reasoning and evidence are not real curation. This was first found and fixed
on mouse Fyn (247 annotations, all templated; it also contained
contradictory REMOVE/ACCEPT pairs on the same core terms). This audit asks how
widespread the pattern is.

Detector

REVIEW_QUALITY_AUDIT/scan_boilerplate.py
scans every genes/**/*-ai-review.yaml and flags three tiers:

• Tier 1 (critical): ≥ 3 annotations carry the generic placeholder
  supporting_text. The evidence is fake/non-specific — the same defect class
  as Fyn. These should be re-reviewed (or reverted to unreviewed stubs).
• Tier 2 (candidate rework): both the summary and the reason are drawn
  from a tiny templated set (unique-summary ratio ≤ 0.15 and unique-reason
  ratio ≤ 0.15). This usually means the per-annotation rationale carries no real
  curation signal even though the supporting_text may be a real quote, but large
  repeated imports of the same GO term can mimic the same signal.
• Tier 3 (reason-only, low severity): only the one-line reason is
  templated; the summary is term-specific and supporting_text is a real
  quote. The substantive review is genuine — only the reason field is lazy.
  Low priority; can be tightened in bulk later.

uv run python projects/REVIEW_QUALITY_AUDIT/scan_boilerplate.py \
    --genes-dir genes --out-dir projects/REVIEW_QUALITY_AUDIT/reports

Outputs (regenerated, not hand-edited):
REVIEW_QUALITY_AUDIT/reports/REPORT.md
and boilerplate_flags.csv.

Findings

The first run flagged 51 of 2801 review files: 4 Tier 1, 47 Tier 2.

Tier 1 — critical (fake placeholder evidence) — RESOLVED

All four were mouse genes from the same generation pass that produced Fyn, with
the identical templated reason set and placeholder evidence. All four have now
been fully re-reviewed (genuine per-term actions, summaries, reasons, and
verified supporting_text), so Tier 1 is now empty:

(Fyn was re-reviewed first and seeded this audit.) Re-running the scanner now
reports Tier 1: 0.

Tier 2 — genuine rework and current candidates

Both summary and reason templated — no real per-annotation curation signal.
The first 13 genuine hits have been re-reviewed: full per-term action/summary/reason regenerated
with the real supporting_text preserved.

• Batch 1: mouse/Mtor (372), human/YWHAZ (220), mouse/Nf1 (195),
  human/NCSTN (99).
• Batch 2 (Alzheimer-risk): human/SORL1 (168), human/ADAM10 (151),
  human/ABCA1 (145), human/FERMT2 (85).
• Batch 3: mouse/Brca1 (167), mouse/Tert (143), mouse/Ccnt1 (50),
  yeast/NTE1 (17).
• mouse/Ctnnb1 (759) — the largest review in the corpus; its 295 unique
  terms were curated by four parallel agents split across β-catenin's roles
  (adhesion/structure, Wnt/transcription, and the developmental-phenotype tail),
  then merged with exact disjoint coverage.

A live 2026-10-05 scan now reports four Tier 2 candidates:
human/GRB2, human/SPRED1, human/CASP6, and human/SOS1. All four are dominated
by repeated rationales on duplicate GO:0005515 protein binding, cytosol, or
high-throughput interactome rows, so they should drive detector tuning before
bulk re-review (#4010).

Tier 3 — reason-only boilerplate (31 live files, low severity)

Only the one-line reason is lazy; the summary is term-specific and the
supporting_text is a real quote, so the substantive review is genuine. This
group includes many large hub genes that look templated by reason alone but
are actually fine — e.g. mouse/Bcl2, mouse/Pten, mouse/Hsp90aa1, the
calmodulins Calm1–3, Drosophila Notch/Delta, and human FGFR/KASH5 adaptor or
interaction-heavy reviews. Low priority.

The flagged genes are heavily weighted toward large, pleiotropic hub genes —
exactly the genes with the most annotations, where templating saves the most
effort and where over-annotation is most likely.

Recommended actions

1. Tier 1: re-review (as was done for Fyn) — these carry no usable evidence.
   The action labels can seed the re-review, but every reason and
   supporting_text must be regenerated against real sources.
2. Tier 2: inspect the current detector hits, then tune the scan so repeated
   rows for the same GO term or the same high-throughput source do not trip the
   same signal as fully templated reviews (#4010).
3. Tier 3: tighten the low-priority one-line reasons in bulk.
4. Treat the placeholder string and the tuned low-ratio checks as a CI smell
   test for future review submissions once the duplicate-row false positives
   have been separated.

Related projects

• OVER_ANNOTATION_PATTERNS — the substantive
  over-annotation patterns these hub-gene reviews tend to contain.
• BEHAVIOR — behaviour over-annotations; several flagged genes
  (Mtor, Hsp90aa1, Pten, Drd1) also carry behaviour terms.