Miscitation Audit

Bottom line: a citation whose identifier resolves to a different paper cannot
support the annotation attached to it, and reviewers here had been flagging such
cases one gene at a time without anyone aggregating them. We wrote
harvest_citations.py, which collects every reference_review.correctness defect
flag across the 4,467 review files and keys it on the citation rather than the gene,
and detect_citation_anomalies.py, which looks for defects nobody has flagged yet. We
keyed on the citation because one bad PMID is usually copied across paralogs or complex
partners, so one correction clears several genes and one discovery says where else to
look. The current register (REPORT.md) holds 28
WRONG_IDENTIFIER rows on 21 distinct citations, 6 of which carry that flag on more than one gene,
plus 257 MISCITED rows (215 citations) that have not been sampled for precision; the
prose below was written at 27 rows / 20 citations, one flag earlier. Most of the defects
came from GOA (233 of 285 flagged citations), so the main deliverable is a bug report to
the assigning groups, and those reports have not been filed yet.

We did this because the existing validators only check internal consistency (the title
matches the PMID, the quote is in the paper), and a wrong PMID imported together with its
own title passes both. The unresolvable-identifier check found exactly one dead PMID
across 24,436 cited; the paralog-mismatch check was measured and does not work.

The two classes are not interchangeable and this page keeps them apart throughout:
WRONG_IDENTIFIER (27 rows / 20 citations) is mechanically checkable and is the
tier to act on; MISCITED (257 rows / 215 citations) is a judgement call that has
not been sampled for precision.

Why the citation, not the gene, is the right key

A single bad citation rarely damages one gene. It is copied across paralogs of a
family or partners in a complex, so one upstream correction clears several genes at
once — and, more usefully, one discovery predicts where else to look.

Seven of the twenty distinct WRONG_IDENTIFIER citations are flagged on more than one
gene. For six the flag is WRONG_IDENTIFIER on every listed gene; PMID:23209302 is
WRONG_IDENTIFIER on NDUFA8 and MISCITED on ACOX1 and SLC25A3 (the committed
REPORT.md lists all seven because it predates a fix that restricts that list to
WRONG_IDENTIFIER genes):

Who owns the fix

This is the split that decides what the project is for. A citation that appears as
an annotation's original_reference_id came from GOA; one that appears only in a
review's references list was added here.

• 26 of the 27 WRONG_IDENTIFIER rows are GOA-sourced. Only one (ADPRH) is ours.
• Across all defect flags the ratio is 232 GOA / 52 ours.

So the primary deliverable is not an internal clean-up — it is a bug report to the
assigning groups (MGI, SGD, UniProt, GOA). The COX17 case found while reviewing the
mitochondrial copper delivery pathway is representative: a
protein farnesylation IDA on the copper chaperone COX17, citing a paper entirely
about COX10 (heme A:farnesyltransferase, one digit away), assigned by MGI
against a S. cerevisiae accession — and with a term that is wrong even for COX10,
since that enzyme farnesylates heme rather than protein.

Failure modes seen so far

Tooling

harvest_citations.py — the register

Aggregates every reference_review.correctness defect flag, keyed on citation.

uv run python projects/MISCITATION_AUDIT/harvest_citations.py

Outputs to MISCITATION_AUDIT/reports/:
citation_flags.tsv (one row per gene × citation, with the GOA/ours split),
bad_citations.tsv (one row per distinct defective citation) and REPORT.md.

Its most useful column is contamination spread: citations flagged
WRONG_IDENTIFIER in one review that are still cited without a flag elsewhere.
Seven such citations currently reach twelve unflagged uses.

Spread is a triage queue, not a verdict. The clearest illustration is
PMID:10970790, flagged wrong on ELOVL1/2/3 and cited unflagged on ELOVL5 — where
it is the correct citation, because ELOVL5 is what the paper actually characterises.
By contrast the two unflagged uses of PMID:34521819 (on JAK1 and STAT1) cannot be
correct, because the identifier resolves to nothing at all. Each row needs a human.

detect_citation_anomalies.py — finding what nobody has flagged

uv run python projects/MISCITATION_AUDIT/detect_citation_anomalies.py --check-pubmed

Check A — unresolvable identifiers. Works, and is nearly free. The insight is that
fetch-gene already caches every citation it can resolve, so absence from
publications/ is itself the signal; the network call only confirms it. Across the
whole repository exactly one cited PMID has no cached record — PMID:34521819 —
and NCBI confirms it returns no document summary. It is cited by three genes and was
flagged on only one. This check should run in CI.

Check B — paralog mismatch. Does not work; recorded so nobody rebuilds it. The idea
was to flag a citation whose cached text names some members of a numbered family but
not the gene citing it. Measured behaviour:

• False negatives from alias drift. It misses the motivating ELOVL case entirely:
  PMID_10970790.md contains zero occurrences of "ELOVL" and three of "HELO1", the
  historical name. Symbol matching cannot see through alias history.
• False positives from complexes. Its 356 candidates are dominated by legitimate
  complex-wide papers — CHMP/ESCRT, the mitochondrial proteome across COX and NDUFA
  subunits, PEX, EMC, VPS — where broad citation is correct and the cached abstract
  simply does not name every subunit.

Bare co-citation across a family is normal and is not a defect signal. A working
version of this check would need alias-aware matching (UniProt/HGNC synonym lists)
rather than literal symbols. Output is retained in paralog_mismatches.tsv and
family_clusters.tsv but should not be treated as findings.

Scope and honest limits

• The 20 WRONG_IDENTIFIER citations are the high-confidence tier and the right
  place to start: "this identifier resolves to a different paper" is checkable.
• The 215 MISCITED citations are a second tier. "Right paper, does not support the
  claim" is a judgement call, some will be defensible on re-reading, and the class has
  not been sampled for precision. Do not report these upstream without re-checking.
• Everything here is harvested from reviewer judgement, so it inherits reviewer
  error and covers only genes that have been reviewed. It is a lower bound, and it is
  biased toward genes someone looked at carefully.
• DISPUTED (75) and LOW_QUALITY (99) are collected for context but are not citation
  defects — they describe the science, not the pointer.

Next steps

1. Adjudicate the 12 unflagged uses of the 7 spreading citations.
2. Assemble the GOA-sourced WRONG_IDENTIFIER set into per-database reports (MGI, SGD,
   UniProt) and file them upstream.
3. Fix the one review-only case (ADPRH).
4. Put Check A in CI — it is cheap, has no false positives by construction, and a
   non-resolving identifier is unambiguous.
5. Decide whether alias-aware matching is worth building for Check B, or whether the
   class is better caught by reviewers reading the cached text.

Relationship to other projects

• REVIEW_QUALITY_AUDIT.md — a different defect: templated
  reasoning and placeholder evidence. A review can have real citations and fake
  reasoning, or vice versa.
• MITO_INTERACTOME.md — where the COX17/COX10 case surfaced.
• IBA_REVIEW.md — propagation failure taxonomy; miscitation is one way
  a propagated annotation acquires unsound support.

Slides

• Slides (Marp source: MISCITATION_AUDIT-slides.md) — AI generated