Function prediction evaluation projects: cross-project review (2026-09-26)

Back to the Function Prediction Evaluation index

This page reviews the projects listed in the function-prediction index:
ProtNLM2, BioReason-Pro / GO-GPT,
DeepECTransformer / E. coli,
Affinage and TreeGrafter. It also covers the index
page itself and the shared *-predictions-review.yaml / prediction-browser infrastructure.

The review was AI-generated (Claude Code). Parallel reviewer agents recomputed the headline numbers
from each project's sidecars and per-gene YAMLs, checked relative links, ran schema validation, and
spot-checked individual calls against the gene reviews. The review changed no project content.
Findings are listed with file:line so they can be acted on.

Status of fixes (updated 2026-10-04)

Each project page now states the commit its results were computed at and the command that reproduces them.
ProtNLM2 and BioReason numbers are read at main's frozen review_snapshot_commit. TreeGrafter and Affinage
were regenerated at dated branch tips after the initial audit; the latest TreeGrafter
refresh is commit f81b9f300 on 2026-10-03.

• E. coli / DeepECTransformer
• Accessions, b-numbers and the ygfF locus_tag are corrected.
• The prediction-review files had wrong-identifier PMIDs, including the DeepECTF paper itself
  (37820725 instead of 37963869). These were fixed on main in PR #3273.
• A production table now sits alongside the retitled blinded-recapitulation table.
• ProtNLM2
• OpenScientist verdicts are reconciled with the current calls. After main incorporated the audits,
  13 agree, 5 fall in the same polarity group, 12 are decided differently and 1 run failed; 30 of 31
  terms cite the report. The report carries a status box.
• COR is reported per cohort and defined as "absent from cached target GOA".
• The headline is "186 of 242 targets assessed".
• The hardcoded generator script is archived.
• A single PLI/NPI rule is documented; UBE2F vs NCU04302 is left as an open inconsistency.
• BioReason
• Matched SFT-vs-RL comparison on 110 shared genes: correctness shows no difference (3.96 vs 4.05);
  SFT is higher on completeness (3.88 vs 2.75). This replaces the disjoint-cohort claim.
• Failure modes are counted with denominators.
• The page discloses that the raters are LLMs, adds the balanced-kappa caveat, and gives scores by
  reference status.
• A GOA-date leakage audit runs under labelled cutoff assumptions.
• Example attributions are fixed, and the manuscript now says eight modes.
• Affinage
• Affinage grounds to goslim_generic, so slim-level capture is added (core MF 38/42, location 33/38).
• PAINT recall is reproducible and split: pooled 51%, non-FA 48%, FA 85%. The "flat by study
  depth" claim is withdrawn.
• The full offline cache is committed.
• The PKM, RASA1, KEAP1 and HMOX1 examples are corrected.
• TreeGrafter
• The snapshot is refreshed: 1,074 annotations on 577 files; ACCEPT 43.2%, REMOVE-only 11.1%,
  retained 77.0%, down-graded 283.
• A drift table and a summary of the re-review are added.
• The OpenScientist/Falcon exemplars are described as illustrative only: n=10, all drawn from
  already down-graded rows, with no accepted-annotation controls.
• ADAR2 is reclassified to mode 1.
• Index and CI
• Structure-based, OpenScientist, prokaryotic-immunity, SPKW, Pfam, Rhea and TCDB projects are added.
• render-bioreason-eval now runs in build-pages and in the daily workflow.
• CLAUDE.md is corrected.

The issue list and per-project findings below preserve the 2026-09-26 audit;
the fixes above supersede findings where the same problem is listed as fixed.
Still open, with each item recorded on its project page:

• Human rating subset (#4087):
  have a human rate a small shared subset. Every reference and every rater
  in these evaluations is still an LLM agent.
• ProtNLM2 (#4005):
  COR calls entailed by an existing MF or domain are not yet computable; the
  pinned GO file has no MF-to-BP links.
• BioReason (#4088):
• The training cutoff is not published, so the leakage audit relies on assumed dates.
• 62 ARGO95 and 124 GO-GPT NPI terms have no error_type.
• The 198-file supplemental SFT set is largely templated.
• The PDF has not been rebuilt.
• TreeGrafter (#4089):
  curate the remaining mode-0 and keyword-placed rows.
• Affinage (#4031):
  per-term precision at slim level.

Overall verdict

The infrastructure is in good shape:

• All 169 prediction-review YAMLs validate.
• All supported_by quotes checked are verbatim.
• The prediction browser data.js matches a fresh regeneration.
• The headline counts on every page reproduce from their own sidecars.

The weaknesses are methodological:

• The reference standard is not always stated. In every project the reference is an AIGR agent
  review. That is the design, but pages should say so, and should flag the one case (Affinage FA)
  where the evaluated tool's output was folded into the reference.
• Novelty is judged against current GOA rather than a training-cutoff snapshot.
• Several "frozen" results have drifted from the reviews they summarize.

None of the pages currently overstate numbers relative to their data. Several overstate what the
numbers mean.

Cross-cutting issues

1. State the reference standard (all projects). Agent reviews as the reference are the design:
   the aim is to check, within each tool's own corpus, whether its outputs are biologically
   meaningful by the project's curation rules. Each page should say what its reference is and
   that it was fixed before scoring. (Revised 2026-10-01; the original wording called this
   "circular or non-independent ground truth", which overstated it.) Two real issues remain:
2. Affinage FA-cohort reviews were built with Affinage input, then used to grade Affinage
   recall. That is ordinary bias, now reported separately.
3. The TreeGrafter OpenScientist/Falcon exemplars came only from already down-graded rows,
   with no accepted-annotation controls, so they cannot support a reliability claim.

A small human-rated subset would still be useful to calibrate the agent ratings.
2. COR/CNN measure GOA sparsity, not model novelty.
- ProtNLM COR is defined against the target's cached GOA. As a result, horse is 37% COR while
pombe, Neurospora and fly are about 0%. Roughly 16 of 53 COR calls are generic terms entailed
by an existing MF or domain.
- BioReason re-labelled 61 calls to CNN in a current-snapshot audit, 25 of them from COR.

Novelty should be judged against a GOA release at the model's training cutoff, or COR should be
renamed "not in target GOA" and reported per cohort.
3. Failure-mode taxonomies are anecdotal.
- BioReason: 62/147 discordant ARGO95 terms have no error_type, and LOCALIZATION_DEFAULT is
tagged only 4 times, yet the page calls it "systematic".
- Affinage: some "wrong branch" examples contradict our own core_functions.

Report mode counts with denominators.
4. Snapshot drift.
- TreeGrafter: re-running the analysis on HEAD changes 63 actions. 48 of the 306 down-grades are
no longer down-graded, which invalidates two hotspot families.
- ProtNLM: the OpenScientist adjudication report disagrees with about 20 current YAML calls.
- Affinage: recall re-runs give 137 genes at 47%, not 91 at 52%.

Each results section should name the commit it was computed at and a command that reproduces it.
5. Two public views of the same predictions disagree (DeepECTF).
deepectf-eval.html is rendered from the blinded recapitulation copies. That experiment matched
the production calls on only 4/7 genes. The prediction browser shows the production calls.
Neither page says which is which.
6. Generated evaluation tables are not built in CI. render-bioreason-eval
(project.justfile:1532) is not part of build-pages, so sft-eval, gogpt-eval and
deepectf-eval.html refresh only by hand.

Per-project findings

ProtNLM2 (page)

• Reproduces: 162 YAMLs, 288 GO claims (COR 53, CNN 32, LSP 83, UNC 103, NPI 17). Per-cohort
  counts and narrative counts match. All 162 files validate.
• HIGH: PROTNLM_EVALUATION/openscientist-adjudication.md:21,44-71 moves about 20 calls to NPI
  (for example the F6LAX4 NPI×6 and the A0A3B6RKV1 ×4). The current YAMLs keep most of them UNC.
  The page says OpenScientist "carries substantial weight" (md:358, 484), and the report says its
  verdicts are "wired into per-gene YAMLs" (line 103). Only F4JLB7 cites it. Add a reconciliation
  table, or mark the report as superseded.
• HIGH: COR is inflated by sparse GOA and ortholog transfer (see cross-cutting issue 2).
• HIGH: "242 prediction targets" (md:350) includes 57 targets with no reviews: FLY_NEXT20,
  FLY_LOCATION_KEYWORD and POMBE_REMAINING8. State the number actually assessed.
• MED: The overlap table at md:402-403 has two transposed cells. NO_OVERLAP should read COR 11 /
  UNC 10, and NOT_IN_GOA should read COR 6 / UNC 16.
• MED: The PLI/NPI boundary is inconsistent. UBE2F (a NEDD8 E2 predicted as ubiquitin) is NPI,
  while NCU04302 (a SUMO E2) is PLI. Several MOD-evolution NPIs are driven by TrEMBL fragments.
• MED: generate_prediction_reviews.py:146-166 hardcodes per-accession verdicts that contradict
  the current YAMLs, and it overwrites them by default. Archive or delete it.
• LOW: The table description at md:356/502 is stale ("77 assessments"; the table now renders
  about 306 rows). The slides contradict the current calls (the patatin case is now NPI). The
  frontmatter runs 342 lines before any content.

BioReason-Pro / GO-GPT (page)

• Reproduces:
• Score distributions and correlations (Pearson 0.688).
• ARGO95: 955 terms, 71.4% CNN.
• Second-rater kappa: 0.950 for correctness, 0.744 for completeness.
• All 139 per-gene RL scores.
• HIGH: The SFT-vs-RL comparison (md:437-481) compares 45 SFT genes with 138 RL genes that
  share no genes. Score both on the same set, or drop the claim.
• HIGH: The failure-mode taxonomy is not backed by counts (see cross-cutting issue 3).
• HIGH: Rater circularity is not disclosed on the page. The kappa sample is balanced across
  score strata, which inflates kappa. 60/139 references are not COMPLETE, and md:121 promises a
  per-status breakdown that the page does not give.
• HIGH: There is no leakage analysis. ARGO139 consists of well-characterized, pre-2022
  proteins, so 71% CNN and CAFA precision of 0.86 largely measure recall of training data.
• MED: Examples are attached to the wrong modes. The aprE hemostasis terms come from
  GO-GPT/SFT, not the RL narrative (which scored 5/5). Spy's summary does say periplasmic.
• MED:
• The scale error "score 0-1/5" at md:465 is impossible on a 1-5 rubric.
• The manuscript counts eight failure modes; the page counts nine.
• article/TODO.md says the auto-reviewed genes "must be redone manually", yet all 95 are marked
  COMPLETE, and 71 of the CNN calls share boilerplate text.
• LOW:
• "Our pilot (5 genes)" is stale.
• The by-organism table silently drops three n=1 species.
• The frontmatter species list is wrong.
• The tracked manuscript.pdf can drift from its source.
• The DeepECTF table is titled "BioReason-Pro".

DeepECTransformer / E. coli (page)

• Reproduces: All 7 genes are reviewed. The production -det-predictions-review.yaml files are
  COMPLETE, validate, and match the page's categories.
• HIGH: All 7 UniProt accessions in the gene tables (md:44-66) are wrong, and so are 2
  b-numbers. The correct values from genes/ECOLI/*/*-uniprot.txt:

Gene	Accession	b-number
ygfF	P52037	b2902 (page says b2878)
yciO	P0AFR4	b1267 (page says b1282)
yegV	P76419
yjhQ	P39368
yrhB	P46857
yjdM	P0AFJ1
fepE	P26266

• HIGH: The linked "Prediction reviews" table shows the blinded recap verdicts: ygfF UNC,
  yegV UNC, yjdM NPI. The project's own calls are COR, PLI and UNC. Neither page mentions the
  recap's 4/7 agreement.
• MED: genes/ECOLI/ygfF/ygfF-det-predictions-review.yaml:3 has locus tag b2898 (should be
  b2902). The det YAMLs cite PMIDs 37820725, 19287007 and 21135036, which have no cached
  publications, and give their titles in paraphrased form.
• LOW: The NOTES stop at 2026-03-22. The CLAUDE.md example path
  yciO-predictions-review.yaml does not exist; the file is -det-.

Affinage (page)

• Reproduces: All 42 per-gene rows match per-gene.json and summary.csv. The FA-cohort sums
  are correct. The ADA symbol collision is confirmed in the cache. The scripts compute from data.
• HIGH: Affinage grounds to only 43 distinct, bin-level GO ids across all 42 genes, which looks
  like a GO slim. Exact leaf-term capture (for example "1/42") is then near-impossible by
  construction. Confirm the vocabulary and score against the slim.
• HIGH: The FA cohort (22 of the 91 PAINT genes) inflates recall: 85% for FA versus 48% for the
  rest. Without FA, recall in the well-studied band is 27%, not 56%. So "recall does not depend on
  how well-studied the gene is" (md:297; results/paint-campaign.md:88) does not hold.
• HIGH: Neither result is reproducible as documented.
• retrieval_recall.py --all now scores 137 genes, and the 91-gene list is not committed.
• affinage-cache/ holds only ADA and GPX4, so compare_affinage.py needs live, unversioned
  API calls.
• MED:
• The PKM, CPT1C and GAPDH "wrong branch" examples conflict with their own core_functions.
• The RASA1, KEAP1 and HMOX1 descriptions pick secondary terms rather than the top-supported ones.
• "Shared" counts include GOA terms that curators REMOVEd; NEG_ACTIONS is defined but unused.
• Sections disagree about whether to integrate Affinage.

TreeGrafter (page)

• Reproduces: Every number matches the sidecars at the 2026-09-06 snapshot. The GO_REF
  separation between 0000118, 0000120 and 0000002 is verified. All 12 hotspot PANTHER ids exist in
  panther.obo.
• HIGH: The snapshot is materially stale.
• Re-running on HEAD gives 969 annotations on 541 files with 63 changed actions, and 274
  down-grades instead of 306.
• PTHR44169 (fogD) and PTHR48078 (ilvA) are no longer hotspots.
• The fogD example at md:247 has been reversed in TREEGRAFTER/rereview-2026-09-20/.
• That rereview folder is not linked from the page.
• HIGH: OpenScientist/Falcon verdicts were cited into the reviews they are scored against
  (ADAR2, NaPMT3). The exemplars are drawn only from down-graded rows, with no accepted negative
  controls. Two exemplar actions have since changed. So "reliably catches exactly the cases
  TreeGrafter gets wrong" is unsupported.
• MED:
• "Outright rejected" (md:145) counts MARK_AS_OVER_ANNOTATED as rejection; REMOVE alone is 13.4%.
• Mode 3 (generic CC) is an informativeness policy, not a grafting error.
• The PAINT contrast compares different gene sets.
• "Same 510 proteins" (md:170) is inaccurate: the contrast sets cover 371 and 353 files.
• ADAR2 is classified as mode 4, but by the page's own rule it is mode 1.
• LOW:
• The analyze_treegrafter.py docstring contradicts itself on GO_REF:0000120.
• graft_check.py hardcodes review actions, which are now stale.
• md:322 says "five exemplars"; there are 10.

Index and shared infrastructure (index)

• All links resolve after rendering.
• 169 prediction-review files hold 295 claims:

Category	Claims
UNC	104
LSP	83
COR	54
CNN	32
NPI	19
PLI	2
REP	1

• No claims are PENDING, and every discordant call has an error_type from the schema enum.
• MED: The index omits several relevant projects:
• STRUCTURE_FUNCTION
• PROKARYOTIC_IMMUNITY_TERM_PREDICTION
• COSCIENTIST
• orthology transfer: ISO, ORTHOLOG_CONJECTURE
• mapping-based IEA sources: SPKW, PFAM, RHEA,
  TCDB
• LOW: The docs/prediction_browser.md link is served as raw markdown. The CLAUDE.md
  error-type list omits DOMAIN_ARCHITECTURE_MISMATCH (the most-used NPI type) and
  COMPLEX_ACTIVITY_TRANSFER.

Prioritized actions

1. Quick, mechanical fixes:
2. E. coli accession and b-number table; ygfF locus_tag.
3. ProtNLM overlap-table cells.
4. BioReason "0-1/5" scale error and the eight-vs-nine mode count.
5. TreeGrafter "five exemplars" and "same 510 proteins".
6. Relabel deepectf-eval.html as the blinded recap and retitle it.
7. Refresh drifted snapshots, pinned to a commit: TreeGrafter (re-run plus re-curation of the
   flipped rows), Affinage recall (commit the gene list; report FA separately), and reconciliation
   of the ProtNLM OpenScientist report.
8. Add a standard "reference standard and leakage" section to each project: name who made
   the reference, whether the evaluated tool's output was visible to them, and the GOA snapshot
   used for novelty.
9. Replace anecdotal failure modes with counted ones, with denominators, and fix the
   mis-attributed examples (aprE, Spy, PKM, RASA1, KEAP1, HMOX1, ADAR2).
10. Build render-bioreason-eval in CI; archive generate_prediction_reviews.py; commit the
    trimmed Affinage cache.
11. Extend the index with the missing projects.
12. Longer term: have a human rate a small shared subset across the BioReason, ProtNLM and
    TreeGrafter evaluations, so that agreement with the agent reviews can be quoted.