Function Prediction Evaluation
Bottom line: new protein-function predictors appear faster than curators can
judge them, and aggregate benchmarks do not say whether a given method's
predictions are safe to import. This page indexes the AI Gene Review projects
that test such predictions claim by claim against agent-adjudicated gene
reviews, scoring GO terms with the COR/CNN/LSP/UNC/PLI/NPI/REP taxonomy from de
Crécy-Lagard et al. 2025 (PMID:40703034). Across projects the errors concentrate
in specificity, paralogs, pseudoenzymes and organism context. In the largest
model benchmark most correct predictions were already known: as of the review
snapshot (2026-09-27, commit c7551cb3db), 682 of 955 BioReason-Pro SFT terms
were correct but not novel and 23 were correct and novel. ProtNLM2's purposive
cohorts, selected for likely-novel targets, have a much larger novel share: as of
its own snapshot (2026-10-29, commit ac95c69686), 35 of 288 GO terms were correct
and novel and 48 correct but not novel.
Affinage's GO layer rarely reached the specific curated function, and reviewers
accepted uncorroborated TreeGrafter inferences much less often than curated
PAINT/IBA ones. Each project below has its own cohorts, methods and denominators.
Browse all predictions — a shared faceted catalog of prediction sets and GO/EC claims, including narrative reviews and assessed empty outputs. Filter by method, species, project, cohort, or assessment; share the resulting URL. Browser guide.
The project pages keep their prose to findings and methods; per-category counts
are facet counts in the browser. Useful starting views:
ARGO95 SFT claims,
ProtNLM2 GO claims,
DeepECTF claims, and the
GO-GPT three-level overlap, which is fixed at the review snapshot.
Cross-project review (2026-09-26) — consistency, independence, and reproducibility audit of the evaluations below, with 2026-10-04 fix status and remaining follow-ups.
Model and agent evaluations
| Project | What is evaluated | Explore |
|---|---|---|
| ProtNLM2 | GO predictions across a taxonomically diverse protein benchmark, assessed for biological support, specificity, and overlap with existing annotations. | Prediction reviews |
| BioReason-Pro and GO-GPT | BioReason-Pro functional summaries and reasoning traces, its SFT GO predictions, and the separate upstream GO-GPT term predictions. | SFT reviews · GO-GPT reviews · Manuscript |
| DeepECTransformer / E. coli | Enzyme-function predictions for selected E. coli proteins, with attention to substrate specificity, paralogs, and physiological context. | Prediction reviews · Blinded recapitulation (table; 4/7 match) |
| Affinage | Literature-derived functional narratives, GO grounding, and retrieval of relevant publications. | Pilot results · Narrative versus GO analysis · Project findings |
| Structure-based prediction | Fold, active-site and structure-aware learned methods for distant homologs, tested against cases from existing reviews. | Project page |
| OpenScientist co-scientist | An autonomous research agent used for focused blinded bioinformatics investigations; its reports are advisory evidence in ProtNLM2 and illustrative exemplars in TreeGrafter. | Project page |
| Prokaryotic immunity term prediction | Scoping: translating family-level defense-system calls into review-ready GO term suggestions. | Project page · Tracker #3962 |
BioReason-Pro SFT, RL narratives, and upstream GO-GPT outputs are separate
evaluation targets. The GO-GPT review includes unresolved predictions; its table
is a review workspace as well as a results browser. DeepECTransformer has two
tables: the project's own calls (rendered from genes/ECOLI/*/*-det-predictions-review.yaml,
the same records shown in the prediction browser) and a blinded recapitulation run,
hosted with the BioReason comparison material, whose calls match the published
expert labels for only 4 of the 7 genes.
Annotation-transfer and rule reviews
These projects examine the methods and mappings behind existing annotations and
provide context for evaluating additional model predictions. Transfers by
orthology, phylogeny, and family membership have their own index,
Propagation by Homology, with a
browser of all propagated annotations.
| Project | Focus |
|---|---|
| TreeGrafter | Automated placement onto PANTHER trees and transfer of ancestral GO annotations. |
| eggNOG-mapper | Orthology-based GO transfer by eggNOG-mapper 2.1, run on 520 reviewed proteins with and without the query's own species. |
| InterPro2GO | GO mappings attached to domain and family signatures, including specificity and propagation limits. |
| NCBIFam | Functional-family mappings and opportunities or risks in extending GO coverage. |
| PAINT / IBA | Curator-assessed phylogenetic function inheritance and the evidence for individual transfers. |
| UniProt keywords (SPKW) | Annotations derived only from UniProt keyword mappings (GO_REF:0000043) and their over-annotation patterns. |
| Pfam → GO | Whether pfam2go adds specificity beyond InterPro2GO, and headroom for new family mappings. |
| Rhea → GO | What rhea2go contributes beyond ec2go, and reactions with no GO target. |
| TCDB → GO | Transporter classifications that never become GO annotations, and candidate TC-to-GO mappings. |
| ARBA rule reviews | Reviews of UniProt's automated annotation rules and their biological scope. |
Reading the evaluations
Biological correctness, annotation specificity, novelty relative to existing
annotations, and reference quality are distinct questions. The projects use
different cohorts and review procedures, so their scores should be read with
their own methods and denominators. Local AI-assisted reviews also vary in
maturity; they are not automatically independent experimental ground truth.
For the shared approach to term-level review, see the
evidence standards. For narrative
correctness and completeness, see the
BioReason evaluation rubric.
All projects in this collection
Generated from project frontmatter (collections: [FUNCTION_PREDICTION]).
| Project | Maturity | Tags |
|---|---|---|
| Affinage Evaluation Project | MATURE | PIPELINE, EVALUATION |
| ARBA Rule Review | IN_PROGRESS | PIPELINE, EVALUATION |
| BioReason-Pro Comparison Project | MATURE | PIPELINE, FLAGSHIP, EVALUATION |
| eggNOG-mapper Evaluation | IN_PROGRESS | EVALUATION, PIPELINE |
| IBA Annotation Quality Project | MATURE | PIPELINE, FLAGSHIP |
| InterPro Mapping Review Project | IN_PROGRESS | PIPELINE |
| NCBI RefSeq vs UniProt/GOA: prokaryotic GO pipeline comparison | IN_PROGRESS | PIPELINE |
| NCBIFAM / CDD → GO Contribution & Gap Project | IN_PROGRESS | PIPELINE |
| ProtNLM2 Evaluation | IN_PROGRESS | EVALUATION, ML_PREDICTIONS |
| TreeGrafter Inference Evaluation | MATURE | EVALUATION, PIPELINE |
| Validating E. coli ML Predictions | COMPLETE | PIPELINE, FLAGSHIP, EVALUATION |