ProtNLM2 Evaluation

Evaluation of Google's ProtNLM2 GO term predictions against expert-curated AIGR gene reviews and existing GOA annotations, using the ARGO-ProtNLM-50 benchmark (50 proteins, 14 taxonomic groups, 75 GO predictions).

Interactive prediction evaluation table — filterable/sortable HTML view of all 50 proteins and 77 prediction assessments.

Independent adjudication, both directions. Predictions were re-tested as blinded, independent
OpenScientist gene-function hypotheses in two rounds. (1) The
borderline UNC calls: ProtNLM2's localization calls held up (3/3 correct — one even caught a
mis-localized curated annotation), while its specific function/process refinements over-reached
(9/14 → NPI); two were undecidable and kept as leads. (2) Our own disputed (NPI/PLI)
calls: 9 of 10 independently confirmed as misassignments, and 1 caught as over-harsh —
autophagosome for tbc1d14, which has a direct experimental IDA in the human ortholog, so it was
overturned NPI → CNN. See the OpenScientist adjudication report
for per-gene verdicts and curation leads.

Key findings

1. ProtNLM2 is strongest for uncharacterized proteins, but its "novel" hits are mostly InterPro2GO coverage gaps, not new biology: 12/23 (52%) of NOT_IN_GOA predictions were likely correct (domain/orthology-based — no experimental support) and absent from GOA — e.g. DNA binding for a KilA-N domain protein (A2FPI7), ECM organization for OLFML2A (A0A8C9H4D2), nuclear localization for MCM-4 (A0A061AL94). These functions follow directly from the proteins' domains; the standard InterPro2GO pipeline (GO_REF:0000002) simply does not emit them, because the relevant domain has no InterPro2GO mapping (olfactomedin, KilA-N), the mapping sits only on a superfamily entry the protein was not assigned (KilA-N → the APSES superfamily IPR036887), or the protein matched only an unintegrated Pfam and never the InterPro entry that carries the mapping (MCM-4 → PF21128, with the rich IPR008047 mapping unassigned). See InterPro2GO coverage gaps.

2. "Exact" matches are mostly less precise, not novel: 13/19 (68%) are parent terms of more specific existing annotations (e.g., predicting "cytoplasm" when "clathrin-coated vesicle" is already annotated). ProtNLM2 captures broad functional categories but lacks resolution.

3. Frequency bias is the dominant error mode (13/22 error annotations): the model over-predicts common GO terms (membrane, transferase activity, phosphorylation) without biological specificity, especially problematic for proteins with unusual functions.

4. Cross-kingdom errors persist despite taxon filtering: animal-specific terms (neuronal cell body, protein antigen binding) predicted for plant proteins, indicating the Evidencer's taxon constraint checking has gaps.

5. Paralog discrimination is a weakness: the model conflates catalytically active and inactive family members (MTMR9 pseudophosphatase, RIC7 lacking kinase domain) — a Type 6 error that sequence similarity methods inherently struggle with.

6. Expert review substantially revises mechanical assessment: the automated category mapping (EXACT→CNN, NO_OVERLAP→UNC) was heavily corrected once protein biology was considered (EXACT→LSP, MORE_SPECIFIC→NPI, etc.).

Aggregate results (75 predictions, 39 proteins)

Assessment categories follow de Crécy-Lagard et al. 2025 (PMID:40703034).

Concordant (CS=2): 46/75 (61%) | Uncertain (CS=1): 13/75 (17%) | Discordant (CS=0): 16/75 (21%)

Results by GOA overlap category

Error analysis (16 incorrect predictions)

Illustrative case studies

These 5 proteins (included in ARGO-50) were identified during exploratory analysis and represent the main error patterns. Each has a full AIGR review (*-ai-review.yaml) and prediction assessment (*-protnlm-predictions-review.yaml).

Trivially correct: A0A3B6GK97 (wheat patatin)

ProtNLM2 predicts lipid catabolic process > GOA's lipid metabolic process. The protein already has IBA annotations for glycerophospholipase + monoacylglycerol lipase activity. In wheat GOA, 94% of proteins with glycerophospholipase activity already have lipid catabolic process annotated. ProtNLM2 is doing bookkeeping, not discovering biology.

Phmmer transfer: A0A3B6RKV1 (wheat JmjC)

ProtNLM2 predicts 5 specific plant biology terms (gibberellin signaling, photomorphogenesis, seed germination, epigenetic regulation, red light response). All 5 trace to one phmmer hit: Q67XX3 = Arabidopsis JMJ22 (score 689.2). This is ISS/ISO-style annotation transfer — the "added value" over IBA is that ProtNLM2 transfers BP annotations that PAINT's more conservative approach chose not to propagate.

False positive: F4JLB7 (Arabidopsis RIC7)

ProtNLM2 predicts kinase activity + phosphorylation (score 0.23). The phmmer hit is mouse LRRK2 (score 33, barely above noise) — a 2,527 aa multidomain protein with LRR + ROC + COR + kinase domains. RIC7 only has LRR repeats and is a ROP GTPase effector, not a kinase. Classic multidomain annotation leakage.

Cross-kingdom error: F6LAX4 (wheat PP2A scaffold)

ProtNLM2 predicts neuron projection, neuronal cell body, and protein antigen binding for a wheat protein. Plants have no neurons and no adaptive immune system. The predictions leak from mammalian PP2A orthologs that are annotated to neuronal compartments. All 6 predictions scored CS=0 (NPI).

Ontology gap: Q9KZ33 (S. coelicolor sigma factor)

IBA: sigma factor activity. ProtNLM2: transcription initiation. These are biologically coupled but classified as NO_OVERLAP because there is no is_a/part_of path between "regulation of transcription initiation" (MF ancestry) and "DNA-templated transcription" (BP ancestry). This is a real limitation of closure-based evaluation, not a ProtNLM2 error.

What is ProtNLM2?

ProtNLM2 is a T5-based seq2seq model developed by Google DeepMind with UniProt, trained on 240M proteins (UniProt 2023_04). It predicts protein names, GO terms, subcellular locations, and function comments from amino acid sequence, organism TaxID, and AlphaFold secondary structure.

Predictions are post-processed by the Evidencer — a corroboration pipeline that checks each prediction against string matches, phmmer sequence similarity (bit score > 25), and TM-align structural similarity. Predictions failing GO taxon constraints or lacking corroboration are excluded. See the UniProt help page for details.

ARGO-ProtNLM-50 benchmark design

50 proteins curated for systematic evaluation, stratified across:
- 14 taxonomic groups (mammals, plants, bacteria, fish, insects, fungi, etc.)
- 4 prediction categories (rich, partial, go_only, name_only)
- Multiple evidence methods (string match, phmmer, tmalign)
- 5 case studies from exploratory analysis (see above)

All 50 received full AIGR annotation reviews with falcon deep research. The 39 with GO predictions received biologically informed prediction assessments. The 11 name-only proteins have reviews but no prediction-review YAMLs. See argo_protnlm_50.csv.

Overlap with existing AIGR reviews

Only 8 of 1,334 previously reviewed genes appear in the ProtNLM2 dataset (all TrEMBL/unreviewed): C5AXM3, O94267, Q09490, Q21303, Q86WA8, Q9BZE2, Q9UNW9, Q9XUS3.

References

• UniProt ProtNLM help page
• ProtNLM2 accession list (FTP)
• de Crécy-Lagard et al. 2025 (PMID:40703034) — assessment categories

Files and methods