ProtNLM2 Evaluation

Challenge: Selecting a benchmark from an arbitrary set of genes

ProtNLM2's full predictions (GO terms, function text, locations) are public only for
a pilot set released by UniProt: about 26,000 TrEMBL entries from "about 400 of the most
searched species" (UniProt ProtNLM help). The REST API
also serves some records from the 28,553-entry pre-release export that are not on the list (at
least 57 are known: all 28 S. pombe and 29 fly records we checked; the total is undetermined),
and our cohorts use 37 of them (28 S. pombe, 9 fly). Other proteins we probed returned 404. We could
only evaluate what was released, so every cohort below is a census of, or a hand-picked sample
from, that material. The last bar is not strictly nested: 86 of our 242 targets are off the list
or have only names, locations or keywords.

This makes the evaluation hard to read as a benchmark:

• The species mix was set upstream. Wheat has 725 pilot records, horse 404, fly 94 and
  human 151 (pilot_taxa in benchmark-funnel.json). Neither UniProt nor Google DeepMind
  documents how records were chosen.
• Every listed record is TrEMBL, but many genes are well studied. A fly record can be
  identical to a reviewed entry with experimental GO (Lcp3 and P07188), so "correct and new"
  is rarely available in model-organism cohorts.
• There is no denominator. Nobody chose the inputs to measure proteome-wide accuracy or
  recall, so neither can be estimated here.
• Cohorts are not interchangeable. Hand-picked, census and rule-ranked cohorts answer
  different questions. Their category rates should not be pooled into one accuracy figure.

What the evaluation can do is judge each emitted claim against sequence, domains, structure
and literature, and report the patterns of error. Figures are built by
build_benchmark_funnel.py from
benchmark-funnel.json; the
data provenance page explains the pre-release export and
the off-list records. The cohort descriptions are curated in
benchmark-cohorts.yaml.

Summary

Bottom line: ProtNLM2 is Google DeepMind's sequence-to-text model that
writes protein names, GO terms and function paragraphs straight from sequence,
and UniProt ships its output on unreviewed entries. We assessed it the way a
curator would, protein by protein, using the COR/CNN/LSP/UNC/NPI/PLI/REP
categories of de Crécy-Lagard et al. 2025 (PMID:40703034). As of the review
snapshot (2026-10-29, commit ac95c69686), the assessment covers 242 prediction
targets and 288 GO-term assessments, plus separate reviews of the narrative
function text.
Of the 288 GO claims, 35 are correct and novel (COR) and 48 correct but already
annotated to the gene (CNN). Novelty is judged against the gene's GOA on any of its
products, and many targets are alternative models of well-annotated genes. Contradicted
predictions (NPI or PLI) are 44 of 288. Narrative text behaves differently from GO
terms: paralog confusion is common in the function paragraphs and nearly absent
from the GO claims.

The recurring failure is not a wrong family but a claim pitched above what the
sequence supports: a catalytic activity where the deposited sequence lacks the
catalytic region (wheat patatin A0A3B6GK97, Arabidopsis LRR F4JLB7), or a
localization that cannot exist in the organism (neuron projection in wheat
F6LAX4). Correctness is judged for the deposited sequence, so a claim true of the
gene but needing a region the selected model lacks is refuted. The most common
verdicts are less precise (LSP, 83) and uncertain (UNC, 78); uncertainty is the honest
result for TrEMBL proteins with no direct characterization.

Function prediction evaluation index

Browse all ProtNLM predictions — filter prediction sets, narrative reviews, and individual GO claims by species, cohort, and assessment. Fly records include reviewed empty GO outputs.

Where the numbers are. The dated counts, deduplicated across overlapping
cohorts, are in the cross-cohort results
and benchmark-summary.json. The
browser gives the same breakdowns as facet counts over the current reviews:
all GO claims, one cohort at a time
(ARGO-50,
HORSE40,
FLY41,
POMBE20,
NEUROSPORA20,
MOD_EVOLUTION20), or only the
contradicted claims.
Narrative records can hold several judgments each and are counted separately from
GO terms. The 57 indexed function narratives contain 4 COR records: HORSE/SHLD2 shieldin-complex
membership, HORSE/GHSR and HORSE/CXCR3 calcium signaling, and DROME/ftz-f1 autorepression. The GHSR and
CXCR3 calls transfer human-receptor experiments to a full-length, closely related horse product (95.1%
and 90.8% identical) rather than observing the horse protein. The SHLD2 call rests on a close
SHLD2-family sequence match for a non-canonical alternative model that keeps the N-terminal SHLD3-MAD2L2
recruitment segment but diverges after about residue 811; its downstream claims are UNC for that reason.
The ftz-f1 call rests on a Drosophila experiment on beta FTZ-F1; the selected product M9NFK2 is identical to UniProt isoform A (P33244-2), whose N terminus is associated with the beta isoform (PMID:8096644). Not every selected target has been reviewed yet; the cross-cohort results
state how many have, and their per-cohort table should be read with the COR caveat in
Reference standard and novelty.

The cross-species cohorts examine biological support, substrate and paralog
specificity, domain completeness, and transfer of annotations across species. The
original ARGO-ProtNLM-50 subset samples 14 taxonomic groups; its findings and
illustrative case studies are below.

Interactive prediction evaluation table — filterable/sortable assessments, rationales and review links for all 162 prediction-review files across the cohorts. It has 288 GO assessments plus 17 reviewed empty outputs, not only the ARGO-50 subset.

COR is not model novelty. Here COR means biologically supported and absent from the gene's cached GOA, on any of its products. It does not mean absent from ProtNLM2's training data (the sense used by de Crécy-Lagard et al.). The COR share differs sharply by cohort:

• 16/89 GO claims (18%) for horse and 16/77 (21%) for ARGO-50;
• 2/50 for fly;
• 1/32 for pombe, 0/21 for Neurospora and 0/19 for MOD-evolution.

These differences reflect the targets' existing annotation, which terms each cohort's predictions emit, and how the LSP/COR boundary was applied, as much as the model. They should not be pooled into a single novelty rate. A mechanical GO-closure check finds none of the 35 COR calls entailed by an existing target annotation; 4 are more specific than an existing term. COR terms also vary widely in how much they say, so the same check records each term's number of is_a/part_of descendants: 8 COR terms are leaves and 5 have 100 or more descendants (protein-containing complex, extracellular region, regulation of monoatomic ion transmembrane transport, and DNA binding twice). Five more are protein phosphorylation on a protein kinase whose kinase activity is already annotated. These are correct and novel only in a narrow sense, and a COR count should be read with them in mind. The COR/LSP line is drawn on GO ancestry, so a COR can also be a less informative description than one the gene already carries in another branch: WHEAT/F6LAX4 protein heterodimerization activity is COR because it is not an ancestor of the existing phosphatase regulator annotation, though the review notes that complex scaffolding describes the A subunit better. See the per-cohort table and entailment check.

OpenScientist investigations are advisory evidence; the reviews adopted many of their verdicts and departed from others. Focused OpenScientist investigations examined the ARGO-50 predictions that were uncertain or disputed at the time. They surfaced useful evidence, such as the missing kinase domain in ARATH/F4JLB7 and the experimental autophagosome annotation on the human ortholog of GADMO/A0A8C5FPT8. The reconciliation against the current reviews covers 31 GO terms in 21 genes:

• 12 agree with the current YAML;
• 5 differ only within the same polarity group (COR vs CNN, or PLI vs NPI);
• 13 are decided differently by the current reviews, 10 of them NPI → UNC;
• 1 run failed.

All 13 differing reviews cite the investigation they depart from. Most apply the project's evidence rule that an unestablished transfer is UNC rather than NPI. See the investigation report for the original verdicts.

ARGO-ProtNLM-50 key findings

1. Useful additional annotations can follow from established biology and mapping gaps. Supported family transfers include chloroplastic EF4 localization for ARTAN/A0A2U1PS28 and autophagosome localization for GADMO/A0A8C5FPT8, from experimentally grounded TBC1D14 evidence. The InterPro2GO coverage analysis identifies absent mappings, mappings on unassigned superfamily entries, and unintegrated Pfam signatures as routes by which plausible functions can be absent from GOA. A mapping gap alone does not establish the predicted function: nuclear localization for the 74-residue CAEEL/A0A061AL94 record remains uncertain. For 9PRIM/A0A8C9H4D2, the short leaderless OLFML2A isoform, extracellular matrix is refuted for this product although it holds for the full-length one, and matrix organization remains uncertain.

2. Exact matches often lack specificity. Most predictions classified as EXACT in the GOA comparison are LSP rather than CNN: the predicted term is already present, but a more informative, supported annotation is also available.

3. Catalytic-domain completeness matters. Shared domains or family membership can support an inference while missing catalytic regions contradict a specific activity. The wheat patatin and Arabidopsis LRR protein below illustrate why the deposited sequence needs to be examined.

4. Cross-kingdom errors occur. Neuronal cell body and neuron projection are incompatible with wheat F6LAX4, and its protein antigen binding prediction transfers an animal immune term from mammalian PP2A literature. Its other predictions require separate assessment: heterodimerization is supported by the conserved PP2A A–C core complex, and meiotic centromere localization by Arabidopsis A-subunit data.

5. Core functions transfer more readily than regulatory context. JMJ22-related sequence and experimental evidence support epigenetic regulation for WHEAT/A0A3B6RKV1, while its four specific light, hormone, and germination predictions remain uncertain. For COLLI/A0A2I0M3K7, TRUB2 family membership does not establish the predicted tRNA substrate.

6. Biological assessment adds information beyond ontology overlap. Sigma-factor activity supports transcription initiation even when an is_a/part_of comparison reports NO_OVERLAP. Conversely, matching an existing annotation does not override target-specific contrary evidence.

ARGO-ProtNLM-50 GO results

Assessment categories follow de Crécy-Lagard et al. 2025 (PMID:40703034), with the project's GO prediction guidelines. About half of the ARGO-50 GO claims are supported, most of the rest are uncertain, and a minority are contradicted. The ARGO-50 claims view gives the category counts; the dated counts are in the cross-cohort results. The mean assessment score is an ordinal summary, not a calibrated estimate of model accuracy: the stratified sample is small, and many proteins lack direct experimental characterization.

The closure-based GOA comparison classifies each prediction's overlap with existing annotation (EXACT, MORE_SPECIFIC, LESS_SPECIFIC, NO_OVERLAP, NOT_IN_GOA). Overlap categories describe the comparison dataset, while assessment categories express biological judgments, and the two diverge in both directions: an exact match can still be less precise than another supported annotation on the same protein, and a prediction with no ontology overlap can still be a sound biological inference across GO aspects.

The contradicted ARGO-50 claims show three patterns. Some are impossible for the organism, such as neuronal localizations and protein antigen binding for the wheat PP2A scaffold F6LAX4. Most assert an intrinsic activity that the deposited sequence or domain architecture cannot support, for example kinase activity for ARATH/F4JLB7, lipase activity for WHEAT/A0A3B6GK97, and dephosphorylation by the auxilin pseudophosphatase region of DANRE/dnajc6. A few transfer a function from the wrong paralog subfamily, such as pollen maturation for the rice BURP protein ORYSI/B8BAB0. These are biological incompatibilities; frequency bias and training-data contamination are not established as their causes, and no error mechanism is inferred where the optional error_type field is unset.

Illustrative case studies

These five proteins, included in ARGO-50, illustrate annotation transfer, catalytic-domain checks, taxonomic constraints, and limitations of ontology-based evaluation. Each has a full AIGR gene review and a separate ProtNLM prediction assessment.

Catalytic-region check: A0A3B6GK97 (wheat patatin)

ProtNLM2 predicts lipase activity and lipid catabolic process for WHEAT/A0A3B6GK97. Existing IBA annotations include more specific lipase activities, making the process prediction look like a straightforward extension of known biology. However, the reproducible alignment and motif analysis show that the deposited 302-residue sequence lacks the patatin catalytic-serine region. Lipase activity is NPI; lipid catabolic process is UNC. The sequence may reflect an incomplete gene model or an inactive protein; neither a corrected full-length product nor a noncatalytic role in lipid catabolism is established. This case shows why annotation overlap alone cannot settle correctness.

Phmmer transfer: A0A3B6RKV1 (wheat JmjC)

ProtNLM2 predicts five plant biology terms for WHEAT/A0A3B6RKV1: gibberellin signaling, photomorphogenesis, seed germination, epigenetic regulation, and red-light response. The corroboration records identify Arabidopsis JMJ22 (Q67XX3; phmmer score 689.2), which has experimental annotations for all five terms. This provides a concrete basis for examining transfer from a characterized relative. Epigenetic regulation is COR, supported by the JMJ22 relationship and histone-arginine-demethylation evidence. The four specific regulatory/process predictions are UNC because their Arabidopsis experimental contexts do not establish the same roles in wheat. The phmmer hit is corroborating evidence; it does not reveal how the model generated its predictions or why PAINT omitted a term.

False positive: F4JLB7 (Arabidopsis LRR protein)

ProtNLM2 predicts kinase activity and phosphorylation for ARATH/F4JLB7. The exploratory analysis identifies a weak phmmer corroboration hit to mouse LRRK2 (score 33), a multidomain protein containing LRR and kinase regions. Sharing an LRR region does not establish kinase catalysis. The focused OpenScientist investigation integrates LRR domain assignments, catalytic-motif analysis, and predicted structure to refute a kinase domain in F4JLB7. Kinase activity is NPI; phosphorylation is UNC, since participation through another protein remains possible. The RIC7 name in the database record does not establish that this LRR protein is the CRIB-domain ROP effector described in RIC7 literature.

The RIC7 locus-identity report examines the distinction between AT4G28560/F4JLB7/EXLRR12 and the adjacent CRIB-domain RIC7 locus AT4G28556/Q1G3K8, with sequence checks, the JBrowse view, TAIR record comparisons, and an independent Cornell dissertation finding.

Cross-kingdom error: F6LAX4 (wheat PP2A scaffold)

ProtNLM2 predicts neuron projection and neuronal cell body for WHEAT/F6LAX4. Wheat has no neurons, so both localizations are NPI. Protein antigen binding is also NPI: binding of the viral small-t antigen inhibitor to human PP2A A scaffolds does not establish antigen-recognition activity, and a focused report traced the term to text transfer from mammalian PP2A literature. The remaining predictions have different evidential standing: protein heterodimerization is LSP, supported by the PP2A A-C core complex but less informative than the existing PP2A scaffold annotations; chromosome segregation and centromeric localization are UNC. This example separates clear taxonomic errors from plausible but unverified transfers.

Ontology gap: Q9KZ33 (S. coelicolor sigma factor)

STRCO/Q9KZ33 has an IBA annotation for sigma factor activity; ProtNLM2 predicts DNA-templated transcription initiation. The closure-based GOA comparison classifies the prediction as NO_OVERLAP. The molecular activity and biological process are nevertheless functionally connected: an ECF sigma factor supports promoter recognition and transcription initiation. The prediction is COR, adding a supported process annotation absent from the cached record. This illustrates a limitation of evaluating biological agreement solely through is_a/part_of paths across GO aspects.

What is ProtNLM2?

ProtNLM2 is a transformer-based sequence-to-sequence model developed by Google DeepMind with UniProt, trained on 240 million protein entries from UniProt release 2023_04. It generates protein names, GO terms, subcellular locations, keywords, and function comments from amino acid sequence. UniProt describes the current model as trained entirely on sequence. See the UniProt ProtNLM documentation.

Predictions are post-processed by the Evidencer, which applies exclusion criteria including GO taxon constraints and seeks corroboration through string matches, phmmer sequence similarity (bit score greater than 25), and TM-align structural similarity. This corroboration can explain the biological source of support for a prediction, but it is separate from the neural model's generation of that prediction. The exploratory XML dataset and public release differ in coverage; the data provenance describes the source versions used here.

Expanded cohorts and family curation

The cross-benchmark family curation covers all 282 selected protein records and paired references, with 211 structured family reviews and individual assessments for 18 inputs without exact-record PANTHER assignments. The gene-to-family index distinguishes direct sequence assignments from verified canonical gene context.

The human and model-organism challenge set selects twenty additional prediction-bearing genes across seven species, with priorities for evolutionary analysis of substrate specificity, catalytic divergence and complex participation.

The Neurospora cohort covers every GO/function-bearing entry in the published species subset plus selected localization cases; GO predictions, function paragraphs and localization claims are reviewed separately against a full source census.

The pombe cohort includes every GO/function-bearing entry among the fission-yeast accessions identified in the original export. These currently reviewed/Swiss-Prot entries are API-accessible despite being absent from the published pilot accession list.

The fly cohort includes every Drosophila melanogaster gene with GO or function-text predictions in the published species subset, plus a separate tier of location/keyword-only records. Its census preserves exact sequences, current FlyBase identifiers and original prediction provenance.

The horse cohort contains 40 selected horse genes with released functional predictions and paired human–horse reviews. See the review findings and evidence gaps. The horse-first benchmark design includes the prediction census and mammalian evidence leads. Because horse is a PANTHER reference proteome, its genes carry PAINT IBA and no TreeGrafter rows; the DONKEY40 cohort pairs 40 donkey proteins (TreeGrafter + ProtNLM2) with their horse orthologs (IBA) to compare all three on the same genes.

ARGO-ProtNLM-50 benchmark design

ARGO-ProtNLM-50 was constructed after the ProtNLM release, rather than specified in advance as a benchmark for the model. ProtNLM predictions were released for a partly arbitrary set of proteins, mostly unreviewed/TrEMBL entries. We then selected 50 proteins from that available set to sample different species and kinds of annotations. The resulting benchmark is an exploratory, stratified sample, not a prospective test set or a random sample of protein space.

The selection covers:

• 14 taxonomic groups, including mammals, plants, bacteria, fish, insects, and fungi.
• Four prediction categories: rich, partial, GO-only, and name-only.
• Multiple corroboration methods: string match, phmmer, and TM-align.
• Five case studies from exploratory analysis, described above.

All 50 proteins have AIGR gene reviews and prediction-review YAMLs; nine prediction lists are empty. The benchmark CSV records selection metadata. Each protein's *-protnlm-predictions-review.yaml preserves the prediction and source-method metadata alongside its assessment, rationale, and supporting sources.

Overlap with existing AIGR reviews

The exploratory comparison against the 1,334-review AIGR collection found eight proteins in the ProtNLM2 dataset, all unreviewed/TrEMBL entries: C5AXM3, O94267, Q09490, Q21303, Q86WA8, Q9BZE2, Q9UNW9, and Q9XUS3. This is the comparison set used in the exploratory analysis, rather than a count of the expanding AIGR collection; ARGO-50 provides a broader dedicated evaluation sample.

Evidence standards

Prediction sidecars are checked with just validate-predictions, including publication titles, source excerpts, local paths, and assessment scores. The CI artifact prediction-evidence-validation records those checks.

The function-prediction review skill defines the review criteria. Assessments integrate primary literature, sequence and domain evidence, structural analyses, experimentally grounded curated annotations, and focused OpenScientist investigations. OpenScientist reports are treated as evidence to weigh, not as verdicts. The per-gene review YAML is the call of record, and the reconciliation shows that most ARGO-50 OpenScientist verdicts were not adopted. Reviews cite the relevant analyses and their limitations, distinguishing computational inference from experimental validation. A well-supported family transfer can establish a reasonable function or localization inference without a new experiment on every target; the rationale identifies the characterized relative, the target's family evidence, and the limits of transfer.

Each prediction is assessed at the specificity of its actual GO term. Extracellular localization does not establish matrix organization, and a catalytic fold does not establish a substrate. Conversely, absence of intrinsic catalytic activity does not exclude participation in the corresponding biological process through a regulatory complex. Missing evidence leads to uncertainty unless there is contrary evidence. Broad but true annotations are not biological errors.

COR and CNN distinguish absence versus presence of an equivalent annotation in GOA for the target's gene, after biological support has been established. That includes annotations GOA carries on another product of the same gene in the same species, since GOA propagates to one representative product per gene, so a term already on the canonical product is CNN (or LSP), not COR. LSP requires an existing, supported, more specific annotation. For the narrative function and location reviews, "annotation" also includes GO terms the curated gene review itself asserts (NEW rows and core-function terms). Prose in a review's description does not count, so a claim the review describes in words but no GO term captures can still be COR. The tie-break: CNN when the predicted term itself, or an equivalent, is already annotated to the gene; LSP when only a more specific descendant (is_a/part_of, checked in OLS) is annotated. These labels do not establish whether an example was in the model's training data. The assessment uses the available evidence, including studies published after the prediction release; it is not a time-restricted prospective benchmark.

PLI versus NPI. One rule applies:

• PLI: the target retains the domain and catalytic architecture of its family, and the prediction names the activity or substrate of a different paralogous subfamily. Examples are the wrong modifier for an E2 enzyme and the wrong substrate for a carrier.
• NPI: the prediction is refuted for reasons other than paralog confusion. Examples are a missing catalytic domain or residue, a taxon-impossible term, or a pseudoenzyme.
• Both reasons apply: a truncated record would also carry the wrong subfamily's activity. Here the architecture defect decides the call (NPI), and the review should say that the subfamily error is secondary.

Known inconsistency, not yet resolved. Two E2 cases are handled differently:

• human/UBE2F (F8WDQ9) ubiquitin-protein transferase activity is NPI. Its rationale combines the missing catalytic Cys116 in the 101-residue record with NEDD8-not-ubiquitin specificity.
• NEUCR/NCU04302, a SUMO E2, has its ubiquitin-specific narrative claims scored PLI.

Under the rule above, UBE2F stays NPI on architecture grounds. Its NEDD8 argument is a PLI-type reason, and the review does not separate the two. At the 2026-10-29 review snapshot there are 2 PLI among the 288 GO claims: ORYSI/B8BAB0 pollen maturation (ARGO-50) and HORSE/WEE1 female pronucleus assembly (HORSE40, a WEE2 oocyte process). There are 13 narrative records with a PLI judgment, so the boundary has so far mostly been applied to narrative text.

Scoring unit: the product, not the gene. Correctness is judged for the exact sequence ProtNLM2 was given. Many benchmark accessions are truncated or alternative gene models. When an activity, location or complex-membership prediction depends on a feature the accession demonstrably lacks relative to the canonical product, it is NPI (DOMAIN_ARCHITECTURE_MISMATCH), even if the term is true of the gene. Such features include a signal peptide, transmembrane segment, targeting signal, catalytic residues, or the region that mediates a binding. The absence must be established by analysis (hydropathy or signal-peptide test, residue or domain mapping), not assumed. For process terms, the rule above that absence of intrinsic activity does not exclude participation still applies. A process prediction is NPI only when the gene's known mechanism of participation is shown to require the missing feature. If the accession could still contribute another way (a regulatory, scaffolding or binding role that is neither shown nor excluded), the call is UNC. When the accession retains what the function needs, gene-level and family evidence apply as usual, and where it cannot be determined the call is UNC. Novelty is still judged at gene level, as above.

Fragment-driven NPIs. Three of the six MOD-evolution GO NPIs (one for UBE2F, two for mouse Spcs2) rest on the selected TrEMBL product being short or incomplete:

• UBE2F F8WDQ9: 101 residues;
• mouse Spcs2 A0A140LHW5: 74 residues, lacking the reference transmembrane architecture.

These NPIs judge the deposited sequence, not the gene, as the scoring-unit rule above requires. They would not transfer to the full-length product.

Reference standard and novelty

• Who made the reference calls. Every assessment is an AI-agent review (coding agents following the function-prediction review skill), grounded in cited literature, cached GOA/UniProt records and reproducible sequence analyses. The categories therefore measure biological validity as judged by that review standard, fixed in each review file before it is counted here. No human curator has re-scored a subset.
• OpenScientist's role. OpenScientist is a separate AI agent run on blinded, neutral hypotheses. Its reports are evidence inputs, not a second rater. Of the 30 ARGO-50 terms it gave a verdict on, the current reviews agree with 12, fall in the same polarity group for 5 and decide 13 differently, citing the report in each case. Horse and fly OpenScientist audits have since been incorporated into the reviews of five horse and five fly genes.
• GOA snapshot for COR/CNN. Novelty is judged against the cached *-goa.tsv of the target and of any other product of the same gene. The latest annotation date in 134 of the 162 files is in 2026; 24 files are empty. ProtNLM2 was trained on UniProt 2023_04. COR therefore means absent from a 2026 target record, not from the training data. A COR term may still have been learnable from annotated orthologs in the training release.

References

• UniProt ProtNLM help page — model and Evidencer documentation.
• ProtNLM2 accession list — public release coverage.
• de Crécy-Lagard et al. 2025 (PMID:40703034) — assessment categories.

Files and methods