TreeGrafter Failure Modes & Tree Placement

TreeGrafter Failure Modes & Tree Placement

← back to TreeGrafter Inference Evaluation

Frozen snapshot (2026-09-06). Every count on this page comes from the
sidecars generated at branch commit 49d8cc0b (main b62182cc) and is
deliberately not updated as the corpus grows; see the snapshot note at the
top of the main page
for what has changed since and how to refresh.

This sub-page drills into the 306 down-graded TreeGrafter annotations
(REMOVE / MARK_AS_OVER_ANNOTATED / MODIFY from the
main evaluation) and asks the question directly: does it
make sense where TreeGrafter placed the protein on the PANTHER tree?

We answer it without re-running TreeGrafter, because the placement is already
recorded for every protein:

The join (graft node + family + subfamily + propagated term + reviewer action)
is computed by analyze_placement.py into
treegrafter_placement.tsv. 305 of the 306 cases
recover a subfamily; 303 recover a graft node.

Every one of the 306 is then assigned to one of the four failure modes below
by classify_failure_modes.py →
treegrafter_failure_modes.tsv. The
assignment has two layers: a per-row curated table,
failure_mode_curated.tsv (163 rows, each with a
one-line justification), which always wins; and a keyword heuristic over the
GO aspect and the reviewer's review.reason for the rest (134 rows). The
operational rule that separates modes 1 and 4: if the PANTHER subfamily
name already describes the protein correctly and the bad term came from a
higher node, it is mode 1; if the subfamily itself is the wrong enzyme, it is
mode 4.

Both tables are joined on (file, term_id), never on the gene symbol: symbols
are not unique across the corpus (mdh in METEA and PSEPK, ALB in CANLF and
FELCA, the two PSEPK dapF paralogs), and a symbol-keyed join silently pulled
one organism's review.reason onto another's row.

Sixty-eight of the 134 heuristic rows are decided by construction rather than
by reading the reviewer: 61 cellular-component terms (mode 3 by aspect alone)
and 7 rows whose term is on an explicit low-information binding allowlist
(binding, protein binding, identical protein binding, small molecule binding — mode 3 whatever the reason says). Terms that name a real ligand
(ubiquinone binding, double-stranded DNA binding, metal ion binding) are
not on that list and are decided from the reviewer's reason like any other
row. That leaves 66 rows genuinely placed by keyword, which is the tier the
second-pass curation below should target.

Mode annotations share proteins MF BP CC
1 Granularity (family / node-level or sibling term) 143 47% 109 73 70 0
3 Generic / out-of-context localization, binding or process 109 36% 94 18 30 61
4 Within-superfamily mis-placement 41 13% 27 26 14 1
2 Pseudo-enzyme / co-opted fold 4 1% 2 1 3 0
0 Unclassified (heuristic declines; curation queue) 9 3% 8 5 4 0

The proteins column counts distinct review files, not gene symbols: the
898-row corpus spans 510 files but only 493 symbols, so a symbol-keyed count
merges the two mdh proteins, the two ALB proteins and eight other
cross-organism or paralogous symbol collisions. Counted by symbol, modes 3 and
4 read 92 and 26.

The 9 unclassified rows are deliberate: the reviewer's reason states a real
problem but not in words the heuristic can safely map to a mode (acoA,
benB, davA, groES, nuoM, PP_0094, I7J3R9, NCGR_LOCUS1270 ×2).
They are listed as mode 0 rather than pushed into the nearest bucket, and they
are the next rows to hand-curate.

Headline: placement is usually fine — the term is the problem

Reading the propagated GO term against the PANTHER subfamily name shows that
TreeGrafter rarely puts a protein in a grossly wrong part of the tree. The
fold/family assignment is almost always reasonable. What fails is the GO term
that rides along with the graft. Four failure modes account for nearly all the
down-grades:

1. Family-level over-generalisation (term granularity)

The protein is placed in the right family, but the propagated term is the
broad family function while the subfamily already knows something more
specific (or divergent).

Gene Propagated term (down-graded) PANTHER subfamily (where it landed)
eryAI / eryAII / eryAIII fatty acid synthase activity (MODIFY) INACTIVE PHENOLPHTHIOCEROL SYNTHESIS POLYKETIDE SYNTHASE**
mcr-1 / mcr2 / mcr-3 / mcr-4 LPS core region biosynthetic process (OVER), phosphotransferase activity, phosphate group as acceptor (MODIFY) PHOSPHOETHANOLAMINE TRANSFERASE EptA
ahpC thioredoxin peroxidase activity (MODIFY) ALKYL HYDROPEROXIDE REDUCTASE C

The erythromycin PKS modules are the clearest case: the family is named
"fatty acid synthase" (PTHR43775) and the family-level GO term fatty acid synthase activity was propagated — but the subfamily correctly identifies a
polyketide synthase. The mcr colistin-resistance enzymes are placed in exactly
the right subfamily (phosphoethanolamine transferase / EptA) yet inherited a
generic phosphotransferase MF and a lipid-A-vs-LPS-core process error. The
graft point is right; the inherited term is too coarse.

2. Pseudo-enzyme / loss of activity

Correct fold family, but the protein has lost catalytic activity or been
co-opted to a non-enzymatic role — something a tree graft cannot detect.

Gene Propagated term Subfamily Reality
OCTS1 glutathione transferase activity, glutathione metabolic process (OVER) GLUTATHIONE S-TRANSFERASE Octopus S-crystallin: GST fold, ~1/700–1/6000 of authentic GST activity; a structural eye-lens protein (PMID:7639695, PMID:27499004, PMID:8587103)
A0A8B6GS20 (MTMR9) phosphatidylinositol dephosphorylation (MODIFY), negative regulation of autophagy (OVER) myotubularin-related Pseudophosphatase: lacks the catalytic cysteine; regulates active MTMR partners

Genuine pseudo-enzymes turn out to be rare in this corpus — four
annotations on two genes — with one open question: two rows currently filed
as mode 4, K9IMD0 (draculin, a neofunctionalised lactoferrin homologue on
the LACTOTRANSFERRIN subfamily) and mdr (an Mdl1 family-level term on a
generic ABC-transporter subfamily), are contested — by the operational
rule below they read as mode 2 and mode 1 respectively — and await a
curator's call rather than a re-file by hand (the same treatment this page
gives its 9 mode-0 rows). Two cases listed here on the earlier snapshot have
been re-filed: TFP's enzyme regulator activity is an outdated node term
(specifier proteins are now known to be Fe(II)-dependent C–S lyases; mode 1),
and IRE1's unfolded protein binding is a generic/obsolete binding term on a
sensor (mode 3).

3. Generic / out-of-context localization and process

Low-information CC terms (cytoplasm, cytosol, plasma membrane, nucleus)
or organism-context BP terms inherited from a distant ancestor, correct-ish but
non-core or absent from the host. Examples: relA → plasma membrane, dinB
→ cytosol, fbp → sucrose biosynthetic process (a pathway the host does
not run), and several Pseudomonas putida genes. Note that zwf →
pentose-phosphate shunt and the mcr LPS-core process call are mode 1,
not mode 3 — both reasons state family/node-level propagation explicitly, and
they appear in the mode-1 table above.

4. True within-superfamily mis-placement (the cases a re-run would change)

A minority are placement errors: the protein lands in a structurally related
but functionally distinct subfamily of a shared fold superfamily. These are the
only down-grades where re-examining the actual graft point (or an independent
bioinformatic/structural check) would change the call.

Gene Propagated term (down-graded) Landed in subfamily Should be
aprA (Desulfovibrio) succinate dehydrogenase activity (REMOVE), electron transfer activity (its anaerobic respiration row is mode 1 — granularity, not mis-placement) SUCCINATE DEHYDROGENASE [UBIQUINONE] FLAVOPROTEIN (PTHR11632:SF51) Adenylylsulfate (APS) reductase α-subunit — shares the FAD fumarate-reductase/SDH flavoprotein fold but reduces APS, not succinate
fcs (P. putida) medium-chain fatty acid-CoA ligase activity (MODIFY), fatty acid metabolic process 2-SUCCINYLBENZOATE–CoA LIGASE Feruloyl-CoA synthetase — adjacent ANL adenylating-enzyme superfamily, wrong specific subfamily
mdh (P. putida and M. extorquens, ×2 each) L-lactate dehydrogenase (NAD+) activity, lactate metabolic process (REMOVE) L-LACTATE DEHYDROGENASE (PTHR43128:SF16; the family PTHR43128 is L-2-HYDROXYCARBOXYLATE DEHYDROGENASE (NAD(P)(+))) Malate dehydrogenase — LDH/MDH superfamily node
mqo1 / mqo2 / mqo3 (S)-2-hydroxyglutarate dehydrogenase activity (REMOVE) L-2-HYDROXYGLUTARATE DEHYDROGENASE, MITOCHONDRIAL Malate:quinone oxidoreductase
dapE, pepV acetylornithine deacetylase activity, L-arginine biosynthetic process (REMOVE) N-ACETYL-L-CITRULLINE DEACETYLASE (PTHR43808, M20A) DapE succinyl-DAP desuccinylase / PepV dipeptidase — ArgE branch of a mixed M20A family
Q53353, lsdB, Saro_0802, Saro_2809 carotenoid dioxygenase activity, carotene catabolic process (REMOVE/MODIFY) CAROTENOID 9,10(9',10')-CLEAVAGE DIOXYGENASE (PTHR10543) Lignostilbene / resveratrol dioxygenases (EC 1.13.11.43) — no stilbene subfamily exists
ptxD glyoxylate reductase (NADPH), hydroxypyruvate reductase (REMOVE) GLYOXYLATE/HYDROXYPYRUVATE REDUCTASE B Phosphonate dehydrogenase (D-2-hydroxyacid DH superfamily)
mupP phosphoglycolate phosphatase activity, DNA repair (REMOVE) PHOSPHOGLYCOLATE PHOSPHATASE (PTHR43434) MurNAc-6-phosphate phosphatase (HAD superfamily)
quiA quinoprotein glucose dehydrogenase activity (OVER) QUINOPROTEIN GLUCOSE DEHYDROGENASE Quinate dehydrogenase (quinone)
kdsC, lytN, ADAR2, pvdD, davD, lpdV, ech, galB, mdr†, PP_1257, K9IMD0† see treegrafter_failure_modes.tsv KdsC in a mixed CMAS/KdsC family; an amidase in a lytic-transglycosylase subfamily; ADAR on the ADAT branch; a pyoverdine NRPS module carrying EntF terms; LPD-val vs LPD-glc across P. putida E3 paralogs; …

† mdr and K9IMD0 are contested mode-4 calls (see the note under mode 2).

Mis-placements are 13% of the down-grades (41 annotations, 27 proteins) —
still the minority. This share was previously reported as 19% (58 annotations)
from a looser heuristic that filed any reason containing paralog or rather than as mode 4; reasons that explicitly say family-level propagation
(zwf, scpC, paaF/paaH/paaJ, the four aroQ rows) now correctly land
in mode 1, which is what the operational rule above says they are.
Almost all of them are the same pattern: a mixed PANTHER family whose
subfamilies are labelled by the best-studied member, so an uncharacterised
paralog in a different substrate class inherits that member's terms
(LDH/MDH, M20A, CCD/stilbene dioxygenase, SDH/FRD/AprA, PDH/AcoA).

Lightweight graft check (PANTHER vs InterPro, no re-run)

Rather than re-install InterProScan, we read the two classifications UniProt
already stores for each exemplar: the PANTHER family/subfamily (the TreeGrafter
graft point that propagated the term) and the InterPro signature entries (an
independent, signature-based opinion). Computed live by
graft_check.py →
treegrafter_graft_check.tsv.

Gene Propagated term (down-graded) PANTHER subfamily (graft) InterPro's specific call Diagnosis
aprA succinate dehydrogenase activity SDH flavoprotein, mitochondrial IPR011803 AprA PANTHER subfamily-resolution gap: no AprA subfamily exists, so it grafted onto the nearest (mito SDH) — but InterPro does identify AprA. An InterPro2GO call would beat TreeGrafter here.
OCTS1 glutathione transferase activity GLUTATHIONE S-TRANSFERASE IPR003083 S-crystallin InterPro names the lens crystallin; PANTHER subfamily stays "GST". Pseudo-enzyme invisible to the graft.
mcr-1 phosphotransferase, phosphate acceptor PHOSPHOETHANOLAMINE TRANSFERASE EptA IPR058128 Mcr1; IPR040423 PEA_transferase Placement is right (both methods); the propagated GO term is just a poor ancestral-node mapping.
eryAIII fatty acid synthase activity INACTIVE…POLYKETIDE SYNTHASE PKS ketoacyl-synthase / acyltransferase domains Placement is right (both methods say PKS); the propagated term is the family-level FAS term.
fcs medium-chain fatty acid-CoA ligase 2-succinylbenzoate–CoA ligase only generic AMP-dependent ligase domain Correct superfamily; no database has a feruloyl-CoA-synthetase-specific entry. Substrate unresolved by either method.

Two conclusions from the graft check:

  1. In 4 of 5 cases the placement is sound — the error is the GO term bound to
    the graft node    , not the tree position. Re-running TreeGrafter would
    reproduce the same (reasonable) subfamily. The fix belongs upstream in
    PAINT/PANTHER's node-to-GO mapping (propagate the subfamily-specific term;
    don't attach a catalytic MF to a pseudo-enzyme subfamily).
  2. InterPro frequently has resolution PANTHER lacks (AprA, S-crystallin,
    Mcr1). For these proteins a signature/InterPro2GO annotation would have been
    more specific or correct than the phylogenetic graft — a concrete argument
    for cross-checking TreeGrafter against InterPro2GO (see next steps).

OpenScientist blinded verification

To test the graft-check conclusions independently, each propagated term was
re-posed to OpenScientist as a blinded function-assignment hypothesis — the
agent saw only "GENE has \<propagated term>" (never the reviewer's action or
the PANTHER subfamily name) using the dedicated
treegrafter_function_hypothesis.md
prompt, which asks it to actively test the three failure modes. Reports and
provenance are committed under each gene's *-hypotheses/ directory.

Gene Blinded verdict Failure mode the agent assigned Decisive evidence it found Held-out reviewer action
OCTS1 REFUTED pseudo-enzyme / activity lost (#2) S-crystallin (IPR003083); lost catalytic Trp39; ~1000× lower kcat; PDB 5B7C MARK_AS_OVER_ANNOTATED
mcr-1 REFUTED wrong node term (placement right) (#3→node) EC 2.7.8.43 ⇒ correct MF is GO:0016780, not sibling GO:0016776; EC2GO mapping; error traced to a TAS annotation on EptA at the PTHR30443:SF0 node MODIFY
eryAIII REFUTED granularity, family-vs-subfamily (#1) DEBS3 type-I modular PKS (EC 2.3.1.94); family-level FAS term propagated over the PKS subfamily MODIFY
aprA REFUTED within-superfamily mis-placement (#3) APS reductase α (EC 1.8.99.2); absent covalent FAD-binding His required by all SDH/FRD catalysis REMOVE
fcs REFUTED within-superfamily mis-placement (#3) feruloyl-CoA synthetase (EC 6.2.1.34 ⇒ GO:0050563); aromatic-phenylpropanoid vs aliphatic-C6–C12 substrate, grafted onto the wrong acyl-CoA-synthetase branch MODIFY

All five blinded runs refuted the TreeGrafter term and independently
recovered (a) the correct specific function, (b) the failure mode, and (c) in two
cases the node-level provenance of the error (mcr-1's EptA/SF0 TAS source;
OCTS1's root-node propagation) — all matching both the held-out reviewer action
and the graft check
above. The aprA run even executed the template's active-site test, pinning the
refutation on a single missing catalytic histidine. (fcs required a
scope-narrowed re-run at max_iterations=2 after the first attempt hit the
7200 s API ceiling.) This is strong evidence that
a blinded LLM-agent function check is a viable QC layer over automated
phylogenetic annotations: it reliably catches exactly the cases TreeGrafter gets
wrong, without being told the answer.

Scale-up: 10 exemplars, two providers, family heterogeneity

The exemplar set was extended to 10 genes across 10 distinct enzyme families
(adding NaPMT3, ADAR2, NaUGT1, aceK, ahpC) and each was run blinded on two
independent providers (OpenScientist autonomous-compute + Falcon/Edison
literature). The blinded conclusion is now cited back in each source review as a
file:…/openscientist.md reference (in references and the annotation's
supported_by).

Gene Held-out action OpenScientist Falcon Independent specific call
OCTS1 OVER_ANNOTATED REFUTED (pseudo-enzyme) pseudo-enzyme S-crystallin, structural (GO:0005212)
mcr-1 MODIFY REFUTED (wrong node term) too general pEtN transferase ⇒ GO:0016780/GO:0043838
eryAIII MODIFY REFUTED (granularity) mis-placed DEBS3 modular PKS (EC 2.3.1.94)
aprA REMOVE REFUTED (mis-placed) mis-placed APS reductase α (EC 1.8.99.2)
fcs MODIFY REFUTED (mis-placed) mis-placed feruloyl-CoA synthetase (GO:0050563)
NaPMT3 REMOVE REFUTED (granularity) mis-placed putrescine N-MTase (EC 2.1.1.53, GO:0030750)
ADAR2 REMOVE REFUTED (mis-placed) mis-placed dsRNA/mRNA ADAR (not tRNA ADAT)
NaUGT1 REMOVE REFUTED (granularity) mis-placed UGT85A clade, wrong substrate
aceK MODIFY too general too general IDH kinase/phosphatase ⇒ GO:0101014
ahpC MODIFY too general too general AhpF-dependent peroxiredoxin

Both providers down-graded the propagated term on all 10 genes (8 REFUTED + 2
"too general" for OpenScientist; 7 REFUTED + 3 "too general" for Falcon), agreeing
on the failure mode in 8/10 and differing only in severity label on mcr-1 and
eryAIII (both still say the term must change). Cross-provider agreement on a
blinded task is strong evidence the QC signal is real, not a single-model artifact.

Family-level companion runs (Falcon) explain why. Each PANTHER family was
characterized for whether one GO MF term is safe to propagate. 9 of 10 families
are HETEROGENEOUS — a single substrate-specific term mis-annotates some
branches:

Family Verdict Why
PTHR11632 (aprA) HETEROGENEOUS SDH + fumarate reductase; GO:0000104 over-annotates FRD
PTHR43775 (eryAIII) HETEROGENEOUS FAS + modular PKS share the ketosynthase fold
PTHR11571 (OCTS1) HETEROGENEOUS active GSTs + co-opted crystallins
PTHR11558 (NaPMT3) HETEROGENEOUS spermidine synthases + neofunctionalized PMTs
PTHR11926 (NaUGT1) HETEROGENEOUS 14 plant UGT groups, divergent substrates
PTHR43201, PTHR30443, PTHR10910, PTHR10681 HETEROGENEOUS substrate/reductant/subfamily divergence
PTHR39559 (aceK) HOMOGENEOUS single IDH kinase/phosphatase function — the only family where the propagated term was right-but-too-general, not wrong

The single homogeneous family is exactly the one whose blinded verdict was "too
general" rather than "wrong" — family heterogeneity predicts the failure severity.
The actionable upstream signal: gate single-term GO propagation on a family
homogeneity check, and split or subfamily-annotate the heterogeneous ones in
PAINT.

What this means for the "re-run TreeGrafter" question

For the bulk of failures (modes 1–3, 84% of the down-grades),
re-running TreeGrafter would reproduce the same, sensible placement — the fix
belongs at the annotation level (propagate the subfamily-specific term,
gate catalysis-implying MF behind active-site checks, suppress generic CC), not
at the placement level. For the 13% in mode 4 (aprA, fcs, mdh,
mqo1–3, dapE/pepV, the stilbene dioxygenases, …), the placement itself is
the suspect — and because they cluster in a dozen mixed families (see the
family hotspots), the
durable fix is a subfamily split or subfamily-level annotation in PAINT rather
than a per-protein re-run.

The main page's corroboration analysis
adds the operational lever: every annotation on this page is by construction an
uncorroborated TreeGrafter call (GO_REF:0000118); the TreeGrafter calls
that another pipeline reproduced surface under GO_REF:0000120 and are
accepted 77% of the time.

Suggested follow-ups