eggNOG-mapper Evaluation

IN_PROGRESS EVALUATIONPIPELINE

Collection: Function prediction evaluation · Propagation by homology

eggNOG-mapper Evaluation

Bottom line: eggNOG-mapper
is the most widely used tool for annotating new genomes. It finds a protein's
orthologs in eggNOG and copies across the union of every ortholog's GO
terms. We ran it (v2.1.13, database 5.0.2, default settings) on 520 proteins
sampled from completed AI Gene Review reviews and compared each predicted
most-specific term with what the review concluded. Run normally, 40% of
predicted leaf terms match a supported claim exactly and two thirds of
reviewed core-function terms are recovered. That run is flattered, because
eggNOG already holds GOA annotations for most of these proteins and returns
them. To approximate a newly sequenced organism we hid the query's own species
from annotation transfer. Exact matches then fell to 33% of leaves and
core-function recall to 44%. A further 44% of leaves were related to nothing
in the review. In a manual check of 76 of those unrelated terms, about half
(39) were wrong and one in six (12) was a correct, informative addition. The
errors are characteristic: process terms that came from an ortholog's mutant
phenotype, terms that cannot apply in the query's lineage (eye development on
a worm protein), activities that belong to a paralog or a complex partner,
and source annotations that were wrong.
Molecular-function terms most often match the review exactly. But in a
24-claim sample per aspect of terms going beyond what the review supports, two
in three MF terms were wrong, against half or fewer for process and location
terms
(failure modes by ontology).

On three non-model genomes the profile depends on the lineage
(deep dive):
- P. putida and poplar: few terms per protein, mostly matching GOA's
electronic annotation.
- Horse: absent from eggNOG and annotated from all of Metazoa, so each
protein gets a long list. That list holds real mammalian knowledge that
horse GOA lacks, mixed with mouse-phenotype errors.

Restricting transfer to a narrow clade raises precision but leaves most
proteins with no GO at all.

We also found and fixed an emapper bug: --excluded_taxa never excludes the
named taxon itself (see Methods).

Why evaluate it

eggNOG-mapper output enters annotation pipelines, MAGs and comparative
studies, and it is often treated as roughly equivalent to curated orthology
transfer. Unlike PAINT/IBA it involves no curator, and unlike
TreeGrafter it does not place the query on a tree and take
one ancestral node's terms. It takes everything any ortholog in the chosen
orthologous group carries. Its GO output never appears in GOA, so the corpus
has no existing rows to tally: we ran the tool ourselves.

Methods

Cohort

build_cohort.py takes reviews in
which every live existing_annotations row has a non-PENDING action (at least
three rows) and the cached UniProt record has a sequence. It samples 40 per
organism, with a fixed seed, from human, mouse, rat, zebrafish, fly, worm,
budding and fission yeast, Arabidopsis, Dictyostelium, E. coli and
P. putida KT2440, plus 40 from all remaining organisms pooled (28 further
organism directories, from phages to sea urchin). That gives 520 proteins; DIAMOND found a
seed ortholog for 515
(cohort.yaml).

Running emapper

The emapper annotation database (eggnog.db) is a 41 GB SQLite file. That is
more than the disk where this was run, so
zchunk_build_http.py
streamed it into a store of independently compressed 256 KiB chunks (6.5 GB).
run_emapper.py serves that
store to SQLite through a read-only VFS. The bytes SQLite sees are identical to
the original:
verify_zchunk.py
re-streamed eggnog.db.gz and decompressed the store. Both gave SHA-256
4d6857c129423889edcd607995b28d97944787a588224f31dd7f745fe9ef1e4f over
41,370,988,544 bytes. emapper's own code is unchanged apart from the connection and the
taxon fix below. The seed search used emapper defaults (DIAMOND sensitive mode
with iteration, e-value 0.001) against eggnog_proteins.dmnd, and GO transfer
used the default --go_evidence non-electronic. The full pipeline is
scripts/run_all.sh.

Leave-species-out

Three annotation runs were made from the same seed orthologs:

Run Orthologs used for transfer
default all (what a user annotating these proteins gets)
species_excluded all except those from the query's own species (--excluded_taxa <species>); approximates a new genome (see Caveats)
species_excluded_experimental as above, and only experimentally supported GO terms (--go_evidence experimental), removing ISS/ISO/IBA terms that may themselves descend from the query

--excluded_taxa has a bug in emapper 2.1.13. normalize_target_taxa expands
each taxid to its NCBI descendants only. So --excluded_taxa 9606 excludes
Neanderthal and Denisovan records but keeps human ones, and
--excluded_taxa 4932 keeps every S. cerevisiae record filed at 4932.
Species whose eggNOG records sit at a strain-level taxid, such as E. coli
K-12 (511145), were excluded correctly, which hid the problem. Our first
exclusion run left human, mouse, rat, yeast, fission yeast and Dictyostelium
outputs byte-identical to the default run. The wrapper now adds the taxid
itself, which is what the help text ("taxa and all its descendants")
describes. The same bug affects --target_taxa. The species-level taxid of
each query comes from eggNOG's own taxonomy
(cohort_species_taxa.yaml).
One phage protein's taxon is missing from that taxonomy, so the seed
ortholog's own taxon was excluded instead.

Scoring

score_emapper.py reduces each
prediction set to leaf terms, then compares each leaf with the review
through the GO is_a/part_of closure (GO release 2026-03-25).

emapper's lists were propagated with a 2019–2020 GO, so leaf reduction uses
the union of ancestry from the 2019-07-01, 2020-12-08 and current releases.
With today's ontology alone, grouping classes that have since lost their
subclasses ("heterocyclic compound binding", "multi-organism reproductive
process") surface as 2,500 spurious leaves. Obsolete leaf terms without a
replacement are dropped (447 in the default run).

The review side has four sets:
- Supported: ACCEPT, KEEP_AS_NON_CORE and NEW rows, MODIFY replacements,
and all core_functions terms.
- Downgraded: MARK_AS_OVER_ANNOTATED and TOO_GENERAL rows, and the
original MODIFY term.
- Undecided: UNDECIDED rows, kept apart because they record that the
reviewer reached no judgment, not a negative one.
- Rejected: REMOVE rows and upheld NOT annotations. Removals of
placeholders (aspect roots, binding, protein binding) are not counted,
since they mean "uninformative" rather than "false". The exempt list is
deliberately narrow; widening it to other generic interaction classes is
left to a follow-up. When a review removes a term but supports one of its
descendants, the review is internally inconsistent, and BROADER wins by
entailment. This overrides the curator's stated reason for the removal. It
affects 1 leaf in the default and species-excluded runs, 3 in the
experimental-only run and none in the narrow-scope run. Two shapes occur.
In one, the review removed a parent term while keeping a child of it
(response to heat on rat Akt1; G protein-coupled receptor binding on rat
Hspa8, whose A1 adenosine receptor binding is kept). In the other, the
review removed a generic binding class two steps up (enzyme binding), so a
narrower class inherited the removal (protein kinase binding on Akt1).

Category Leaf term is…
EXACT a supported term
BROADER an ancestor of a supported term (true, less precise). It takes precedence over REJECTED, since a broader term is true by entailment
REJECTED a rejected term or one of its descendants
NARROWER a descendant of a supported term (more specific than the review; unverified)
DOWNGRADED related to a downgraded term
UNDECIDED related only to an undecided term
UNRELATED related to nothing in the review

Core-function recall counts a core_functions term as recovered when it lies
in the prediction's closure.

Because the reviews start from GOA, UNRELATED means "absent from current GOA
in any related form". It is not the same as wrong, which is why it was sampled
for manual review.

Results

Automatic comparison

Run Proteins with GO Leaf terms EXACT % BROADER % NARROWER % DOWNGRADED % REJECTED % UNDECIDED % UNRELATED % Core recall %
default 449 9,050 39.9 1.8 11.4 6.0 1.7 0.5 38.7 66.2 (1583/2391)
species_excluded 332 6,674 33.1 2.2 12.4 6.1 1.8 0.3 44.2 43.5 (1040/2391)
species_excluded_experimental 288 4,941 31.6 2.5 11.4 6.0 1.9 0.2 46.5 32.1 (767/2391)

Scores are computed against the reviews as of commit b9da9839c
(--reviews-at in the scorer), so they do not drift as the corpus changes.

The 515 proteins with a seed ortholog received a median of 10 leaf terms each
(up to 243). Large sets come from big, heavily annotated mammalian orthologous
groups, so pooled percentages are weighted towards them. Per protein, the
median share of leaves that were EXACT was 0.50 in the default run and 0.38
with the own species excluded.

By aspect, in the species-excluded run (% of leaves):

Aspect Leaves EXACT BROADER NARROWER DOWNGRADED REJECTED UNDECIDED UNRELATED
MF 1,095 50.9 2.6 4.6 12.3 2.6 0.2 26.9
BP 4,309 25.5 2.2 10.8 5.3 1.6 0.3 54.3
CC 1,270 43.3 2.1 24.5 3.4 1.6 0.4 24.7

Biological-process terms are two thirds of all leaves and over half of them
are UNRELATED. Molecular-function leaves match exactly twice as often. MF
leaves are also the most often DOWNGRADED, mostly protein-binding-type terms
(identical protein binding, protein kinase binding) that reviews mark as
over-annotations or replace with more specific terms.

Coverage depends on the query's lineage having other annotated species in
eggNOG (proteins with GO / core recall):

Stratum default species_excluded + experimental only
human 37 / 0.63 35 / 0.52 28 / 0.31
mouse 39 / 0.81 38 / 0.73 37 / 0.60
rat 40 / 0.81 40 / 0.78 39 / 0.57
zebrafish 34 / 0.72 33 / 0.63 32 / 0.52
fly 40 / 0.79 32 / 0.47 31 / 0.36
worm 36 / 0.73 29 / 0.52 29 / 0.45
budding yeast 38 / 0.74 28 / 0.48 19 / 0.26
fission yeast 36 / 0.66 30 / 0.45 30 / 0.39
Arabidopsis 38 / 0.50 17 / 0.14 12 / 0.11
Dictyostelium 36 / 0.69 1 / 0.01 1 / 0.01
E. coli 37 / 0.57 12 / 0.12 1 / 0.02
P. putida 20 / 0.38 19 / 0.37 16 / 0.26
other species 18 / 0.27 18 / 0.26 13 / 0.20

Manual adjudication

Two rounds of leaves were drawn from the species-excluded run with fixed
seeds (adjudication/).
Each draw picked proteins at random from those with a leaf in the stratum,
then one leaf at random within each protein:
- Round 1: 80 leaves (40 UNRELATED, 20 REJECTED, 20 NARROWER), at most
one per protein per stratum.
- Round 2: 72 leaves, 24 per GO aspect (12 UNRELATED, 6 REJECTED,
6 NARROWER each), from proteins not used in round 1. The limit was one per
protein per aspect, across strata. A protein could recur only in a
different aspect: 9 proteins did, so round 2 covers 63 distinct proteins.
Two of them recur within a stratum (once each in NARROWER and REJECTED);
the other seven span two strata. The strata in the table below are
therefore not drawn from fully disjoint protein sets.

AI reviewers judged each against the gene's review, UniProt record and notes,
using the COR/LSP/UNC/PLI/NPI/REP scheme of de Crécy-Lagard et al. 2025
(PMID:40703034). Counts below pool both rounds by the category each claim was
drawn from.

The rates are per protein: "for a random protein, is a random term of
this kind wrong?" They are not per term. A protein with 100 unrelated terms
had the same chance of being drawn as one with 2, so neither round is
proportional to terms, and whether the term-level rate is higher or lower
is untested. The two rounds also differ in aspect mix:
- Round 1 follows the protein-level population. Its 40 UNRELATED leaves
are BP 27, CC 8, MF 5, whereas BP is 79% of UNRELATED terms.
- Round 2 is balanced across aspects by design.

Round 1 alone gives 22 of 40 UNRELATED wrong (55%) and 6 correct (15%).
The pooled UNRELATED stratum is 76 distinct proteins, one leaf each.

Eight round-1 claims drawn as REJECTED changed category once placeholder
removals stopped counting. They are tallied in their own row,
since they are not a random slice of their new categories. The REJECTED row
is therefore conditioned on the claim still being rejected.

Drawn as n COR LSP UNC Wrong (NPI/PLI/REP)
UNRELATED 76 12 11 14 39
NARROWER 38 15 0 11 12
REJECTED (still rejected) 30 1 1 6 22
REJECTED, since recategorised 8 3 4 0 1

Error types among all 74 wrong calls (both rounds):

Error type n
CURATION_MISTAKE (source annotation wrong) 23
TAXON_CONSTRAINT_VIOLATION 13
MULTIPLE_FUNCTIONS (phenotype-derived process) 9
DOMAIN_ARCHITECTURE_MISMATCH 7
PARALOG_OVERANNOTATION 6
FREQUENCY_BIAS 4
LOCALIZATION_DEFAULT 4
COMPLEX_ACTIVITY_TRANSFER 3
IN_VITRO_NOT_IN_VIVO 2
PATHWAY_CONTEXT_IGNORED 2
PSEUDOENZYME_OVERANNOTATION 1

Examples for every row, split by GO aspect, are on the
failure-modes page. The three
low-count types (frequency bias, in vitro only, pathway context) are grouped
there, under "others" for MF and CC and "generic or out-of-context" for BP.

The pattern follows from the method. emapper takes the union of everything
any ortholog in the group carries, so one ortholog's mutant phenotype, one
lineage's anatomy, or one database's mistaken row becomes a prediction for
every member. Transfer at a deep level (Metazoa, Eukaryota) adds terms that
belong to one paralog or one complex partner. Single wrong rows are amplified
rather than outvoted, because the output keeps no ortholog-support threshold.

Failure modes by ontology

Round 2 (above) compares the three ontologies like for like. Of each
aspect's 24 claims, 16 MF, 12 BP and 10 CC were wrong. Each ontology fails in
its own way:

Across all 152 adjudicated claims, 23 of the wrong calls copied a mistaken
source annotation. The full catalog, with examples and rationale for every
wrong call, is on the
failure-modes page.

Narrowing the taxonomic scope

emapper's default auto scope transfers at broad clades: Metazoa, Fungi,
Streptophyta, Gammaproteobacteria. The pre-configured --tax_scope all_narrow
list reaches Mammalia, Brassicales and similar clades. Re-running the
species-excluded cohort with it:

Run Proteins with GO Leaf terms EXACT % REJECTED % UNRELATED % Core recall
species_excluded (default scope) 332 6,674 33.1 1.8 44.2 43.5%
species_excluded_narrow 92 1,723 51.7 3.9 27.2 11.8%

The leaves that remain match better, but three quarters of the proteins lose
all GO. The narrowest clade around the seed usually has no annotated member,
and nearly all that survive are rat and mouse proteins borrowing from each
other. Neither setting gets a broad scope's coverage without its errors.

Non-model genomes

The deep dive ran every completed
review for P. putida KT2440 (847), horse (30), poplar (113) and pennycress
(4). eggNOG holds essentially no GO for these organisms, so the default run is
already independent.

Genome Proteins with GO Median leaves EXACT % UNRELATED % Core recall Manual check: correct / wrong
P. putida 483 / 847 4 54.7 27.1 0.41 8 / 14 of 24
Horse 29 / 30 18 12.5 70.0 0.69 7 / 5 of 16
Poplar 84 / 113 4 26.8 44.6 0.35 4 / 11 of 16
Pennycress 4 / 4 2 25.0 50.0 0.27 1 / 4 of 9

The characteristic errors differ by lineage:
- P. putida: E. coli phenotypes and paralog pooling (GABA shunt on
DavD).
- Horse: mouse knockout phenotypes.
- Poplar: Arabidopsis chloroplast proteomics and paralog merges.

For genomes already in UniProt, emapper recovered fewer reviewed core
functions than GOA already held, and added a core term GOA lacked for about
one P. putida protein in ten.

Caveats

Only the displayed label was blinded. The reviewers still had to read the
gene's review, and for a claim drawn as REJECTED that review contains the
curator's REMOVE row for the same term, with its reason. The stability of
the wrong verdicts is therefore partly expected. The re-check rules out
anchoring on the displayed category, not deference to the curator's
visible verdict or blind spots shared by two AI passes over the same
files.
- Leave-species-out is only partly blind, and the bias favours emapper.
--excluded_taxa filters the annotation step only. The DIAMOND search still
finds the query's own eggNOG record as its seed ortholog (human queries seed
on 9606.ENSP…). That record then fixes the orthologous group, the
taxonomic level and the preferred name. A protein from a genuinely new
genome has no such record, so its seed is a more distant homolog and its
group assignment can be worse. The species-excluded numbers are therefore an
upper bound on what a new genome gets. The deep-dive genomes, which have
little or no own-species GO, are a closer test.
- Transfer is not independent of the reviewed annotations. Even with the
own species excluded, orthologs' terms can be ISO/ISS copies of the query's.
The experimental-only run bounds this.
- Ontology versions differ. eggNOG 5.0.2 is from 2020. Leaf reduction
accounts for that, but a few correct-then terms may now be obsolete or
moved.
- Only GO is scored. emapper also reports KEGG KO, EC, CAZy, BiGG, PFAM
and preferred names, which are not evaluated here.

Files

File Contents
cohort.yaml, cohort.fasta the 520 proteins and their sequences
runs/<run>/<run>.emapper.annotations raw emapper output for each run
results/<run>_leaf_scores.tsv one row per predicted leaf, with category
results/<run>_protein.tsv per-protein counts, core recall and missed core terms
results/summary.yaml all aggregate tables
adjudication/batch_*.yaml, adjudication/round2_batch_*.yaml the 80 + 72 adjudicated claims with rationale
adjudication/summary.yaml adjudication tallies, overall and by aspect
adjudication/failure_modes.yaml every wrong call, by aspect and error type
adjudication/blind_recheck/ blinded re-check: batches, key, comparison script and summary
failure-modes.md failure modes by ontology, with examples
deep-dive.md, deep_dive/ non-model genome cohort, runs, results and adjudication

To reproduce: projects/EGGNOG_MAPPER_EVALUATION/scripts/run_all.sh WORK_DIR
(about 17 GB of disk and two hours, mostly the DIAMOND search), then
uv run python projects/EGGNOG_MAPPER_EVALUATION/score_emapper.py and
uv run python projects/EGGNOG_MAPPER_EVALUATION/adjudication/summarize.py.

Next steps