Accessible Surfaceome (SurfaceBench) Comparison

Status: INCOMPLETE — handed off mid-analysis (2026-10-21). See
Handoff at the bottom before continuing.

Background

The Deliverome Project preprint The accessible human surfaceome
(https://deliverome.org/paper/accessible-surfaceome/; viewer
https://surfaceome.deliverome.org/; code
https://github.com/Deliverome-Project/accessible-surfaceome; data Zenodo record
23445521, v1.3.0) uses an LLM pipeline to call which human proteins are
accessible on the extracellular face of an intact cell. It releases:

• a catalog of 1,783 canonical + 1,252 likely surface proteins (5,332 genes
  given a "deep dive" in total), and
• SurfaceBench: 147 proteins on which five public databases (UniProt, GO,
  SURFY, CSPA, HPA) disagree, curated as yes (68) / contextual (31) / no (48)
  (https://surfaceome.deliverome.org/benchmark/).

The paper reports GO as a weak baseline (42.2% on SurfaceBench). This project
(a) makes sure every SurfaceBench protein has an ai-gene-review human review,
(b) checks our localization calls against the benchmark, and (c) asks how well
public GO annotations can answer the "accessible surface" question, and why
the paper's GO baseline scored so low.

1. Coverage

From scripts/overlap.py (results/catalog_overlap.tsv):

SurfaceBench: 147/147 reviewed. 43 already existed; the other 104 were
created in this project (branch surfacebench-reviews, one commit per batch).
The new reviews were written by parallel agents; no automated deep-research
provider was available (no API keys), so each gene's research and provenance is in
genes/human/GENE/GENE-notes.md. All pass just validate, and each has a history record.

Edits to existing reviews prompted by the benchmark: HSPA1A and HMGB1
plasma-membrane/cell-surface rows changed ACCEPT → KEEP_AS_NON_CORE, because their
surface pool is stress- or activation-induced and core locations are
nucleus/cytosol.

2. How the paper scored GO

From src/accessible_surfaceome/sources/go.py in the authors' code:

• Surface if any non-IEA, non-NOT annotation to GO:0009986 cell surface,
  GO:0009897 external side of plasma membrane, or GO:0005887 integral
  component of plasma membrane (each with is_a/part_of descendants).
• GO:0005886 plasma membrane was deliberately excluded ("too broad, includes
  cytoplasmic-face"), as was GO:0098552 side of membrane.
• Scored as binary accuracy with yes + contextual collapsed to surface.

Two problems found:

1. GO:0005887 is obsolete (replaced by GO:0005886) in the current GO
   (local build 2026-07-26). No benchmark protein carries it in the current GAF, so
   the rule effectively reduces to "cell surface or external side of PM". The
   anchored and intrinsic component terms (GO:0031225, GO:0046658,
   GO:0031362, GO:0031226) are also obsolete, so GO no longer has a live term
   for GPI-anchored or integral PM proteins.
2. The benchmark's own db_go column does not follow that rule. It flags
   KRAS, LYN, the JAKs, SYK, RAC1, RHOA, SYNE1/2 and IL6 as GO-surface (their only PM-type
   terms are cytoplasmic side of plasma membrane), and flags GCGR, SSTR2 etc. as
   not surface (they carry only plasma membrane). The term set that best
   reproduces db_go is outward CC + side of membrane (GO:0098552) + plasma
   membrane protein complex (GO:0098797): 131/147 agreement. That is the
   "Tier C" set the authors' code comments say was dropped on 2026-04-17. So the
   published 42.2% appears to count inner-leaflet proteins as GO surface calls.
   This is an inference from matching the column against the annotations, not a
   statement in the paper or code — confirm with the authors.

3. Public GO annotations vs SurfaceBench

scripts/score_gaf.py on goa_human.gaf (downloaded 2026-10-21; header
!date-generated: 2026-05-21T08:10). Accuracy uses the paper's convention
(yes+contextual = surface, n=147), and separately yes vs no only (n=116).

The TM functions in R2 are transmembrane signaling receptor activity
(GO:0004888), transmembrane transporter activity (GO:0022857) and channel
activity (GO:0015267), each with descendants. An extra inner-side exclusion rule
(R3: drop proteins whose only PM evidence is cytoplasmic-side terms) changes
nothing, because those proteins already fail R2. Plain PM looks competitive on
the 147-set only because it lets through 32/48 no proteins; the contextual-heavy
benchmark rewards that.

R2 errors (no IEA):

• yes missed (22) — fall into GO gaps:
• GPI-anchored (no live GO term): LYPD1, LYPD3, PSCA, DPEP1.
• Adhesion/junction (no receptor/transporter/channel MF): CLDN6, CLDN18, DSG3, DSG4, EPCAM, NECTIN4, MUC1, TACSTD2.
• Ectoenzymes: CA9, STEAP1, STEAP2, GUCY2C.
• Orphan or unknown MF: GPRC5D, GPNMB, SEZ6, DLK1, GYPA, TM4SF1.
• no called surface (5): ITPR1, ITPR3 (ER channels with a PM annotation),
  F2, TF, VEGFA (secreted ligands with cell-surface annotations). These five are
  disputed labels rather than clear GO errors; see §6. ITPR1 in particular has
  experimental evidence for a functional, outward-facing PM pool.
• contextual (18 not called surface): mostly lysosomal/secretory-pathway
  proteins with a minor surface pool (LAMP1/2, CD68, PMEL, TYRP1, TGOLN2, STIM1).

Per-gene calls: results/gaf_per_gene.tsv.

4. Our review calls vs SurfaceBench (secondary)

scripts/score_reviews.py, using accepted, kept, modified (replacement term) and
new rows plus core_functions locations:

Reviews add a little (a few NEW cell-surface rows, refinements to
cytoplasmic-side terms), but they follow GO convention and do not add cell
surface to every TM receptor. A blanket addition would conflict with the repo rule
against manufacturing annotations (see CLAUDE.md "Do not add what curators
deliberately declined to add").

Review-level calls worth expert eyes:

• DLL3: two experimental PM rows were MODIFYed to Golgi/late endosome, with a NEW non-core cell-surface row for the tumour pool.
• LGR5: called mostly internalized at steady state.
• IZUMO4: called soluble; its IEA membrane row was removed.
• F2: core location is external side of PM.
• TMED10 / KLK2: neither has a surface location.
• Source defects recorded:
• F2 thrombospondin receptor activity: the paper's receptor is CD36.
• FLT3 VEGF receptor rows: from FLT name confusion.
• FZD10 ISS rows: the WITH donor is the ligand WNT7B.
• CD24 interaction row: FGFR1 where the paper studies FGR.
• ABCB9 ER row: cites mis-keyed PMID 1101155 (should be 11011155).

5. Why "plasma membrane" is ambiguous for accessibility

1. Inner-leaflet / peripheral cytoplasmic proteins (KRAS, LYN, BTK, JAKs):
   cytoplasmic-side terms exist but are rarely used.
2. Outer-leaflet GPI-anchored proteins: now plain PM, since the anchored-component terms are obsolete.
3. Secreted ligands bound at the surface (F2, TF, VEGFA, latent TGFB1,
   LDL/APOB, C3b). GO:0009986 is defined as "The external part of the cell wall
   and/or plasma membrane", so these legitimately carry it. Note that cell surface
   is not an is_a/part_of descendant of PM.
4. TM proteins with minimal exposure: claudins and tetraspanins expose only
   small loops, and tight junctions hide them.
5. PM subdomains with tissue-dependent accessibility: apical vs basolateral,
   T-tubule, sarcolemma, cilium, postsynaptic membrane, acrosome.
6. Between membranes:
7. ER–PM contact site (GO:0140268; STIM1): the ER membrane reaching across to
   the PM at a junction. In the reasoned GO hierarchy it is under plasma
   membrane (part of the PM region), so a PM-descendant rule does pick it up — but
   it denotes the ER side of the junction, not PM insertion.
8. Intercellular junctions spanning two PMs: desmosome (GO:0030057) and gap junction (GO:0005921) are not under PM either, so PM-descendant rules miss DSG3 and the connexins.
9. Synaptic cleft.
10. Transient or vesicle-derived surface presence: lysosomal exocytosis (LAMP1, CD63), the phagocytic cup, exosomes.
11. Evidence artifacts: PM from membrane-fraction proteomics (HDA), and
    Reactome TAS assigning PM to a whole complex (the APOB/LDL case).

6. Disputed benchmark labels: location and function vs targetability

Three questions are easy to run together:

GO answers 1 and 2. It has no abundance threshold, and "core" vs "non-core" in our
reviews is a separate judgment about centrality to the gene's main function.
SurfaceBench defines surface as "accessible on the extracellular face of an intact
cell", which is question 1 plus topology. Some of its labels, however, read as
answers to question 3. When the benchmark and GO disagree, record which case it
is. Do not count it as a GO error, and do not change a rule to make GO agree
by discarding true annotations.

Effect on the headline numbers: relabelling ITPR1 as contextual moves R2 (no
IEA) from 69.4% to 70.1%. On yes vs no, with ITPR1 dropped from the no set,
it moves from 76.7% to 77.4%. The point is interpretive, not numerical.

Both reviews were updated (2026-10-24). The ITPR1 and ITPR3 PM rows now cite the
surface-IP3R literature (PMID:16840702, PMID:10828023, and for ITPR3
PMID:21062895) with findings/reference_review, replacing the placeholder "IBA
pool" reasons. The ITPR3 platelet row carries a structured part_of CL:0000233 platelet annotation extension to record the cell-type-restricted surface context
(see "Annotation extensions" below). ITPR1 keeps the subtype caveat in prose,
because the evidence does not establish that ITPR1 (vs ITPR3) is the surface
subtype. All rows stay KEEP_AS_NON_CORE.

Annotation extensions for contextual localization

Contextual localization is recorded structurally with an annotation extension
on the existing-annotation row, not in prose, wherever the context is grounded.
The slot is extensions (a predicate from the Relation Ontology plus a term).
Prefer a cell-type-specific CC term over a generic term plus an extension,
whenever GO has one. GO already carries cell-type-specific surface terms such as
GO:0097524 sperm plasma membrane, GO:1990913 sperm head plasma membrane and
GO:0097524-like leaf terms; using one in core_functions says "surface, in this
cell type" natively, which is cleaner than tagging a generic plasma membrane row
with a part_of extension. From the tissue_restricted_surface set:

• CRISP1 — core location GO:0097524 sperm plasma membrane (the exemplar).
• IZUMO1 — core location GO:1990913 sperm head plasma membrane (exposed at
  the equatorial segment after the acrosome reaction).
• LY6K — core location GO:0097524 sperm plasma membrane.
• ZP3 — core location is already the oocyte-specific egg coat.

Use a part_of extension only when GO lacks a specific term:

• BFO:0000050 (part_of) + a cell type (CL:…) or tissue (UBERON:…) — e.g.
  LRP4 cell surface part_of neuron; ITPR3 plasma membrane part_of platelet;
  ALPG external side of plasma membrane part_of testis (UBERON:0000473),
  because GO has no "germ-cell surface" term. This is the idiom for "surface only
  in cell type / tissue X" when no specific CC term exists. KLK2 gets neither: it
  is secreted (a disputed benchmark label; see §6), with no surface row.
• BFO:0000066 (occurs_in) + a CC — mainly for an MF/activity row localised to a
  compartment.
• RO:0002233 (has input) + a UniProtKB:… partner — substrate/partner context,
  the most common extension overall.

Generic GO:0005886 row with a subdomain/topology-specific reason — MODIFY
vs ACCEPT+NEW. When a plasma-membrane row is correct but the evidence places
the protein on a specific leaflet, subdomain or topological side, the deciding
question is whether the specific term corrects the generic one or merely adds
resolution to it:

• MODIFY → the specific term when the generic term is uninformative or
  actively misleading about topology/accessibility, so it should be retired into
  the specific one. This covers the uninformative parent GO:0016020 membrane →
  the real CC term (MSLN → GO:0009897), the inner-leaflet correction where bare
  plasma membrane wrongly implies surface exposure (GO:0005886 → GO:0009898;
  KRAS and the inner-leaflet set), and the "not in the PM at all" case (GO:0005886
  → GO:0140268 ER–PM contact site; STIM1).
• ACCEPT the generic row + add a NEW specific row only when the generic
  row's own evidence cannot resolve the subdomain while the subdomain is
  established by separate, independently-cited evidence. The intended case is one
  where the quoted evidence on each generic row stops at the generic term —
  e.g. CA9's GO:0005886 rows quote only "Human carbonic anhydrase IX (CA IX) is
  an integral membrane protein", which does not fix inner vs outer leaflet — while
  a NEW GO:0009897 external-side row carries its own topology evidence. (CA9's
  generic rows are a mix of IBA/IDA/IEA/TAS; their shared reason has been reworded
  to stop arguing the ectoenzyme topology, leaving the external-side claim on the
  NEW GO:0009897 row per the settled convention below. The distinction the page
  intends lives in what each row's quoted evidence resolves, not in its reason
  prose.)

The sharp discriminator is what each row's own evidence resolves, applied
per row: if the evidence supporting a generic row already fixes the subdomain
(a lipid anchor, a TM topology, a contact-site mechanism, or a quote that names
the subdomain), that row is MODIFY, because ACCEPT-ing it while the reason
argues a more specific term is supported is the mismatch CLAUDE.md forbids. Only
a row whose evidence genuinely stops at the generic term stays ACCEPT, with the
subdomain added as a separately-grounded NEW row.

Settled convention — ACCEPT-at-own-resolution for multi-source CC rows
(signed off by cmungall, 2026-11). For a plasma-membrane term carried by
several rows whose sources resolve to different levels, ACCEPT on a generic row
is a judgement about what that row's source established at its own level (an
IBA/IEA/Reactome row legitimately asserts only plasma membrane), not a claim the
term is as precise as the biology allows. The subdomain is added as a separate
NEW row on its own evidence. Two rules make this a disciplined exception rather
than a licence to leave generic rows vague:

1. The reason must not argue generality. It states "accepted at the resolution
   this row's evidence supports" and cross-references the NEW subdomain row —
   never "apical is proposed as a more specific term," which is the
   ACCEPT-while-arguing-too-general mismatch CLAUDE.md forbids.
2. Subdomain-resolving evidence lives on the NEW row, not bundled onto the
   generic rows. A quote that fixes the subdomain (e.g. apical targeting) is
   cited on the NEW GO:0016324 row; the generic rows keep only the evidence that
   stops at plasma membrane (surface localization, membrane integrality).

Applied to all three ectoenzyme/mucin cases in this PR:
- GUCY2C — five GO:0005886 rows ACCEPT at source resolution; their shared
anchor now carries only the surface-localization quote PMID:23269669, and the
apical quote PMID:11950846 sits on the NEW GO:0016324 row.
- MUC16 — generic PM rows ACCEPT with the reason no longer arguing apical;
the apical microplicae quote (PMID:17898272) moved off the generic row onto the
NEW GO:0016324 row.
- CA9 — generic PM rows ACCEPT on membrane-integrality evidence, reason
reworded to drop the external-side (ectoenzyme) argument; the external-side
topology sits on the NEW GO:0009897 row.

This exception is scoped to multi-source CC rows. A single row whose own
evidence fixes the subdomain (a lipid anchor, a TM topology, a contact-site
mechanism) still MODIFYs per the discriminator above — the blanket rule is
unchanged for those.

The stable_surface_attachment set is already handled by the same
specific-term-first logic and needs no extensions: C3 carries cell surface
(covalent thioester attachment to target surfaces) as a core location; CRISP1 uses
sperm plasma membrane; FN1 is core extracellular matrix with the integrin-bound
plasma-membrane rows non-core; TGFB1 is core extracellular region with the
GARP-presented cell surface pool non-core (GARP/LRRC32 presents it on Tregs,
platelets and more, so no single cell type to pin a part_of on).

The dual_localization set (ATP5F1B, B4GALT1, CD68, EREG, PMEL, STIM1, TGOLN2,
TMED10, TYRP1, VDAC1) is a subcellular dual residence, not a cell-type context,
so no part_of CL/UBERON extension applies. The right handling is to pick the
correct core compartment and the correct action for the surface pool, and the set
is handled consistently: one core location (mitochondrial outer membrane for
VDAC1, Golgi for B4GALT1, ER for STIM1, melanosome for PMEL/TYRP1, endolysosome
for CD68, TGN for TGOLN2, ER/ERGIC/Golgi for TMED10) with the plasma-membrane pool
KEEP_AS_NON_CORE — or MARK_AS_OVER_ANNOTATED where PM is actually wrong
(TMED10, an early-secretory cargo receptor), or NEW/ACCEPT where the surface
pool is a real added finding (TYRP1 melanoma cell surface; EREG membrane proform).
One row was inconsistent and is now fixed: ATP5F1B had its ecto-ATP-synthase
cell surface row KEEP_AS_NON_CORE but its plasma membrane row UNDECIDED
only because the publication was uncached; fetching Moser et al. 1999
(PMID:10077593, the angiostatin/surface ATP-synthase paper) resolved it to
KEEP_AS_NON_CORE with verbatim support. STIM1 was also refined: all six of its
non-core plasma membrane rows (one IBA, four SOCE-function IDA, one UniProt IEA)
were MODIFYed to GO:0140268 ER–PM contact site, since STIM1 stays in the ER
membrane and gates ORAI from the ER side at junctions rather than inserting into
the PM. The contact-site term is more specific than plasma membrane (a part of
the PM region in the reasoned hierarchy), so this is a refinement, not a lateral
move; the IBA row carries a propagation_review with TERM_SCOPING_PROBLEM. The
debated minor genuinely-PM-inserted STIM1 pool is noted in the reasons but is not
what these six translocation/function rows demonstrate.

Not every contextual mechanism is structurable this way, and the contextual
buckets split cleanly:

• tissue_restricted_surface (ALPG, IZUMO1, KLK2, LY6K, ZP3) — a cell type,
  captured by part_of CL/UBERON as above.
• cell_state_induced (CALR, DLL3, HMGB1, HSPA1A, HSPA5, HSPD1, SRC) and
  lysosomal_exocytosis (CD63, LAMP1, LAMP2, LAMP3, SCARB2) — a mechanism or
  condition (stress/apoptosis-induced display; fusion of a lysosome or granule
  with the plasma membrane). GO has no extension predicate for "surface only under
  condition X", so the correct representation is KEEP_AS_NON_CORE plus a prose
  reason, not a manufactured extension. GO does have the process term
  GO:1990927 (calcium ion regulated lysosome exocytosis), but the LAMPs/CD63 are
  cargo carried by that exocytosis, not performers of it, so adding it would be
  over-annotation (the "do not add what curators declined" rule). The lysosomal set
  is already handled consistently in our reviews (lysosomal membrane core, surface
  non-core). CD63 is the one exception, with a genuine constitutive surface pool as
  a core location — arguably the benchmark under-labels it (closer to yes than to
  exocytosis-only contextual).

Across the human corpus ~48 reviews use extensions (predicates seen: has input 134,
occurs_in 73, part_of 53, plus a few regulates/has-output). So extensions are the
sanctioned mechanism and should be used for contextual localization, but most
contextual-location calls today are still captured only as KEEP_AS_NON_CORE plus
a prose reason. Converting those to part_of CL/UBERON extensions where the
cell type or tissue is known is a worthwhile structural cleanup (candidate for a
follow-up pass; not done corpus-wide here).

Reproducing

projects/SURFACEOME/scripts/fetch_inputs.sh   # Zenodo v1.3.0 files + current goa_human.gaf.gz -> data/ (git-ignored)
projects/SURFACEOME/scripts/run_all.sh        # writes results/*.tsv and *.txt

run_all.sh needs the oaklib GO sqlite at ~/.data/oaklib/go.db (created by any
runoak -i sqlite:obo:go call). Results drift as GO and the GAF are updated; the
numbers above are from the run on 2026-10-21.

Handoff: next steps

The work is incomplete. In priority order:

1. Extend the public-GO rule (R2) in scripts/score_gaf.py, then re-run:
2. count cell-adhesion MF as TM evidence (e.g. GO:0098631 cell adhesion
   mediator activity), and possibly ectoenzyme evidence. Check FP risk against the no set.
3. count junction CC terms that sit outside PM (desmosome GO:0030057,
   tight junction / bicellular tight junction, gap junction GO:0005921)
   as surface evidence.
4. do not add rules whose only purpose is to remove ITPR1/ITPR3 (for
   example, dropping IBA PM rows). ITPR1 has a real, outward-facing PM pool, so
   that would make GO match a disputed label by discarding a true annotation
   (§6). Instead, report the score with and without the disputed labels.
5. Report every change in a new row in §3; never tune on per-gene names.
6. Confirm the db_go finding (§2, point 2) with the authors before
   publicizing it. Their code is pinned at commit 80930d0 (cloned 2026-10-21).
7. Inner-leaflet proteins → cytoplasmic-side term (partly done). The clean
   lipid-anchored set — KRAS, NRAS, RAC1, RHOA, LYN, HCK, BTK — has been
   refined: every GO:0005886 plasma-membrane existing-annotation row MODIFYed to
   GO:0009898 cytoplasmic side of plasma membrane (supported by each gene's
   inner-leaflet lipid-anchoring/recruitment evidence; several of these genes did
   not already carry a separate GO:0009898 GOA row, so the reasons cite that
   evidence rather than asserting the gene already carries the term), core_functions
   locations updated to match, and the IBA row given a
   propagation_review (TERM_SCOPING_PROBLEM). JAK2/JAK3/TYK2/GNAQ/GNB1 were
   already done by the review agents, and AKT2's three PM rows plus its
   core_functions.locations were converted in the same pass (it is PH-recruited
   but constitutively enough at the inner leaflet to resolve like BTK). This makes
   them machine-distinguishable from genuinely surface-exposed proteins (they fail
   both the outward-CC and the PM-AND-TM-function rules once PM is re-pointed).
   Still open: the transiently-recruited kinases BRAF, MAP2K1, SYK, IKBKB,
   LRRK2 — the inner-side call is softer there (they are recruited to the
   cytoplasmic face rather than constitutively lipid-anchored), so they were
   deliberately left for a separate judgement. Re-running §3/§4 scores after this
   change is a to-do.
8. Expert-review the flagged review calls in §4 (DLL3, LGR5, IZUMO4, F2,
   the source defects).
9. Optional: compare against UniProt topology (TOPO_DOM Extracellular,
   LIPID GPI-anchor) as a non-GO reference. The paper's best single database was
   UniProt's expanded rule (89.8%). The user asked to focus on GO first.
10. Branch surfacebench-reviews is PR
    #5312; push follow-up
    commits there, and get it green and merged. The project card in
    pages/projects/index.html is already added.

Work log: the session that started this project ran 10 parallel review agents
(10–11 genes each) over the 104 missing SurfaceBench genes, then committed each batch after
just validate passed. The scratch versions of these scripts were moved here.