Chapter 26 — Four Models Compared, and Where to Go Next

The four readings (scGPT, Tahoe, BulkFormer, EVA) each stood alone; this chapter lines them up. Five questions, in order: how would each model represent the same expression profile? which are built for single-cell, perturbational, or bulk data? which design choices actually earn their keep? which gaps do all four still share? and — the payoff for this book — which model-level research directions are the most realistic and valuable for the toxicogenomics foundation model the whole volume has been circling? The recurring lesson from 18_GENOMIC_FM_LANDSCAPE.md §18.7 holds throughout: in this field, data and evaluation usually move the needle more than architecture, so a fair comparison has to separate “clever design” from “favorable data/benchmark.”

One-sentence version: the four models are converging — from scGPT’s random-init, magnitude-binned single cells toward EVA’s knowledge-initialized, magnitude-preserving, count-modeled, cross-species samples — and the next transcriptomic (and toxicogenomic) model is mostly a matter of assembling the choices that already work and adding the four axes none of them cover: rat, dose, time, and fold-change.

26.1 The same profile, through four models

Imagine one small liver expression readout — a handful of genes after a compound exposure:

ALB (albumin) high,  CYP1A2 (drug metabolism) up,  ACTB (housekeeping) mid,  TP53 low

Because the four models live at different resolutions, “the same profile” is partly a thought experiment (scGPT/Tahoe want single cells; BulkFormer/EVA want a whole-sample vector). But run it through each and the representational philosophies line up cleanly:

Read the middle rows top to bottom and you see the field’s trajectory in one column-sweep: random → warm-started gene identities, binned → continuous values, MSE → count-likelihood objectives, single-species → ortholog-unified. scGPT sits at the “everything learned from scratch, magnitude discarded” end; EVA sits at the “everything primed with priors, magnitude and counts respected, species unified” end. Tahoe’s contribution is orthogonal to that axis — it is the only one whose data carries the intervention (drug + dose) — and BulkFormer’s is the modality (dense bulk) and the whole-genome context.

26.2 Who is built for what: single-cell, perturbational, bulk

The models are often lumped as “transcriptomic FMs,” but they occupy three different data regimes, and confusing them is the source of most unfair comparisons (recall EVA’s off-distribution baselines, §25.6).

Consequences that matter for the book:

• Perturbation lives only in Tahoe. scGPT, BulkFormer, and EVA pretrain on observational data; only Tahoe’s corpus was generated by intervention. Every one of the others must bolt on perturbation downstream (BulkFormer via LINCS fine-tuning; EVA via its treatment tasks), and all of them hit the same wall — perturbation prediction barely beats a linear baseline (18 §18.4).
• Toxicogenomics is bulk. DrugMatrix, TG-GATEs, and much of L1000 are bulk, so BulkFormer and EVA are the modality match; scGPT and Tahoe are not, and using them for tox means a resolution conversion (pseudobulk) or a type-error (11_SC_FOUNDATION_MODELS.md §3).
• Nobody is both perturbational and bulk and dose/time-resolved — which is exactly the empty cell toxicogenomics needs filled (§26.4–26.5).

26.3 Which design choices actually earn their keep

Separating genuine, portable wins from data/benchmark effects (18 §18.7). Ranked from most to least clearly beneficial:

Clearly worth adopting (ablated or mechanistically sound, architecture-agnostic):

1. Continuous, magnitude-preserving value encoding (BulkFormer’s REE, EVA’s MLP) over rank-binning (scGPT, Tahoe-x1). Binning discards the dose/effect-size axis by construction (§22.7); continuous encoding keeps it, and both bulk models chose it. The single most important representational fix.
2. Knowledge-initialized gene embeddings — EVA’s five-source warm start (protein + text + knowledge-graph + prior-model), BulkFormer’s ESM-2. EVA’s ablation shows it converges faster to lower loss. A cheap, strong prior that also enables cross-species transfer, and fully portable.
3. A count-aware objective (ZINB) — EVA — over MSE-on-values/bins. Respects the sparsity and overdispersion of real counts; the same reason scVI uses it.
4. Ortholog-unified cross-species vocabulary — EVA (human+mouse). The first-class way to get transfer, and (with ESM-2 init) nearly free to extend to new species. The book’s rat↔human bridge, demonstrated.

Effective but domain-specific:

5. Interventional pretraining data — Tahoe. Not an architecture at all, but the thing most likely to actually improve perturbation prediction; its absence is why the others fail that task.
6. Efficient attention for whole-genome context — BulkFormer’s Performer (linear), Tahoe-x1’s dense FlashAttention-2. Enabler, not accuracy driver: it lets you keep all ~20k genes instead of subsetting, which matters for bulk.
7. Set-level (cell-as-token) attention — STATE. A genuinely different inductive bias for modeling perturbation as a population shift; promising but not yet clearly decisive.

Probably over-credited:

8. Model scale alone. EVA shows clean scaling on held-out loss (no plateau at 305M), but headline task gains across all four are heavily confounded by data and benchmark choice; scale helps, but is not where the big wins have come from.
9. The GCN co-expression prior (BulkFormer). Helps, but risks baking in batch/technical correlation (§24.7); a learned or batch-corrected graph would be safer.
10. Custom generative masking (scGPT’s known/unknown scheme). Elegant, but Tahoe-x1 dropped it for plain dense attention and got a big efficiency win with no reported accuracy loss — evidence it wasn’t carrying much weight.

The through-line: the wins that survive scrutiny are representational (how you encode genes and values, what objective you use, how you unify species) and data-level (interventional vs observational). The pure-architecture flourishes (custom masks, graph priors, even scale) are the least load-bearing — a direct restatement of the book’s skepticism.

26.4 The gaps all four still share

Line the models up against what toxicogenomics actually requires and the same empty columns appear for every one:

Five gaps are shared by every model and are exactly the toxicogenomics axes:

1. No dose × time. Tahoe has 3 doses at one time point; the rest have neither. Toxicology is concentration × time — the one thing none of them models jointly.
2. No fold-change representation. All four ingest absolute expression; DrugMatrix/TG-GATEs/L1000 speak in signed treated-vs-control log-fold-change (11 §3). Every model would need an input/output mode change to be natively comparable.
3. No rat. Only EVA is cross-species, and it is human+mouse; the tox workhorse species is absent from all four. (The ESM-2/ortholog route makes this the most fixable of the five.)
4. Statistics, not mechanism. Every model is, on the evidence, a strong co-expression learner; none demonstrates causal, dose-dependent regulatory understanding — which is why they all fail the perturbation task against a linear baseline.
5. Evaluation you can’t fully trust yet. Favorable/in-house benchmarks, off-distribution baselines, and thin or absent independent replication recur across all four. The honest bar — beat a linear model on truly held-out perturbations — is cleared by none.

That empty region — interventional + bulk + dose/time-resolved + fold-change + rat-capable — is not a niche. It is precisely the toxicogenomics foundation model that does not yet exist, and the reason the book keeps returning to the gap.

26.5 The research directions most worth pursuing

Synthesizing the four improvement sections (§22.8, §23.8, §24.8, §25.8) into one prioritized agenda for a toxicogenomics transcriptomic foundation model, ranked by value × feasibility. Each names the source idea, the datasets, and the metric+baseline — the 21_TASK_ARITHMETIC.md §21.11 discipline (a defined evaluation before the model is built).

Tier 1 — high value, high feasibility (do these first; single- to few-GPU on existing checkpoints):

1. Start from BulkFormer or EVA-RNA, not scGPT. They are already the right modality (bulk), magnitude-preserving, and — EVA — cross-species and count-aware. The next tox model is a continuation of these, not a rebuild. (§24.8, §25.8)
2. Add a fold-change (treated-vs-control) input/output mode. Use matched controls to predict signed log-fold-change, aligning the model’s currency with tox. Data: DrugMatrix/TG-GATEs/L1000. Metric: signed-direction accuracy + magnitude correlation on held-out compounds vs a per-gene mean-shift and a linear baseline. (§24.8-1, §23.8-5)
3. Add dose + time conditioning tokens with a monotonicity constraint. Data: Open TG-GATEs (dose × time). Metric: hold out a dose/time, predict it; dose-ordering + interpolation vs linear-in-log-dose. (§24.8-2, §23.8-3)
4. Extend the ortholog-unified vocabulary to rat via ESM-2. EVA’s mouse↔human machinery + BulkFormer’s ESM-2 init make this mechanical: embed rat proteins, align orthologs, continue-pretrain on rat bulk. Data: rat DrugMatrix/TG-GATEs. Metric: rat→human direction transfer, stratified by ortholog conservation (20 §20.6). (§24.8-3, §25.8-2)

Tier 2 — high value, heavier (a pretraining run, but existing corpora):

5. Interventional + count-aware pretraining. Continue-pretrain on bulk perturbation corpora (L1000, DrugMatrix, TG-GATEs) with a ZINB objective and the fold-change + dose/time modes above — the one change most likely to finally beat the linear baseline on perturbation. Baseline: the linear model (18 §18.4), non-negotiable. (§23.8-4, §25.8-3)
6. Knowledge-initialized gene embeddings (5-source) for a tox model, so a small tox corpus inherits mechanism priors (pathways, targets) it can’t learn itself. (§25.8-1)

Tier 3 — important correctness/verification work:

7. Deconvolution-aware modeling (bulk mixes cell types; toxicant effects are often composition shifts). Add an auxiliary cell-fraction head. (§24.8-6)
8. Shortcut-proof, decision-relevant evaluation — held-out drugs/lines/species, score on DE genes + discrimination, always report the linear baseline, and validate cross-species integration with EVA-style interpretability probes (do rat and human orthologs align in the learned space?). (§23.8-4, §25.8-6)

What the composite looks like. Put Tier 1–2 together and the design writes itself: a BulkFormer/EVA-style bulk encoder — continuous values, ESM-2 + knowledge-initialized rat/mouse/human ortholog-unified gene embeddings, a ZINB objective — pretrained on interventional bulk perturbation data with fold-change output and dose + time conditioning, and evaluated against the linear baseline on held-out compounds and species. Every component already exists in one of the four models; none of the four combines them; and that combination is the toxicogenomics foundation model the book has been arguing for. The contribution is not a new architecture — it is the disciplined assembly of the choices that survived §26.3, aimed at the gap that survived §26.4.

26.6 Closing: convergence, and the one missing model

Read as a group, the four models tell a coherent story. The field started (scGPT) by treating a cell like a bag of ranked gene tokens with everything learned from scratch, and has moved — through Tahoe’s interventional data, BulkFormer’s magnitude-preserving bulk modeling, and EVA’s knowledge-primed, count-aware, cross-species representation — toward models that respect what transcriptomic data actually is. The remaining distance to a toxicogenomics foundation model is short and specific: four axes (rat, dose, time, fold-change), an interventional bulk corpus, and an honest linear-baseline evaluation. None of the four crosses that last stretch, but between them they have built every piece required to. That is the opening the rest of this book’s program (20–21, and the four readings here) is aimed at — and it is, encouragingly, an assembly problem more than an invention problem.

This chapter is synthesis, not new results; every claim traces to the four readings (22_READING_SCGPT.md–25_READING_EVA.md) and the sources cited there. The comparison deliberately foregrounds the book’s standing caveat — data and evaluation usually outweigh architecture — so the “which choices work” ranking (§26.3) credits ablated, portable, representational wins over pure-architecture flourishes, and the shared-gap table (§26.4) is what the research agenda (§26.5) is built to fill. Model facts (bins, params, objectives, species, corpora) are as stated and flagged in the individual chapters; several models’ headline numbers remain developer-reported without independent replication. This closes the four-model reading series.