Chapter 18 — The Foundation Model Landscape for Biology, and What We Can Do With It

A deep but plain-language study of every kind of genomic / DNA / RNA / single-cell / protein foundation model: how many there are, how they differ from GPT, how they’re trained, and — the part your mentor cares about — what we can actually do with them on our toxicogenomics projects.

This chapter is the wide-angle companion to 11_SC_FOUNDATION_MODELS.md, which goes deep on the single-cell models specifically. Read that one for the worked numeric examples; read this one for the whole map.

18.1 What “foundation model” means here, and why the word is slippery

A foundation model is one model pretrained once, on a huge pile of unlabeled data, with a self-supervised objective (predict a hidden part of the input), that you then reuse for many downstream tasks. GPT is the famous example: pretrain on text by predicting the next word, then reuse for translation, summarization, coding.

In biology the word gets attached to models trained on five very different kinds of data. It’s worth separating them up front, because a “genomics foundation model” and a “single-cell foundation model” are almost nothing alike — different input, different objective, different use.

The first three read sequences — a molecule spelled out letter by letter. The last two read profiles — a snapshot of how much of each gene is switched on. Our data (DrugMatrix, TG-GATEs) is the profile kind. So the single-cell and perturbation families are the ones we’d use directly, and the sequence families matter mostly as feature providers (especially protein models — hold that thought for §18.5).

18.2 How these differ from GPT — the five real differences

Everyone’s mental model is GPT, so the fastest way to understand a biology FM is to list exactly where it departs from GPT. There are five departures, and every model in this chapter is some combination of them.

Difference 1 — the token is not a word

GPT’s token is a wordpiece. A biology FM’s token is whatever the smallest meaningful unit of its molecule is:

• DNA/RNA models: a nucleotide, or a k-mer (a short run like ACGTAC), or a BPE-merged chunk of nucleotides.
• Protein models: an amino acid.
• Single-cell models: a gene. This is the big one. scGPT’s “vocabulary” is ~60,000 genes instead of ~50,000 wordpieces.

Difference 2 — often there is no order, so no positional encoding

This is the deepest difference, and it’s specific to the profile models.

In a sentence, order is meaning (“dog bites man” ≠ “man bites dog”), so GPT adds a positional encoding to every token. In a cell, the genes have no order — a cell is a set of (gene, amount) pairs, not a sequence. So scGPT and its relatives drop positional encoding entirely. Instead they inject the expression level where GPT would inject position.

The one-line version: scGPT is BERT where “position” has been replaced by “how much of this gene is present.” A cell is a set, not a sequence, so it must be.

The sequence models (DNA/RNA/protein) do have order — a genome is a string — so they keep positional information. But they need it to reach across enormous distances (a regulatory element can sit a million letters from the gene it controls), which drives the architecture choices in Difference 4.

Difference 3 — the objective is usually “fill in the blank,” not “predict the next thing”

GPT is autoregressive: predict the next token, left to right. That’s great for generation.

Most biology FMs are instead masked (BERT-style): hide some tokens and predict them from both sides. Why? Because for DNA, RNA, protein, and cells you usually want a representation (an embedding you can reuse), not left-to-right generation. And because there is no natural “left to right” for a cell.

The exceptions are the generative genome models — Evo / Evo 2 are autoregressive over DNA, precisely because they’re built to design new sequences, which is a generation task like GPT’s.

Difference 4 — the architecture is often not a plain transformer

GPT is a stack of self-attention layers. Self-attention costs $O(n^2)$ in the sequence length $n$. That’s fine at $n$ = a few thousand words. It’s fatal at $n$ = a million nucleotides. So the sequence models reach for cheaper long-range machinery:

• HyenaDNA replaces attention with long convolutions → handles ~1 million-nucleotide context at single-base resolution.
• Caduceus uses the Mamba state-space architecture, plus a trick so the model reads a strand and its reverse-complement the same way (DNA is double-stranded).
• Evo 2 uses “StripedHyena 2,” a hybrid, to reach a 1-megabase context with 40 billion parameters.

The profile models face the opposite squeeze. A cell has ~20,000 genes, and our own TransTissueFormer has 8,565–31,042. Plain attention over 31,042 genes needs ~2 TB of memory (see 07_TRANSTISSUEFORMER.md §7.4). So they compress first: scFoundation only reads the non-zero genes; TransTissueFormer pools all genes into 512 “slots” before any attention runs.

The pattern across the whole field: plain quadratic attention is the thing nobody can afford, and each model is defined largely by how it escapes it.

Difference 5 — the “value” problem, which has no analog in GPT

A word is a discrete symbol. But a gene comes with a number — its expression level. GPT never has to answer “how do I feed a 450 and a 0.42 into a transformer?” The profile models do, and it’s where they differ most from each other, and where they break on our data:

• scGPT bins the number and looks up a bin-embedding.
• Geneformer throws the number away and keeps only the rank (which gene is most expressed, 2nd, 3rd…).
• scFoundation encodes the number but only for non-zero genes.

This “value encoder” is the single most important thing to understand for our purposes, because it is the part that breaks on toxicogenomics data — and the part we route around. That’s §18.5 and all of 11_SC_FOUNDATION_MODELS.md §3.

18.3 The catalog — how many are there, and what each one is

There are dozens — well over a hundred if you count every variant. The community-maintained Awesome-Bio-Foundation-Models catalog organizes them into exactly six families: DNA & Gene, RNA, Protein, Single-cell, Multimodal, and Pathology. That’s the same split I use below, with two additions (multimodal and pathology) at the end, because pathology models turn out to be directly relevant to toxicology.

The important thing is not the count. It’s that all hundred-plus models cluster into a manageable number of designs — a few objectives, a few architectures, a few answers to the value problem. Learn the design and you’ve learned the family. Here is the working set, grouped by modality, with the one thing that distinguishes each. (Named models beyond the headline ones are listed so you can recognize them in papers; you don’t need all of them.)

A. DNA sequence models — “read the genome as text”

These learn from raw DNA, with no expression data at all. They’re good at variant effects, regulatory-element detection, and sequence design.

Others you’ll see in this family: GENA-LM (open long-sequence DNA LMs, up to 36 kb, extended by a recurrent memory mechanism; Nucleic Acids Research 2025, open weights), DNAGPT (generative, multi-task), GPN-MSA (alignment-based, genome-wide variant effects), DNABERT-S (species-aware embeddings), EpiGePT (epigenomics), and GROVER. The recurring theme: transformers win short-range classification (promoters, splice sites), long-context architectures (HyenaDNA, Evo, Caduceus) win long-range interactions (enhancer–gene links, variant effects).

The DNA side has two design axes worth naming, because they pull in opposite directions:

1. Scale plus long context — bigger models with longer windows. Evo 2 and AlphaGenome.
2. Efficient, biologically-principled architecture — smaller models whose built-in assumptions match the structure of DNA. Caduceus and HyenaDNA.

A 2026 independent benchmark (GENEB, §18.4) found the second axis often wins: architecture and pretraining alignment frequently outweigh raw parameter count. Here are the four headline models in detail, because the specifics matter.

Evo 2 — the scale-and-context frontier. Trained on 9.3 trillion DNA base pairs across all domains of life. Two sizes: 7B (trained on 2.4T tokens) and 40B (9.3T tokens), both with a 1-million-token context at single-nucleotide resolution. Training ran in two phases: first an 8,192-token window focused on gene-rich regions, then a midtraining phase that stretched the context to 1 million tokens. The architecture is StripedHyena 2, a hybrid of convolution and attention operators — 1.3× faster than an optimized transformer at 16k context, 3× faster at 1M. It can retrieve a 100 bp “needle” from anywhere inside a 1-million-bp haystack of random DNA. It is fully open: weights, code, and the OpenGenome2 dataset (8.8 trillion nucleotides) are all released. On variant effects it is honestly mixed: for coding single-nucleotide variants the 40B and 7B models ranked only 4th and 5th, behind AlphaMissense, ESM-1b, and GPN-MSA. For noncoding variants it beats all others, gets the highest zero-shot score on splice variants, and set a new state of the art on BRCA1 noncoding variants — all with no task-specific fine-tuning. An honest negative the authors volunteer: Evo 2’s likelihood shows no correlation with the fitness of viruses that infect humans, because those sequences were deliberately excluded from training.

AlphaGenome — regulatory variant effect at base-pair resolution. It reads 1 megabase of DNA and predicts thousands of functional tracks at single-base resolution: 5,930 human (or 1,128 mouse) tracks across 11 modalities — gene expression, splicing, chromatin state, and chromatin-contact maps. It matched or beat the strongest external model on 24 of 26 variant-effect evaluations, and hit state of the art on 22 of 24 track-prediction tasks. It exists to fix a real tradeoff: base-resolution models like SpliceAI were capped at ~10 kb (missing distal regulators), while longer-context models (Enformer, Borzoi) reached 200–500 kb but only at coarse 128 bp / 32 bp output. AlphaGenome gets both at once, via a two-stage pretrain-then-distillation scheme (one student model reproduces an ensemble of teachers), running under 1 second per variant on an H100. The catch for a lab: it is API-only and non-commercial — you cannot freeze and audit the weights. That matters because over 98% of human genetic variation is noncoding, exactly where AlphaGenome is strongest.

Caduceus — the case that architecture beats scale. It is the first reverse-complement (RC) equivariant, bi-directional, long-range DNA model, built on Mamba state-space blocks instead of attention. The biological reasoning is clean: DNA has two strands that are reverse complements carrying the same information, so building that symmetry into the model as an inductive bias should help — and it does. On a hard long-range variant-effect task, Caduceus beat models 10× larger that lacked bi-directionality and equivariance, with the advantage strongest at long distances from the transcription start site. Mamba handles hundreds of thousands of nucleotides without attention’s quadratic cost.

HyenaDNA — the proof that long context need not be expensive. It reached 1-million-token context at single-nucleotide resolution (a 500× jump over prior dense-attention models), scales sub-quadratically, and trains up to 160× faster than a transformer at long sequence lengths. On the Nucleotide Transformer benchmark it hit state of the art on 12 of 18 datasets with 1,500× fewer parameters (≈1.6M vs the 2.5B Nucleotide Transformer) and 3,200× less pretraining data. A caveat from independent evaluations: its accuracy is more mixed than its efficiency — for sequences under ~32 kb, DNABERT-2 or Nucleotide Transformer often score better despite costing more.

DNABERT-2 made the efficiency argument on training cost. Swapping k-mer tokenization for byte-pair encoding (BPE) — which merges the most frequent co-occurring genome segments into a learned vocabulary — it matched the prior state of the art with 21× fewer parameters and ~92× less GPU time, and beat the original DNABERT on 23 of 28 GUE-benchmark datasets. Concretely: ~14 days on 8 gaming GPUs versus 28 days on 128 A100s for the model it matched. (BPE also fixed a subtle bug: overlapping k-mers leak information across adjacent tokens, which can inflate a score without any real learning.)

B. RNA sequence models — “read the transcript”

Same idea as DNA models, applied to RNA molecules. Mostly used for structure prediction and RNA-property tasks, not for expression.

C. Protein language models — “read the protein”

Trained on amino-acid sequences by masked modeling. These matter to us more than they look, because of one downstream use (§18.5): a gene’s protein sequence is a species-agnostic fingerprint of that gene.

This is the largest and most mature family by far — the catalog lists ~50 protein models against a handful for cells. That maturity is why the protein route (ESM2 inside UCE) is the most reliable way to get a species-agnostic gene feature for our rat data.

A few specifics worth carrying, because protein models are where FMs first earned real trust:

• ESM-2 / ESMFold. As the model scales from 8M to 15B parameters, an atomic-resolution picture of protein structure emerges in the learned weights — nobody told it structure; it fell out of masked-language pretraining. Because ESMFold works from a single sequence (no multiple-sequence-alignment step), it is 1–2 orders of magnitude faster than prior pipelines. That speed produced the ESM Metagenomic Atlas: structures for 617 million proteins (225M high-confidence) in two weeks on 2,000 GPUs, of which 12.6% had no match to any experimentally determined structure.
• The speed-vs-accuracy tradeoff is quantified. An independent 2025–2026 benchmark on hard targets (1,666 monomers, 994 dimers, all <40% sequence identity) found AlphaFold2/3 correct on 88% of monomers and 77% of dimers, versus ESMFold’s 76% and 41%. The rule that survives: ESMFold for speed and scale, AlphaFold for accuracy, and the gap widens sharply on complexes.
• AlphaFold3 / RoseTTAFold All-Atom push to all-atom modeling — proteins plus DNA, RNA, small molecules, metals, covalent modifications. RFAA does it by expanding the residue alphabet to 28 (20 amino acids + 4 DNA + 4 RNA bases). But know the failure modes: AF3 shows chirality violations in 4.4% of top predictions, can hallucinate structure, and is unreliable on antibodies (each B cell’s sequence is uniquely shuffled, so there’s no usable alignment). And a generalist all-atom model underperforms specialists on any single interaction type — while specialists simply fail when a complex mixes several. “Generalist vs specialist” is the recurring shape of the whole field.
• ESM3 is the multimodal, generative direction (sequence + structure + function jointly). Treat its headline numbers cautiously — they are hard to verify against a primary archived source.

D. Single-cell / transcriptomic models — “read the cell”

This is the family we’d use directly. All of them are BERT-over-genes with different answers to the value problem (Difference 5). 11_SC_FOUNDATION_MODELS.md has the worked numbers; here’s the map.

The independent verdict (2025–2026 benchmarks): across 11 models and ~29 datasets, scGPT, Geneformer, and CellFM come out as the most usable/robust overall, with Geneformer and scFoundation strong on gene-level tasks. But — see §18.4 — “best of the FMs” is not the same as “beats a simple baseline.”

E. Perturbation / cellular-response models — “read how the cell moves”

These are the most directly relevant to toxicology, because toxicology is a perturbation problem: you dose a cell/organ and ask what changes.

STATE’s SE + ST split — “where am I” plus “how do I move” — is the same additive decomposition as CPA’s basal + drug and ToxCompl’s bias + P·Q. Three separate literatures arriving at the same structure is a strong hint it’s the right one (see 11_SC_FOUNDATION_MODELS.md §6).

F. Multimodal models — “read two languages at once”

These bind two of the above modalities into one model, so a representation learned in one lands in the space of the other. Examples: ESM3 (protein sequence + structure + function), ProtST (protein + biomedical text), GeneCompass (expression + prior knowledge, across species). UCE belongs here in spirit — it binds gene identity to protein sequence.

Why we care: multimodal binding is exactly the trick that gets us across the gaps that block us — species (rat↔human via protein), platform (probe↔probe via the shared gene), and eventually modality (expression↔histopathology). Every bridge in §18.6 is a small multimodal model.

G. Pathology models — “read the slide” (more relevant to tox than it looks)

These are foundation models over whole-slide histopathology images (H&E-stained tissue), pretrained on millions of image tiles: UNI, CONCH, Virchow, GigaPath, and others. They’re a different data type entirely (pixels, not sequences), so they don’t touch our expression pipeline directly.

But hold on — histopathology is one of toxicology’s core endpoints. DrugMatrix and TG-GATEs pair each expression profile with a pathologist’s scoring of tissue damage (necrosis, fibrosis). ToxCompl’s own biological validation predicts “apical endpoints” — i.e. histopathology (04_TOXCOMPL.md §4.12). So a pathology FM is the natural encoder for the other half of the tox data. The genuinely ambitious version of our program is multimodal: predict the histopathology embedding from the expression profile, and vice versa — a transcriptome↔pathology bridge. Nobody in the four papers has done this, and it’s the kind of “deep thinking” direction worth raising with your mentor (see §18.6, Experiment 7).

18.4 How they’re trained — the taxonomy, and the honest scorecard

The training recipes, in one place

Strip away the biology and there are only a handful of self-supervised objectives in use. Knowing which one a model uses tells you most of what it can and can’t do.

Two axes then separate the good from the mediocre:

• Data scale: 20–50M cells for single-cell models; 9+ trillion nucleotides for Evo 2. Interestingly, single-cell benchmarks suggest 20–30M cells is already “enough” — more data stops helping, which itself says the ceiling is set by something other than scale.
• Observational vs interventional data: this distinction turns out to be decisive, and it’s the crux of the critique below.

Read the literature with two ledgers

Here is the single most useful habit for reading this field, borrowed from a 2026 clinical-genomics review. Keep two ledgers as you read.

• The capability ledger records what a model can demonstrably do at scale. This is what the paper’s abstract and the press release report.
• The validity ledger records what still holds up when you push each claim through an independent test set with an honest baseline the model actually has to beat.

The marketing reports the first ledger. A lab has to act on the second. The two often diverge inside the same model. Example: Evo 2 sets a real state of the art on noncoding BRCA1 variants (capability), while five foundation models fail to beat a linear baseline on perturbation prediction (validity) — both are true, in the same field, sometimes the same model.

The honest-evaluation problem, before any number

The single most important 2026 result for us is not a model. It is GENEB, a benchmark that took 40 genomic foundation models and evaluated their frozen representations across 100 tasks in 13 functional categories, under one common probing protocol. Its conclusions:

• Aggregate leaderboards are unstable — model rankings reshuffle sharply as you move across task categories. No model is best everywhere.
• Scale gives only modest and inconsistent gains. Architectural and pretraining alignment frequently outweigh parameter count. On its size-vs-performance frontier, small architecture-aligned models sit on the frontier while some large models fall below it.
• The same model gets called a breakthrough in one paper and an underperformer in another — not because the evidence conflicts, but because there was no common evaluation framework until now.
• Crucially, none of the 40 models was built by GENEB’s authors or funders. There is no vendor incentive in the result.

Before trusting any single benchmark number, know the five traps that make honest comparison hard. An expert reader assumes you don’t know the field if you skip these.

1. Tokenization is not neutral. DNA models disagree on what a token even is. HyenaDNA and Evo work at single-nucleotide resolution; DNABERT-2 uses byte-pair encoding; older models use overlapping k-mers. Overlapping k-mers leak information across adjacent tokens, which can inflate a score with no real learning. And the resolution at which a model tokenizes limits the resolution at which it can call a variant. Two models reporting the same accuracy may be solving slightly different problems.
2. Context length is asymmetric. Early transformer DNA models saw 512–4,000 tokens — under 0.001% of the human genome — which structurally prevents modeling long-range effects. But regulatory elements up to 1 million base pairs away matter. A 4 kb model and a 1 Mb model are not competing on a level field for an enhancer question, and many published comparisons pretend otherwise.
3. Contamination is the default, not the exception. These models pretrain on public references (GRCh38, ClinVar, gnomAD) and are then tested on benchmarks built from those same references. A high retrospective AUROC may just mean the model memorized labels already in its training distribution. That is not the same as behaving well on a genuinely novel case. This gap — retrospective discrimination vs prospective utility — is the single most important caveat in the whole field.
4. Benchmarks are fragmented and unstable. Different papers use different tasks and metrics. Efforts like GUE (DNABERT-2: 36 datasets, 9 tasks, 70–10,000 bp), BEND, and OmniGenBench exist precisely because results didn’t compare across studies.
5. DNA is not protein. Protein structure had CASP, a curated competition that made AlphaFold’s progress legible. Genomics never had a folding-competition equivalent. And DNA is genuinely harder: signal spans very long ranges, high-signal regions are sparse, and signal density is lower than in proteins. BEND’s finding is that DNA-LM embeddings approach expert methods on some tasks but capture only limited long-range information. Treat any DNA result presented with protein-level confidence skeptically.

The scorecard, task by task

With those caveats set, here is what actually holds up.

GREEN — variant effect prediction is where the two ledgers nearly agree. This is the one task with a real, standardized, clinically-labeled yardstick, so capability and validity come closest to meeting. On the protein side that yardstick is ProteinGym — 250+ deep-mutational-scanning assays, 2.7M+ mutated sequences, 200+ protein families, plus ~65,000 expert-annotated clinical mutations, evaluating 70+ models. Protein language models are competitive with or ahead of alignment-based methods on clinical missense there. On the DNA side, frontier models are state of the art on noncoding and splice variants (Evo 2, AlphaGenome), but not on coding SNVs, where specialists like AlphaMissense still lead. The calibrated posture: zero-shot variant scores are now good enough to contribute evidence under an ACMG-style framework — especially for noncoding and splice variants where classical tools are weakest — and not good enough to act alone. The maturity is real, and it is bounded.

RED — perturbation prediction: trivial baselines still win. The flagship result is Ahlmann-Eltze, Huber & Anders, Nature Methods 2025. They compared five foundation models plus two other deep networks against deliberately simple baselines for predicting transcriptome changes after gene perturbations. None beat the baselines, in any setting tested. For double perturbations, every model did worse than a simple additive baseline. For single perturbations, none beat the mean predictor or a linear model. The mechanistic finding is the one for us: pretraining on the single-cell atlas gave only a small benefit over random embeddings — only pretraining on perturbation data itself helped. (For scale of the difficulty: they found only 5,035 real genetic interactions out of ~124,000 possible at 5% FDR.) This is the crux: perturbation is interventional, but the atlases these FMs learn from are observational. Observation doesn’t teach intervention.

RED — zero-shot representations: a 2010 heuristic wins. Kedzierska et al., Genome Biology 2025 evaluated Geneformer and scGPT zero-shot (no fine-tuning — the realistic discovery setting). Both were beaten by simply selecting highly variable genes and running established methods (Harmony, scVI) for clustering. HVG selection outperformed both FMs across all metrics. They even varied scGPT’s pretraining scale — 814k, 10.3M, 33M cells — and still the only unseen dataset where scGPT beat both baselines was a single PBMC study. “One PCA still rules them all” reaches the same verdict for perturbation analysis. Lesson: strong fine-tuned benchmark numbers can mask weak general representations.

RED — attention is not a regulatory-network oracle. This one matters for your own papers, so read it carefully. A 2026 systematic study (37 analyses, 153 statistical tests, 4 cell types, 2 perturbation modalities) asked whether scGPT’s and Geneformer’s attention encodes regulatory biology — as both papers claim and many downstream studies assume. It does not. Attention captures co-expression, not unique regulatory signal. Trivial gene-level baselines scored AUROC 0.81–0.88 for predicting CRISPRi targets (variance alone reached 0.881), while attention-derived and correlation edges sat near 0.70. Pairwise attention edge scores added zero predictive value. And causal ablation of the heads claimed to carry regulatory signal produced no degradation at all. The attention does encode some layer-specific structure — it just adds nothing for the prediction task. (The same paper offers a constructive fix, “Cell-State Stratified Interpretability,” which improves gene-network recovery up to 1.85×.)

⚠️ Direct hit on our own program

TransPlatformer reports attention over toxicology genes as a sanity check (05_TRANSPLATFORMER.md §5.7), and TransTissueFormer leans on attention for interpretation. The 2026 critique says: attention weights here mostly recover co-expression, which you could get from a plain correlation matrix, and they do not demonstrate learned regulatory causation. This lines up with the “attention is not explanation” caution already in 05_TRANSPLATFORMER.md §5.7 — but now with hard numbers. Takeaway for our writing: present attention as a smell test, never as evidence of mechanism, and if we want a regulatory-network claim, benchmark it against a co-expression baseline (exactly the control we already argue for elsewhere).

The community bake-off agrees with all of the above. Arc Institute’s Virtual Cell Challenge 2025 drew 1,200+ teams; the wrap-up reported models “not yet consistently outperforming naive baselines,” with winners combining deep learning and classical statistical features.

What the scorecard does not mean

It does not mean FMs are useless. Two things are simultaneously true:

• The value / expression-prediction side is unsolved — a linear baseline is often cheaper and at least as good.
• The gene-embedding side — the co-expression structure learned across tens of millions of cells — is real and reusable.

The catch is that most published attempts throw the good part away with the bad part, because they fine-tune the whole model (value encoder included) on the wrong modality. That is precisely the opening for us (§18.5).

The one-sentence scorecard: as expression predictors, current biology FMs roughly tie linear baselines; as gene-relationship encoders, they carry genuine pretrained structure that almost nobody has isolated and used properly. Note the honest self-critique too: these skeptical results are themselves task-specific (perturbation, zero-shot clustering, attention-as-network). They do not show FMs are worthless for, say, supervised cell-type annotation after fine-tuning. Weight each critique against the claim it actually tests — which is exactly the discipline the two-ledger habit enforces.

18.5 How this maps onto our toxicogenomics projects

Now the part that matters. Our data is bulk (or pseudobulk) ${\log }_{10}$ fold-change — DrugMatrix, TG-GATEs. Our models are ToxCompl, TransPlatformer, GenTox, TransTissueFormer. Where do these FMs plug in?

The one insight that governs everything: the type error

You cannot feed fold-change into scGPT’s value encoder. Worked in full in 11_SC_FOUNDATION_MODELS.md §3, but the short version:

• scGPT’s bins start at 0 (you can’t have negative RNA counts). A fold-change of −0.71 (gene suppressed 5×) lands in a bin that doesn’t exist.
• A fold-change of +1.85 (gene up 71×, the loudest signal in the profile) bins to the same slot as “a count of 1–5 molecules,” i.e. “essentially off.” No error is raised. The most important signal is silently relabeled as its opposite.

This is a type error, not a domain gap. A domain gap (“trained on news, you have tweets”) is fixed by more training. A type error (“the function expects an int, you passed a list”) is fixed by not passing the list. No amount of fine-tuning fixes it.

This one fact explains why naïve “fine-tune scGPT on tox data” attempts underwhelm, and it tells us exactly what to do: use the part of the FM that never touches the value encoder.

What actually transfers: the gene embedding table

Every single-cell FM contains a gene embedding table $E_{\text{gene}}$ — one vector per gene, learned purely from which genes co-occur across tens of millions of cells. It never touches expression values. It encodes “what kind of gene is this, what does it co-express with, what pathway.” That is modality-independent and largely species-conserved. It is the part that transfers, and it’s the part almost nobody in the tox literature has used.

And here’s the structural gift: our own models already have a slot shaped exactly like it.

scGPT’s gene table is 60,000 × 512. TransTissueFormer’s bottleneck is 8,565 × 512. Same width. You can load one into the other, no adapter, no architecture change — and it dodges the type error completely, because these models multiply raw fold-change straight into the gene embedding; they have no value encoder to break.

The per-model fit table

18.6 The deep part — what we can actually do, ordered by effort and payoff

Your mentor asked for deep thinking on “what we can do.” Here are concrete, rankable proposals. Each one is designed so that both outcomes publish — a positive result is an FM bridge, a negative result is a documented “we tested it, here’s the evidence,” which is exactly what the critique literature (§18.4) is asking the field to produce.

Experiment 1 — Initialize the gene table from a foundation model 🟢 do this first

The move. Take scGPT’s (or UCE’s) gene embeddings and use them as the initialization of TransTissueFormer’s bottleneck $W$ (or ToxCompl’s $P$, or GenTox’s gene lookup). Then train normally.

Why it’s the first experiment. It touches 96.6% of TransTissueFormer’s parameters, which are currently random and fit from 425 examples. It requires no architecture change. And it’s the concrete answer to the papers’ own hand-wave (“we plan to explore the adaptation of these models in future work”) — the adaptation is one tensor load.

The ablation that makes it rigorous (this is the important part — don’t skip it):

If scGPT loses to DrugMatrix’s own co-expression, that’s a real result, consistent with §18.4. If UCE’s advantage concentrates in the non-ortholog genes (the P450s), that’s a mechanistic result, not a leaderboard bump. Either way you learn something publishable. Effort: days.

Experiment 2 — Build the fold-change–native pretraining corpus 🟡

The move. Stop trying to reuse a counts-native model. Build a corpus in our modality and pretrain natively, with a signed value encoder (one that has bins for negative fold-changes).

The corpus already exists in pieces:

That’s ~1.4M interventional differential signatures. This corpus does not currently exist, and nothing prevents it existing. The mechanistic bet (§18.4): FMs underperform because they pretrain on observational data — but LINCS, Tahoe, and DrugMatrix are all interventional. A fold-change-native FM trained on interventional data is the version of the field’s dream that’s actually matched to the question. Effort: weeks–months. Payoff: potentially a paper in its own right.

Experiment 3 — Be inductive on genes, not just drugs 🟡

The move. GenTox already replaced the drug lookup with a function of drug features (a GNN pretrained on 1M compounds — itself a foundation model). It left genes as a random lookup table. Apply the same lesson to genes: replace the gene lookup with a function of gene features (scGPT/UCE embeddings).

Why it’s more than a leaderboard bump. It buys four things a lookup table can never do (detailed in 06_GENTOX.md §6.7): (1) new platforms — a CodeLink probe and an Affymetrix probe for the same gene get the same features, dissolving TransPlatformer’s entire problem; (2) new species — rat→human, which a lookup cannot do at all; (3) the low-data tissues — a brain gene with 65 observations and 300 parameters is underdetermined as a lookup but fine as a shared function; (4) genes never measured. Effort: days–weeks.

Experiment 4 — Add the missing conditioning, then go zero-shot 🟡

The move. TransTissueFormer’s multi-task model collapses (PCC 0.53 → 0.23) because it’s asked to predict a target tissue it’s never told the identity of. Add a target-tissue embedding vector. Then — because TransPlatformer already demonstrated zero-shot transfer to an unseen brain on the platform axis (05_TRANSPLATFORMER.md §5.5) — attempt zero-shot to the empty tissue pairs on the tissue axis. Effort: days for the conditioning, then a real research question for the zero-shot.

Experiment 5 — Use protein / DNA sequence models as a second, independent gene basis 🔵

The move. UCE already shows that a gene’s protein sequence (via ESM2) is a usable gene feature. Push further: test ESM2/ESM3 embeddings and Evo 2 sequence-level features as alternative gene bases in the Experiment 1 ablation. This is where the DNA/protein families earn their place in a tox project — not as expression predictors, but as independent priors on what a gene is, learned from evolution rather than from co-expression. If two totally different pretraining signals (co-expression vs sequence) agree, that’s strong evidence; if they disagree, the disagreement is informative. Effort: adds a row or two to Experiment 1.

Experiment 6 — Report the metrics that would actually reveal all this 🟢

The move. None of the above is interpretable if we keep scoring with column-wise PCC, which a mean-predictor can game. Add: row-wise PCC (undefined for a mean predictor, so it exposes one), macro-F1 over the up/no-change/down categories (the all-zeros predictor can’t game it), and enrichment-consistency (do predicted and true profiles imply the same biology?). This is the highest-value non-modeling contribution available and it makes every other experiment legible. Effort: about a week.

Experiment 7 — The transcriptome ↔ histopathology bridge 🔵 the ambitious one

The move. DrugMatrix and TG-GATEs pair each expression profile with a pathologist’s histopathology score of the same tissue. Encode the slide with a pathology FM (UNI / Virchow / GigaPath), encode the profile with our gene-embedding model, and learn a map between the two spaces — predict tissue damage from expression, or retrieve the matching expression signature from a slide.

Why it’s worth raising even though it’s hard. It turns the program from “predict one expression vector from another” into “predict the outcome that regulators actually act on (organ damage) from molecular data.” That’s the difference between an ML result and a toxicology result. It’s also the honest destination of the “biological validation” ToxCompl already gestures at (04_TOXCOMPL.md §4.12) — done as a learned bridge instead of a manual check. Effort: a real project, not a week. Payoff: the highest of anything here. This is the “deep thinking on what we can do” answer in one sentence: the molecular models and the pathology models are two halves of the same tox problem, and nobody has connected them.

The ordering, in one line

Steal the gene table and ablate it properly (Exp 1 + 6) → be inductive on genes (Exp 3) → add conditioning and reach for zero-shot (Exp 4) → if the signal is there, build the fold-change-native interventional corpus (Exp 2), using sequence models as a second basis (Exp 5).

18.7 The decision map — which model for which question

Two operating principles run through the whole table, and they are the ones to actually remember:

1. Anchor every model to a baseline on your own data before you trust it. The field’s clearest 2025–2026 lesson is that baselines win more often than the marketing admits. The right first experiment makes this a habit: run a frozen open model and a trivial baseline on the same split, and report them side by side — the model only counts if it beats the baseline.
2. Prefer open, version-lockable weights for anything you need to reproduce. An API behind a vendor’s terms (AlphaGenome, for now) is a different proposition from weights you can freeze and audit. For a research program that must be replicable, that is not a footnote.

18.8 The practical guide — picking one and running it

The last few sections were about ideas. This one is about hands. The models are open and free. The hard part is matching one to your task and your GPU. This layer is drawn from a practitioner’s guide (rewire.it, 2026), and it lines up neatly with our own plan.

Step 1 — pick by task and sequence length

You don’t need to compare all hundred models. You need the one that fits your task. Here’s the short version.

The pattern: transformers win short sequences; long-context models (Hyena, Mamba, Evo) win long ones. The cutoff is around 4 kb, where plain attention’s $O(n^2)$ cost stops being affordable.

Step 2 — you will not pretrain. you will reuse.

Pretraining one of these from scratch is a big-lab expense. Look at the numbers:

So nobody starts by pretraining. You download open weights and do one of two things: pull out embeddings, or fine-tune lightly.

Step 3 — three cheap tricks that matter

1. Mean pooling. To turn per-token outputs into one vector per sequence, average all the token vectors. Don’t use the [CLS] token or max pooling. The GenBench study found mean pooling wins, consistently.
2. LoRA, not full fine-tuning. LoRA updates about 0.1% of the parameters. It fits on one GPU and matches full fine-tuning on most tasks. Full fine-tuning is rarely worth it.
3. Try zero-shot first. If you have fewer than ~1,000 labels, just take the embeddings and train a small classifier on top. Fine-tune only if that isn’t accurate enough.

When to fine-tune vs. just take embeddings

What this means for us

The practitioner’s advice and our own type-error argument point the same way — take embeddings, don’t fine-tune the whole model.

• Experiment 1 is exactly the recommended path. We pull out scGPT’s or UCE’s gene embedding table and use it. We do not fine-tune the value encoder — which is the broken part on our data. Cheapest path and safest path, same move.
• It fits on one modest GPU. The weights are open; the gene table is a few GB. This is a graduate-student experiment, not a data-center one.
• The mean-pooling tip is already our architecture. When we need one vector for a whole profile, we average gene vectors weighted by fold-change. That is exactly what TransTissueFormer’s bottleneck computes (11_SC_FOUNDATION_MODELS.md §5).

The honest limits (same guide)

• Context is tiny next to the genome. Even HyenaDNA’s 1-million-token window is ~0.03% of the human genome. Chromosome-scale modeling is still out of reach.
• Bigger isn’t always better. On some single-cell tasks, small and large models tie — CellPLM ≈ scGPT ≈ Geneformer.
• You get predictions, not explanations. Interpretability is limited.
• Benchmarks don’t transfer. Always validate on your own held-out data. This is the same “baseline-first” point from §18.4, seen from the practical side.

18.9 Summary — what to tell your mentor

1. There are six catalogued families of biology foundation model — DNA, RNA, protein, single-cell, multimodal, and pathology — over a hundred models in total, but only a handful of underlying designs. Only single-cell/perturbation read our kind of data directly; protein/DNA models matter as gene-feature providers; and pathology models are the encoder for the other half of tox data (organ damage).
2. The differences from GPT are five and specific: the token is a gene (not a word); there’s often no order, so no positional encoding; the objective is usually “fill in the blank,” not “predict the next thing”; the architecture escapes quadratic attention by different tricks; and there’s a “value” problem GPT never faces — which is exactly where these models break on our data.
3. The training taxonomy is small — a handful of self-supervised objectives — and the decisive axis turns out to be observational vs interventional data.
4. Read the field with two ledgers — capability (what a model does at scale) vs validity (what survives an independent test set with an honest baseline). GENEB (40 models, 100 tasks, no vendor incentive) showed leaderboards are unstable and architecture often beats scale. The honest 2025–2026 scorecard is task-specific: green for variant-effect prediction (protein missense via ProteinGym; noncoding/splice via Evo 2, AlphaGenome — good enough to contribute ACMG evidence, not to act alone); red for perturbation prediction (no FM beat a linear baseline — Nature Methods 2025), zero-shot clustering (highly-variable-gene selection beats scGPT/Geneformer), and attention-as-regulatory-network (attention ≈ co-expression, not causation; trivial baselines 0.81–0.88 AUROC vs attention ~0.70). As gene-relationship encoders, though, FMs carry real reusable structure almost nobody has isolated properly. That gap is our opening.
   • A direct hit on our papers: the attention-interpretability sections in TransPlatformer (§5.7) and TransTissueFormer should be softened — attention here recovers co-expression, which a plain correlation matrix gives you, and is not evidence of learned regulatory mechanism.
5. The type error (fold-change ≠ counts) is why naïve fine-tuning underwhelms — and why the right move is to take only the gene embedding table, which never touches the value encoder and happens to be exactly the shape of the random tables at the core of all four of our papers.
6. In practice it’s cheap and hands-on: you never pretrain (that’s a $1M job); you download open weights, take embeddings, and fine-tune lightly with LoRA if needed. Use mean pooling, start zero-shot, validate on your own held-out data. Our Experiment 1 is exactly this — a single-GPU job, not a data-center one (§18.8).
7. The concrete program: initialize the gene table from an FM and ablate it against DrugMatrix’s own structure (Exp 1) — report metrics that can actually see the difference (Exp 6) — go inductive on genes (Exp 3) — add conditioning and reach for zero-shot (Exp 4) — and, if warranted, build the interventional fold-change corpus that doesn’t yet exist (Exp 2), with sequence models as an independent gene basis (Exp 5). Every experiment is designed so both outcomes publish.

Sources

• Awesome-Bio-Foundation-Models catalog (apeterswu) — the six-family taxonomy and full model list this chapter’s catalog draws on
• A Bioinformatician’s Guide to Choosing Genomic Foundation Models (rewire.it, 2026) — the hardware, cost, pooling, and fine-tuning guidance in §18.8
• Genomic Foundation Models in 2026: Two Ledgers, and What Survives a Held-Out Test Set (rewire.it, 2026) — the two-ledger framing, five traps, GENEB, and the task-by-task scorecard in §18.4
• GENEB: a diagnostic benchmark of 40 genomic foundation models (Ledneva et al., arXiv:2606.04525, 2026)
• AlphaGenome: regulatory variant effect at base-pair resolution (Avsec et al., bioRxiv 2025 / Nature 2026)
• Caduceus: reverse-complement equivariant long-range DNA models (Schiff et al., ICML 2024)
• HyenaDNA: long-range genomic modeling at single-nucleotide resolution (Nguyen et al., NeurIPS 2023)
• DNABERT-2: efficient multi-species genome foundation model (Zhou et al., ICLR 2024)
• ProteinGym: standardized benchmark of protein variant-effect predictors (Notin et al., NeurIPS 2023)
• Independent ESMFold vs AlphaFold2/3 benchmark on hard targets (PMC12809598, 2026)
• scFM interpretability critique — attention captures co-expression, not regulatory signal (arXiv:2602.17532, 2026)
• Evo 2: genome modeling and design across all domains of life (bioRxiv 2025 / Nature 2026) · Arc Institute Evo
• Benchmarking DNA Foundation Models for Genomic and Genetic Tasks (bioRxiv)
• Foundation Models for Genomics — overview (Technology Networks)
• Evaluating the Utilities of Foundation Models in Single-Cell Data Analysis (Advanced Science, 2026)
• Benchmarking Transcriptomics Foundation Models for Perturbation Analysis: one PCA still rules them all
• BioLLM: standardized framework for benchmarking single-cell foundation models (Patterns, 2025)
• Deep-learning gene perturbation prediction does not yet outperform linear baselines (Nature Methods, 2025)
• Assessing the limits of zero-shot foundation models in single-cell biology — Kedzierska et al. (Genome Biology, 2025)
• Virtual Cell Challenge 2025 Wrap-Up (Arc Institute) · STATE model
• ESM3: simulating 500 million years of evolution (EvolutionaryScale, 2025)
• RiNALMo: general-purpose RNA language models (Nature Communications, 2025)
• Bridging organ transcriptomics for multi-organ toxicity (TransTox, npj Digital Medicine)
• Machine Learning-Enabled Drug-Induced Toxicity Prediction (Advanced Science, 2025)

The model catalog is enriched from the Awesome-Bio-Foundation-Models list. A linked YouTube talk was also suggested as a source; its transcript couldn’t be extracted automatically (the page is JavaScript-rendered), so its content is not reflected here — I can mine it with the Claude-in-Chrome extension if you’d like it folded in. This chapter surveys a fast-moving field; model sizes and rankings are as of mid-2026 and will shift. The strategic core — type error, gene-table transfer, interventional data, baseline-first evaluation — is architecture-level and durable. Numeric worked examples for the single-cell mechanics live in 11_SC_FOUNDATION_MODELS.md; the four in-house papers are 04–07.