Chapter 22 — Reading scGPT at Author Depth

First of four model readings (scGPT → Tahoe → BulkFormer → EVA), each written to let an ML/NLP reader understand the model the way its authors do, not the way a press release does. This chapter assumes you know transformers cold and biology barely at all; every biological idea is defined where it is used, and the recurring move is the NLP↔transcriptomics dictionary from 09_WALKTHROUGH.md. Where a number is checkpoint- or config-dependent, or the paper prints a mechanism but not a scalar, it is flagged — scGPT is not one model but a family of checkpoints, and pretending otherwise is the first mistake.

One-sentence version: scGPT is a BERT-style encoder over genes-as-tokens whose one real trick is that it does not feed expression as a number — it feeds each gene’s rank stratum within its own cell — and whose pretraining objective is to regenerate masked expression values, once from neighboring genes and once from a single cell-summary vector.

22.1 What problem scGPT solves

The data. A single-cell RNA-seq (scRNA-seq) experiment measures, for one cell, how many mRNA molecules of each gene were captured. The readout for a cell is a vector over ~20,000 human genes of non-negative integer counts, overwhelmingly zero (a typical cell has detectable counts for a few thousand genes; the rest read 0, partly because the gene is off and partly because the assay missed it — “dropout”). Stack many cells and you get a cell × gene count matrix. This is the substrate of essentially every single-cell foundation model, scGPT included. (If you have 01_BACKGROUND.md, this is the “absolute expression, observational, human, in vitro” corner of the data map.)

One training example is therefore one cell: a sparse vector of gene counts, optionally tagged with metadata (which experimental batch it came from, which sequencing modality, and — during fine-tuning — a cell-type label or a perturbation flag). There is no sequence, no time, no pairing. The “example” is a bag of (gene, expression) pairs.

Input → output. scGPT ingests, for a cell, a set of gene tokens plus a discretized version of each gene’s expression, and produces (i) a cell embedding — one $D$-dimensional vector summarizing the whole cell, read off a prepended <cls> token exactly as in BERT — and (ii) a per-gene output used either to predict masked expression values (pretraining) or, after a task head is attached, to predict a label. So it is simultaneously an encoder (cell → vector) and a conditional generator (context → missing expression values).

Downstream tasks it supports (all revisited in §22.5): cell-type annotation (classify a cell), batch integration (embed cells from many experiments so that biology, not the experiment, dominates the geometry), multi-omic integration (fuse RNA with ATAC or protein readouts), perturbation-response prediction (given a control cell and “knock out gene X,” predict the perturbed transcriptome), and gene-regulatory-network (GRN) inference (read gene–gene structure out of attention). One backbone, many heads — the foundation-model pitch.

22.2 How the input is represented

This section is where scGPT earns or loses its identity, so we go slowly, starting from a concrete cell.

A small realistic cell

Take five genes and their normalized expression in one cell (real cells have thousands of non-zero genes; five keeps the arithmetic visible):

Two problems have to be solved before a transformer can touch this. First, the numbers are not comparable across cells: a cell sequenced twice as deeply has roughly twice every count, with no biological difference. Second, the vector is mostly zeros and the non-zeros span orders of magnitude. scGPT’s representation is engineered around both.

Step 1 — gene tokenization

Each gene name maps to a fixed integer ID in a shared vocabulary. The vocabulary is built as the union of gene sets across all training studies, so “ACTB” is the same token everywhere, which is what lets a model trained on many datasets be applied to a new one. Alongside real genes sit special tokens: <cls> (prepended; its final hidden state is the cell embedding), <pad> (length-filling), and condition tokens for batch and modality (and, in perturbation mode, a perturbation flag). Note there is no meaningful order among gene tokens — unlike words in a sentence, genes have no left-to-right sequence — which will matter enormously in §22.3.

So our cell becomes the token sequence [<cls>, ACTB, CD3D, MT-CO1, GAPDH, IL7R] → IDs [<cls>, 15, 8891, 21, 44, 6012] (illustrative IDs).

Step 2 — value binning (the actual idea)

Here is the choice that defines scGPT. Expression is not fed as a float. For each cell independently, the non-zero expression values are rank-ordered and cut into $B$ quantile bins, each holding an equal fraction $1/B$ of the expressed genes. Zeros are kept as their own value (bin 0). The released models use $B=51$; we use $B=4$ non-zero bins here to keep it legible.

Rank the four non-zero genes of our cell and assign quantile bins (bin 4 = highest stratum):

The critical property: bin index encodes relative rank within the cell, not absolute amount. If the same cell were sequenced twice as deeply — every count doubled — the ranking is unchanged, so the bins are unchanged. This is a deliberate, cheap normalization against sequencing-depth and batch scale differences, and it is more robust than log-normalizing floats, which does not fully remove depth effects.

The NLP analogy, and its cost. Binning-by-rank is like replacing every word’s exact TF-IDF weight with its quartile among the words present in that document. You gain robustness to document length; you throw away magnitude. We will keep returning to what that discards.

Step 3 — the three summed embeddings

A transformer can only consume vectors of numbers. So for each gene in the cell we must build one vector that packs together everything we know about that gene. For MT-CO1 in our cell we know three facts: which gene it is (MT-CO1), how strongly it is expressed as a bin (bin 4, from Step 2), and which experiment the cell came from (say batch b3). We turn each fact into its own vector and add the three, element-wise, into a single vector $h_i$:

$$h_i={\mathrm{emb}}_g(g_i)+{\mathrm{emb}}_x(x_i)+{\mathrm{emb}}_c(c_i)$$

This is exactly BERT’s move — BERT builds each input as word + position + segment embeddings summed — with the pieces swapped for biology:

• ${\mathrm{emb}}_g(g_i)$ — the gene embedding. A learned lookup table (like a word-embedding matrix) with one row per gene in the vocabulary; you pull out MT-CO1’s row.
• ${\mathrm{emb}}_x(x_i)$ — the expression-bin embedding. A learned table with one row per bin index $x_i\in \{0,\dots ,B\}$; because MT-CO1 landed in bin 4, you pull row 4. (Ordinal-aware: nearby bins get related vectors.)
• ${\mathrm{emb}}_c(c_i)$ — the condition embedding. A learned table with one row per batch/modality (and the perturbation flag in perturbation mode); you pull the batch-b3 row.

Where the numbers come from — and how they are actually learned. This is worth getting right, because it is tempting to assume the numbers are looked up from some biological database. They are not. Each table is a block of learnable parameters — the gene table, for instance, is a matrix of shape (vocabulary size × $D$), i.e. roughly $20,000\times 512$, one row per gene. Three facts about how that matrix gets its values:

1. They start as random noise. Before training, every entry is a small random number, so MT-CO1’s row is initially meaningless.
2. They are learned by gradient descent, exactly like every other weight in the model. During pretraining (§22.4), each time the model predicts a masked gene’s value and is wrong, backpropagation nudges all the weights a tiny step to reduce that error — and the embedding rows are just more weights, updated the same way as the attention and MLP matrices. There is no separate “embedding algorithm”; the tables are trained by the ordinary training loop.
3. They self-organize into meaning. After millions of cells, the rows settle into an arrangement where genes that play similar roles end up with similar vectors (co-regulated genes cluster together), and bin 4’s row sits in a consistent relation to bin 3’s. Nobody hand-sets these geometries; they emerge because that arrangement is what lowers the loss. This is the same mechanism that produces word embeddings in NLP — “king” landing near “queen” was learned from text, not entered by hand.

So “MT-CO1’s row” is a learned parameter that, by the end of training, encodes how MT-CO1 behaves across the corpus. The individual numbers stay uninterpretable; only their relationships (which rows are close to which) carry meaning.

One nuance you’ll meet in later chapters: not every model starts fully random. Some initialize an embedding table from an external source and then keep training from there — a “warm start.” BulkFormer seeds its gene rows from a protein language model (24_READING_BULKFORMER.md §24.2), and Tahoe-x1 seeds its drug token from a chemical fingerprint (23_READING_TAHOE.md §23.3); scGPT uses the plain random-then-learned route for all three of its tables.

Below the toy vectors are made-up length-4 numbers, purely to show the summing mechanics (a real row is 512 learned numbers). Take:

$${\mathrm{emb}}_g(\text{MT-CO1})=[0.2,-0.1,0.4,0.0],{\mathrm{emb}}_x(\text{bin }4)=[0.1,0.1,-0.2,0.3],{\mathrm{emb}}_c(\text{b3})=[0.0,0.2,0.0,-0.1].$$

The addition, done one dimension at a time. “Add the vectors” means add dimension 1 to dimension 1, dimension 2 to dimension 2, and so on:

So $h_{\text{MT-CO1}}=[0.3,0.2,0.2,0.2]$ — one dense vector that now means “gene MT-CO1, expressed at a bin-4 level, in a cell from batch b3.” The transformer reads this single vector; it no longer needs the three facts separately.

The whole cell. Do this for every gene (plus <cls>), stack the resulting vectors as rows, and you get the matrix the transformer actually processes — $6\times 4$ for our toy cell, and $M\times 512$ in the real model:

            dim1  dim2  dim3  dim4
<cls>      [ ...   ...   ...   ... ]
ACTB       [ ...   ...   ...   ... ]
CD3D       [ ...   ...   ...   ... ]
MT-CO1     [ 0.3   0.2   0.2   0.2 ]   <- the row we just built
GAPDH      [ ...   ...   ...   ... ]
IL7R       [ ...   ...   ...   ... ]

What is absent from that sum: a positional embedding. In a sentence you add one because word order carries meaning (“dog bites man” ≠ “man bites dog”). Genes have no order, so scGPT drops positional embeddings entirely — the gene’s identity (its ${\mathrm{emb}}_g$ row) plays the role that position plays in text. This is why the next subsection can say the input is an unordered set.

Ordering, masking, padding, metadata — the bookkeeping

Four practical details about how a cell is packaged. Each is stated plainly, then why it matters.

Ordering — the cell is a bag of genes, not a sentence. Because there is no positional embedding (Step 3), the order in which you list the genes does not change anything: feed [ACTB, CD3D, MT-CO1] or [MT-CO1, ACTB, CD3D] and the model produces the same result. Attention lets every gene look at every other gene regardless of listing order, so the model treats the cell as an unordered set. One useful consequence: since order is irrelevant and the model simply attends over whatever genes you hand it, you can feed a subset of genes (say, only the most informative ones) without breaking the model — nothing depends on including all ~20,000.

Masking — the fill-in-the-blank game that trains the model. scGPT learns without human labels by hiding some information and asking the model to reconstruct it — like BERT’s masked-language-modeling, but the thing hidden is the expression level, not the word. Concretely: pick some genes in the cell, and for each one keep its identity but erase its bin value, replacing the value with a special “mask” marker. The row still says “this is gene CD3D”; it just no longer says how much CD3D was expressed. The model must predict that missing bin from the other, unhidden genes. Deciding which genes to hide and predict is the entire self-supervised objective (spelled out in §22.4).

Padding — making every cell the same length. Transformers process cells in fixed-size batches, so every cell must be trimmed or filled to one common length. A cell with too few genes gets <pad> filler tokens appended (they are ignored by attention and loss); a cell with too many genes is truncated to fit. The target length depends on the job — about 3,001 tokens in the annotation setup, 1,536 in the perturbation setup. The often-repeated “1,200 genes” is just one particular choice of highly-variable genes (HVGs — the genes that differ most across cells, a standard way to pick the informative ones), not a fixed rule; think of the input as “roughly 1–3k gene tokens, depending on the configuration.”

Metadata — the model sees almost nothing about the experiment. The only non-expression information that reaches the model is what you encode as a condition token: batch, modality, and (in perturbation mode) the perturbation flag. Everything else about how the data was generated — the drug, the dose, the exposure time, the tissue, the species — is invisible to the model unless you deliberately add a condition token for it. That blind spot is precisely the problem for toxicogenomics, where dose, time, and chemical are the experiment, and it is the gap we build on in §22.8.

What biology is preserved or lost

One term first, because it appears in the table: a gene is “on” (expressed) when the cell is actively transcribing it — copying its DNA into mRNA molecules — so its measured count is nonzero; it is “off” (silent) when the cell is not transcribing it, so its count is zero. In any one cell only a few thousand of the ~20,000 genes are on; the rest read zero. That pattern of which genes are on versus off — a mostly-empty vector — is called the sparsity pattern, and it is itself biologically meaningful (a liver cell and a T cell are “on” for very different gene sets). scGPT’s binning keeps which genes are on and their relative ranking, but flattens how strongly each on-gene is expressed into just 51 rank strata.

That right column is the whole argument of the book’s 11_SC_FOUNDATION_MODELS.md §3 restated in scGPT’s own terms: a model that keeps only within-cell rank has, by construction, thrown away exactly the axis (magnitude of change) that a fold-change experiment is about. Hold that thought for §22.7.

22.3 How the architecture works

We now push our cell all the way through, tracking the exact tensor shapes and doing one attention step by hand. Structurally scGPT is BERT; the parts worth your attention are the shapes, the masking scheme, and the heads.

The backbone, and the numbers that define it

The whole-human checkpoint is a 12-layer Transformer encoder: hidden dimension $D=512$, 8 attention heads (so each head works in $d_k=512/8=64$ dimensions), feed-forward width up to 2048, roughly 51–53M parameters (the CZI model card says 53M; secondary write-ups say ~51M — a minor discrepancy, not worth resolving). FlashAttention is used in pretraining, which is what makes attention over thousands of gene tokens tractable.

Input and output, with actual shapes

Let $M$ be the number of tokens in the cell (genes + the <cls> token). Our toy cell has $M=6$; a real cell has $M\approx 1,000$–$3,000$. Here is what flows through, with both the toy shape and the real shape:

Two outputs matter. The first row of the final layer, $h_{\text{cls}}$, is a single $512$-vector summarizing the whole cell — this is what you cluster, classify, or hand to a downstream task. The per-gene head produces one predicted expression per gene — this is what the training objective scores. The shape never changes through the 12 layers ($M\times 512$ in, $M\times 512$ out of each); the layers only re-mix information across genes.

One transformer layer, with a worked attention example

Each layer applies standard multi-head self-attention followed by residual + LayerNorm + a feed-forward MLP. For one head:

$$Q=H{W}_Q,K=H{W}_K,V=H{W}_V,A=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+\text{mask}\right),\text{out}=AV.$$

$Q,K,V$ each have shape $M\times d_k$; the attention matrix $A$ is $M\times M$ (every token against every token). Let us actually compute the row of $A$ for gene CD3D, using $D=2$ toy vectors and identity projections ($W_Q=W_K=W_V=I$, so $Q=K=V=H$). Take three tokens:

$$h_{\text{cls}}=[1.0,0.0],h_{\text{ACTB}}=[0.3,0.2],h_{\text{CD3D}}=[0.2,0.9].$$

Step 1 — similarity scores (dot product of CD3D’s query with each key), divided by $\sqrt{d_k}=\sqrt{2}\approx 1.41$:

$$\text{CD3D}\cdot \text{cls}=0.20,\text{CD3D}\cdot \text{ACTB}=0.24,\text{CD3D}\cdot \text{CD3D}=0.85\Rightarrow \frac{[0.20,0.24,0.85]}{1.41}=[0.14,0.17,0.60].$$

Step 2 — softmax turns those into attention weights that sum to 1:

$$A_{\text{CD3D},\cdot }=\mathrm{softmax}([0.14,0.17,0.60])\approx [0.28,0.29,0.44].$$

Step 3 — blend the value vectors with those weights:

$${\text{out}}_{\text{CD3D}}=0.28[1,0]+0.29[0.3,0.2]+0.44[0.2,0.9]\approx [0.45,0.45].$$

So CD3D’s new representation is a weighted mixture: 28% of the cell-summary token, 29% of ACTB, 44% of itself. Biological reading: the attention weight $A_{ij}$ is “how much gene $i$’s updated representation draws on gene $j$ in this cell,” so genes that co-vary — members of the same pathway or regulon — pull toward each other. This is exactly the signal scGPT later reads out as a gene network (§22.5), and exactly the signal skeptics call co-expression, not causation (§22.7). Repeat this for all $M$ rows and all 8 heads, concatenate the heads back to width 512, add the residual, LayerNorm, MLP — that is one layer, done 12 times.

The masking scheme — not causal, and that is the point

Because genes have no order, GPT-style left-to-right causal masking is meaningless (there is no “left”). scGPT instead splits the genes into known (expression observed) and unknown (to be predicted), and shapes the attention mask so each unknown gene may attend to the known genes and <cls> (and itself), while unknowns may not attend to one another. Prediction is then iterative over $K$ rounds: predict all unknowns, commit the most confident $1/K$ fraction to “known,” repeat — so it generates in confidence order instead of left-to-right order.

Picture it with our cell, taking CD3D and GAPDH as “unknown”:

known:  <cls> ACTB MT-CO1 IL7R          unknown: CD3D GAPDH
CD3D  may attend to -> <cls>, ACTB, MT-CO1, IL7R, (itself)    but NOT -> GAPDH
GAPDH may attend to -> <cls>, ACTB, MT-CO1, IL7R, (itself)    but NOT -> CD3D
round 1: predict CD3D and GAPDH; commit whichever is more confident (say CD3D) to "known"
round 2: re-predict GAPDH, now allowed to attend to the freshly committed CD3D

The prediction heads

After 12 layers we have $H^{(12)}\in {\mathbb{R}}^{M\times 512}$. Three heads read from it:

1. Expression decoder (ExprDecoder). A small MLP that turns each gene’s final vector into one predicted expression number (optionally also a probability that the gene is a true zero). It looks at gene $i$’s own contextualized vector — which already absorbed the other genes through attention. $${\hat{x}}_i={\mathrm{MLP}}_{\text{expr}}\left(H_i^{(12)}\right)\in \mathbb{R}.$$
2. Cell-conditioned decoder (MVCDecoder, the GEPC objective) — in plain terms. This head answers a deliberately harder question: can you predict gene $i$’s level using ONLY the one 512-number cell summary $h_{\text{cls}}$, without looking at gene $i$’s own row? It builds a query from the cell summary and takes a dot product with each gene’s identity embedding: $${\hat{x}}_i^{\text{cell}}=f(⟨W_q{h}_{\text{cls}},{\mathrm{emb}}_g(g_i)⟩),f=\text{small nonlinearity}.$$ Why bother? Because if a single vector can regenerate the whole transcriptome gene-by-gene, that vector is a genuinely rich summary of the cell — and it is that summary, $h_{\text{cls}}$, that every downstream task (clustering, annotation) actually consumes. So MVC is a training pressure that says “force all the cell’s information into the one summary vector.” (The name unpacks as Gene Expression Prediction for Cell modeling / Masked Value prediction from the Cell embedding — the “for cell modeling” is the whole point: it exists to make the cell vector good, not to predict genes well per se.)
3. Classification head (ClsDecoder). A 3-layer MLP on $h_{\text{cls}}$ for cell-type annotation, trained with cross-entropy (fine-tuning only). A separate adversarial batch discriminator also reads $h_{\text{cls}}$ for the DAB objective (§22.4).

End-to-end toy, with data

Let us run our real five-gene cell once, masking CD3D, and name the shapes:

INPUT (M=6)      : [<cls>, ACTB=bin3, CD3D=MASK, MT-CO1=bin4, GAPDH=bin2, IL7R=bin0], batch=b3
  |  sum 3 embeddings per token  (§22.2)
H^(0)  : 6 x 512   (toy: 6 x 4)
  |  12 transformer layers, CD3D restricted to attend to the known genes + <cls>
H^(12) : 6 x 512
  |  read outputs
h_cls   = H^(12)[0]           -> 1 x 512   (the cell embedding)
x_hat_CD3D = ExprDecoder(H^(12)[CD3D])  -> a scalar, e.g. "bin 1-2"

Concretely: CD3D is a canonical T-cell marker. The unmasked genes in this cell (and the cell summary) look like a T cell, so the model predicts CD3D at a modest positive level (bin 1–2) rather than zero. That prediction is scored against CD3D’s true hidden bin by the training loss (§22.4). The honest caveat, which the whole book keeps pressing: when the model gets CD3D right because it reasoned “T-cell context ⇒ CD3D on,” that is mechanism; when it gets it right merely because CD3D statistically co-occurs with these genes in training, that is correlation — and scGPT cannot tell you which (§22.7).

22.4 How the model is trained

The objectives

Two self-supervised losses do the pretraining, both masked-value regression (masked-position MSE), differing only in what the prediction is conditioned on:

• GEP (Gene Expression Prediction, = MLM). Mask a fraction of a cell’s gene values; predict them from the unmasked genes via the ExprDecoder. Loss: $${\mathcal{L}}_{\text{GEP}}=\frac{1}{|\mathcal{M}|}\sum _{i\in \mathcal{M}}({\hat{x}}_i-x_i{)}^2,$$ over the masked set $\mathcal{M}$. This teaches gene–gene predictability: “given these genes’ levels, what is that gene’s level?”
• GEPC / MVC (Gene Expression Prediction for Cell modeling) — the “regenerate the cell from its summary” objective. Same masked-MSE target, but the prediction is made only from the one cell-summary vector $h_{\text{cls}}$ (the inner-product head of §22.3), not from the gene’s own row. In plain terms: hide a gene, then force the model to guess it using nothing but the single 512-number summary of the whole cell. If it can do this for every gene, the summary must contain essentially all the cell’s information — and that summary is exactly what clustering and annotation use downstream. So GEP teaches gene-from-other-genes, while GEPC teaches gene-from-the-cell-vector; the second is what makes the cell embedding good.

Two regularizers/adapters appear (mostly in fine-tuning, some in pretraining variants):

• ECS (Elastic Cell Similarity). A contrastive-style regularizer on cell embeddings with a similarity threshold ($0.3$ in the code): pull similar cells together, push dissimilar apart, so the embedding space is coherent and well-separated.
• DAB (Domain Adaptation by reverse Back-propagation). A batch classifier tries to read batch label off $h_{\text{cls}}$; a gradient-reversal layer flips its gradient into the encoder, pressuring the encoder toward batch-invariant embeddings. This is scGPT’s built-in batch-correction knob.

(Supervised heads — cross-entropy for annotation, CCE contrastive — are fine-tuning-only.)

Flag: the paper prints the mechanism (mask “a proportion” of genes) but not, in the prose I can verify, the exact pretraining mask ratio; fine-tuning configs range 0–0.4 depending on task. Treat the specific percentage as unconfirmed.

The data, and what it is supposed to teach

What the corpus is. Pretraining uses >33 million human single cells pulled from CZ CELLxGENE Discover (the Census snapshot of 15 May 2023) — a public aggregator that harmonizes thousands of published scRNA-seq studies into one queryable collection. The whole-human model draws on roughly 51 tissues across ~441 studies, restricted to non-spatial scRNA-seq and (for the main model) normal, non-diseased cells. Every cell is observational: a snapshot of a cell in some tissue, with no experimental intervention applied.

Why this composition matters. Because the corpus is an aggregate of many labs, protocols, and donors, it is enormous and diverse — which is the foundation-model bet — but it is also heavily unbalanced: common, easy-to-sample cell types (blood, immune) are over-represented; rare cell types, hard tissues (brain), and non-human data are scarce or absent. A model trained by masked reconstruction on such a corpus will be best at reconstructing the abundant states and weakest on the rare ones, which is a large part of why the out-of-distribution failures in §22.7 fall where they do.

What the objective is supposed to teach. The intended lesson is a general “grammar” of human cell states: which genes tend to switch on together, what a coherent cell state looks like as a point in embedding space, and how cell types relate. If that grammar is real and transferable, then a single pretrained backbone should give any downstream task (annotation, integration, perturbation) a strong head start — the whole promise of a foundation model.

But the knowledge is partly localized, not universal. scGPT is released not as one model but as a family of checkpoints — a whole-human model plus organ-specific and pan-cancer ones (brain ~13M cells, blood/immune, heart, lung, kidney, and a cancer variant continually pretrained on tumor cells). The revealing fact: the organ-specific checkpoints frequently beat the whole-human one on in-domain tasks. If the pretraining had learned one truly universal cell grammar, a bigger, broader model would dominate; that the specialized ones win in-domain says the “knowledge” is substantially local to the tissues each model saw. Practical consequence: pick the checkpoint that matches your tissue, and do not assume “scGPT” means a single, uniform representation — a point the task-arithmetic chapter (21_TASK_ARITHMETIC.md §21.5) leans on directly.

Shortcuts and weaknesses in the objective

A “shortcut” is a way for the model to lower the training loss without learning the biology you hoped it would — the machine-learning version of a student who aces the exam by memorizing past answers. scGPT’s masked-value objective has three, each with a concrete alternative that would remove it.

Shortcut 1 — predict the average and stop. Gene expression is sparse (mostly zeros) and heavy-tailed (a few genes very high). Under mean-squared error, the single prediction that minimizes loss for a gene before looking at anything else is that gene’s average level across cells. So a model can drive the loss down a long way by learning per-gene averages and barely using the cross-gene context at all — the context is where the biology lives, and the shortcut skips it. Kedzierska et al. (2025) show the sharp symptom: on their probe, scGPT’s own reconstructions underperform a naive “predict the mean” baseline, i.e. the expensive model is not beating the trivial one it should dominate. What could be done instead: add a term the mean cannot satisfy — a contrastive / ranking loss that forces the model to tell this cell’s gene levels apart from a decoy cell’s (a mean-predictor scores identically on both, so it loses), or predict each gene relative to the cell’s own profile rather than in absolute terms. (This is improvement 6 in §22.8.)

Shortcut 2 — the target itself is coarse, so effort has nowhere to go. The thing being predicted is a rank bin (1 of 51), not a real number. Even a model that wanted to learn fine magnitude structure gets no gradient signal for it, because two cells with very different absolute expression can share a bin — the distinction was thrown away in preprocessing (§22.2). The objective’s ceiling is therefore set by the representation, not the architecture. What could be done instead: give the target back its magnitude — predict the bin and a within-bin continuous residual, or replace MSE-on-bins with a proper count likelihood (negative-binomial / zero-inflated, as scVI uses) that models the actual counts and their overdispersion. (Improvement 1 in §22.8.)

Shortcut 3 — observational data can only teach co-occurrence, never intervention. Every training cell is a passive snapshot; none is the result of someone pushing a gene and recording what moved. So even a perfectly trained model learns “gene X and gene Y tend to be on together,” never “if I knock out X, Y falls” — association, not causation. This is the deepest limitation, and it is not fixable by a better loss on the same data; it is a property of the data. It is the exact gap 18_GENOMIC_FM_LANDSCAPE.md §18.4 identifies, and the reason Ahlmann-Eltze et al. (2025, Nature Methods) find foundation models fail to beat simple linear baselines on perturbation prediction — the one task that requires causal content. What could be done instead: pretrain (or continue-pretrain) on interventional corpora — Perturb-seq, and the 100-million-cell drug-perturbation atlas of Chapter 23 (Tahoe) — with an explicit intervention token, so the objective can actually reward getting a perturbation response right. (Improvement 2 in §22.8.)

22.5 How it is used for downstream tasks

The pattern is always BERT’s: keep the pretrained backbone, attach or switch on a head, fine-tune (usually full fine-tuning — all weights move, which is what makes the task-arithmetic idea in 21_TASK_ARITHMETIC.md even well-posed). Each task below is given the same way: what it is, the concrete input, how the model is adapted, the concrete output, and a small example.

Cell-type annotation — put a label on one cell

The task. Given a cell’s expression, name its cell type (“CD8 T cell,” “B cell,” …). This is ordinary classification on top of the cell summary.

Input. One cell, as gene+bin tokens (§22.2). Example:

(CD3D and CD8A are T-cell genes and are “on”; the B-cell gene MS4A1 is “off” at bin 0 — the pattern that should give it away.)

Adapt. Attach the ClsDecoder on top of $h_{\text{cls}}$ and fine-tune with cross-entropy against labeled cells; the reconstruction objectives are switched off.

Output. A probability over the $C$ cell types, then its argmax:

cell c -> h_cls (512 numbers) -> ClsDecoder -> [B:0.01, mono:0.02, CD8T:0.93, ...] -> "CD8 T cell"

Use. Train on an annotated reference, then label an unannotated dataset automatically — the standard “annotate my new experiment” workflow.

Batch integration — make cells from different experiments comparable

The task. A batch is a set of cells processed together (one lab, day, machine, donor). Technical differences between batches leave a fingerprint on the counts that has nothing to do with biology, and that fingerprint is often stronger than the difference between two cell types. Integration produces a representation where the fingerprint is removed and the biology remains.

Input. Many cells, each carrying its usual gene/bins plus a batch token naming its experiment. Two labs, three shared cell types:

Note c1 and c4 are the same kind of cell in two labs; they should end up together.

Adapt. Fine-tune with three losses working together: reconstruction (keep the embedding biologically faithful), DAB (a batch-classifier + gradient-reversal that makes the embedding carry so little lab-identity that even a dedicated detector can’t tell Lab A from Lab B — this erases the fingerprint), and ECS (keep similar cells close so the geometry stays clean).

Output. One 512-number embedding per cell, whose arrangement is the deliverable:

BEFORE (raw): cells split by LAB          AFTER (integrated): cells split by TYPE
  [ Lab A: c1 c2 c3 ]  [ Lab B: c4 c5 c6 ]   [ T: c1 c4 ]  [ B: c2 c5 ]  [ mono: c3 c6 ]
        (technical artifact wins)                     (biology wins; labs mixed within each)

Use / scoring. Cluster or UMAP the embeddings; success is “colored by type → clean clusters” and “colored by lab → labs mixed inside each cluster.” The standard scorecard for that trade-off (biology kept vs batch removed) is scIB (§22.7–22.8).

Caveat (§22.7). Done zero-shot — pretrained embeddings with no DAB/ECS fine-tuning — this can go the wrong way and even amplify batch (Kedzierska et al.), so cells separate by lab even more than the raw data.

Multi-omic integration — fuse different measurement types

The task. The same cell can be measured in more than one way: RNA (gene expression), ATAC (which stretches of DNA are physically open/accessible, a proxy for which genes could be switched on), and surface protein (antibody counts of proteins on the cell’s outside — the “CITE-seq” assay measures RNA and surface protein in the same cell). These are different data types about one cell. Integration puts them into one shared space.

Input. Cells whose tokens can be genes (with expression bins), open-chromatin regions, or proteins (with their own level bins), each tagged by a modality token saying which measurement it is. Example: a CITE-seq cell contributes RNA gene tokens and protein tokens; an ATAC cell contributes region tokens.

Adapt. Add a modality embedding (a fourth thing summed into $h_i$, alongside gene/value/batch) and fine-tune to fuse.

Output. One shared 512-number embedding per cell, so a T cell measured by RNA-only lands next to a T cell measured by RNA+protein:

RNA-only T cell    ┐
RNA+protein T cell ┼─> same region of embedding space  (modality no longer decides position)
ATAC T cell        ┘

Use. Jointly analyze datasets collected with different technologies — e.g. transfer labels from an RNA reference onto ATAC-only cells.

Perturbation prediction — the toxicology-shaped task

The task. Predict what a cell’s transcriptome becomes after you perturb a gene — knock it out or force it on. This is the one task that requires causal content, and the one closest to toxicogenomics.

Input. A control (untreated) cell’s expression plus a perturbation flag naming the target gene(s) — the flag rides on the value channel of the targeted gene via pert_encoder. Example:

control T cell:  CD3D=bin3, IL2=bin1, MYC=bin2, ...      + perturbation flag: "knock out CD3D"

Adapt. Fine-tune to reconstruct the treated profile; masked-MSE over all genes (not just a masked subset).

Output. The predicted post-perturbation expression across all genes (shape $M\times 1$) — “if you knock out CD3D in this cell, here is the whole transcriptome afterward”:

predicted after KO of CD3D:  CD3D -> ~0 (knocked out), IL2 -> down, MYC -> ~unchanged, ...

You then compare this predicted profile to measured perturbed cells.

Benchmarks. Against GEARS (a graph model) and CPA on the Adamson (87 single-gene) and Norman (single + two-gene combinatorial) Perturb-seq datasets. This is exactly the “apply intervention → predict response” shape of toxicogenomics — and the task where the field’s negative results (18 §18.4) bite hardest: no foundation model reliably beats a simple linear baseline here.

Gene-regulatory-network (GRN) inference — read structure out of attention

The task. Produce a gene–gene graph (which genes act together in a cell state). Unlike the others, this is a read-out, not a fine-tune — no training happens.

Input. A set of cells of a given state; you run them through the frozen model and collect the attention matrices (§22.3).

Output. A graph whose edges come from aggregating attention across cells and heads (plus gene-embedding similarity): gene pairs that consistently attend to each other become edges.

attention (gene x gene), averaged over cells  ->  threshold  ->  edges
   CD3D <-> CD3E strong  => edge      MS4A1 <-> CD14 weak => no edge

Caveat. Useful for hypothesis generation, but sensitive to which layer/heads you aggregate, and — the standing critique — the edges largely recover co-expression (genes that move together), which a plain correlation matrix already gives you; it is not demonstrated causal regulation.

Toxicity relevance (why this book cares)

The perturbation task is structurally the toxicogenomics task: intervene on a system, predict the transcriptomic response. But scGPT’s native inputs are absolute-count single cells with a gene-knockout flag, whereas toxicogenomics is bulk log-fold-change under a chemical, at a dose, for a duration, in a tissue (03_TOXICOGENOMICS_RESEARCH.md). Using scGPT here forces a choice: re-encode fold-change into the binned-count world (the type-error of 11 §3), or add chemical/dose/time condition tokens the model never pretrained on. Neither is free — which is exactly what §22.8 tries to fix.

22.6 What is technically distinctive

Separating scGPT from earlier transcriptomic FMs (scBERT, Geneformer) and asking which differences actually carry weight:

1. Value binning by within-cell rank (vs Geneformer’s rank-ordering of genes with no value channel, and scBERT’s raw-value binning). scGPT keeps a per-gene value channel but makes it depth-robust. This is the genuine representational idea, and it is plausibly responsible for real robustness gains — but it is also the source of the magnitude blindness in §22.7. Distinctive and double-edged.
2. Non-sequential, confidence-ordered generation (the known/unknown iterative scheme). A real architectural adaptation of “generative pretraining” to an unordered set — more thoughtful than bolting causal masking onto genes. Whether it beats plain BERT masking on downstream metrics is not cleanly isolated in the ablations; treat as elegant, not proven-decisive.
3. Cell-conditioned reconstruction (MVC/GEPC). Forcing the <cls> vector to regenerate the transcriptome is a principled pressure toward good cell embeddings and is one of the more defensible design choices.
4. A batteries-included multi-objective kit (GEP + MVC + ECS + DAB + CLS). Convenient, and DAB/ECS are what let one backbone attempt integration; but multi-objective stacks also make it hard to attribute gains.

Genuine architecture vs. cheaper explanations. Be skeptical in the book’s usual way (18 §18.7). Much of scGPT’s headline performance is a fine-tuned result on favorable in-domain benchmarks, and a large share of “why it works” is plausibly (a) 33M cells of pretraining data and (b) the depth-robust binning preprocessing — i.e. data and preprocessing, not the transformer per se. The zero-shot evidence (next section) supports that reading: strip fine-tuning and the architecture’s advantage over PCA/HVG largely evaporates.

22.7 Limitations and research gaps

This is one of the two sections the whole chapter builds toward. I separate the weaknesses by where they live — representation, architecture, objective, evaluation — then ask the deeper question (mechanism or statistics?) and map exactly where the model breaks out-of-distribution.

Representation — magnitude is gone, and some tasks are only magnitude. Binning-by-rank (§22.2) keeps which genes are high and their order, but discards how high and destroys cross-cell comparability; 51 strata is coarse against a real dynamic range of $10^4$–$10^5$. Concrete failure: give a drug at a low dose (say a gene induced 2×) and a high dose (the same gene induced 20×). If the gene is the cell’s top-ranked gene in both, it lands in the same top bin both times — the model literally cannot see the dose difference, because it was erased before the transformer. For dose–response, fold-change, and effect-size tasks — i.e. toxicogenomics — this is a structural handicap, not something more training fixes (11_SC_FOUNDATION_MODELS.md §3).

Architecture — attention has no reason to learn regulation. Self-attention over an unordered gene set is free to encode any statistical co-variation; nothing in the architecture biases it toward causal regulatory structure. So the GRN read-out (§22.5) recovers co-expression — which genes move together — which a plain gene–gene correlation matrix already gives you, at a fraction of the cost. Calling those attention edges a “regulatory network” over-claims: co-movement is not control, and the model has no way to tell an upstream regulator from a downstream responder.

Objective — the loss can be won cheaply and teaches only association. As detailed in §22.4, masked-value MSE is beatable by predicting per-gene means (Kedzierska et al.’s probe shows scGPT’s reconstructions losing to a mean-predictor), and because every training cell is observational, the objective can only ever teach co-occurrence, not intervention — which is why perturbation is the field’s hardest case.

Evaluation — the wins are mostly fine-tuned and in-domain. The headline results come after supervised fine-tuning on benchmarks drawn from the same data distribution. The central independent check, Kedzierska et al., Genome Biology 2025 (doi:10.1186/s13059-025-03574-x), tests the model zero-shot (embeddings straight out of pretraining, no fine-tuning) and finds it beaten by:

• highly-variable-gene selection (a preprocessing step, not even a model) on clustering,
• scVI (Lopez et al., Nature Methods 2018) on cell-type structure, and
• Harmony (Korsunsky et al., Nature Methods 2019) on batch integration,

winning on only one dataset (PBMC 12k), and in some cases producing embeddings whose variance is more batch-explained than the raw data — i.e. it added batch signal. The honest reading: “foundation model” here means “a good initialization for supervised fine-tuning,” not “a strong general-purpose zero-shot representation.” That distinction should frame every claim about scGPT.

Mechanism or statistics? On the current evidence, mostly statistics — and that is not nothing (co-expression manifolds and cell-state geometry are genuinely useful for annotation and integration). But there is no demonstrated grasp of causal regulatory mechanism, and the field-wide perturbation results (Ahlmann-Eltze et al., Nature Methods 2025, doi:10.1038/s41592-025-02772-6) — no foundation model beating a simple linear baseline — are exactly what you would expect from a model that learned association rather than intervention.

Where it breaks out-of-distribution (the map that matters for toxicology).

Stack these and you get toxicogenomics — rat, bulk, chemical, dosed, a new platform — which is close to a worst case for stock scGPT on every axis at once. That is not a knock on the model; it is a precise statement of the distance between what it was built for and where the book wants to use it, and it sets up §22.8.

22.8 How the model could be improved

This is the section to take furthest. Each proposal names the gap (from §22.7), the concrete change (with enough how-to to start), the data to use, the metric and baseline that would prove it, and why it would be a contribution rather than a tweak. They are ordered from most self-contained to most ambitious. The discipline throughout (per 21_TASK_ARITHMETIC.md §21.11) is: a bounded change with a defined evaluation beats a speculative redesign.

1. Give the value channel its magnitude back

• Gap. Binning-by-rank erases absolute magnitude, so dose/fold-change/effect-size is invisible (§22.7, Representation).
• How to. Two compatible options. (a) Bin + residual: keep the 51-bin embedding but add a small head that also predicts a continuous within-bin offset, so the target is “bin 4, and 0.7 of the way up it” — restoring magnitude without throwing away the depth-robustness binning bought. (b) Count likelihood: replace MSE-on-bins with a negative-binomial (or zero-inflated NB) likelihood on the raw counts, exactly as scVI does (Lopez et al., Nature Methods 2018, doi:10.1038/s41592-018-0229-2); this models both the magnitude and the overdispersion/dropout that make counts non-Gaussian.
• Data & metric. Fit on the same CELLxGENE cells; evaluate on dose–response recovery using LINCS L1000 (multi-dose compound signatures) and Open TG-GATEs (dose × time), scoring whether the model can now rank doses of the same compound (Spearman of predicted vs true fold-change across doses). Baseline: the stock binned model, which should be at chance on dose ranking by construction.
• Why it’s a contribution. It directly tests the book’s central “type-error” claim (11 §3) and would be the first demonstration that a magnitude-aware head recovers exactly the axis toxicogenomics needs.

2. Pretrain on interventions, not just snapshots

• Gap. Observational-only pretraining teaches association, not causation, so perturbation prediction fails (§22.7, Objective; Ahlmann-Eltze et al. 2025).
• How to. Continue-pretrain scGPT on interventional corpora with an explicit intervention token (a condition embedding that says which gene/drug was applied and, ideally, at what dose), and score the model on predicting the change from control, not the absolute post-state. Use genetic-perturbation Perturb-seq (Adamson, Norman) and the drug-perturbation atlas of the next chapter, Tahoe-100M (~100M cells, ~1,100 drugs × ~50 lines; bioRxiv 2025.02.20.639398) — the largest interventional single-cell resource, and the natural cure for this exact gap.
• Data & metric. Held-out perturbations (unseen genes; unseen drug–line combinations) scored against GEARS (Roohani et al., Nature Biotechnology 2024) and CPA (Lotfollahi et al., Mol. Syst. Biol. 2023) — and, non-negotiably, against the linear baseline (18 §18.4), which is the bar the whole field currently fails to clear. A result that beats the linear model on truly unseen perturbations would be genuinely new.
• Why it’s a contribution. It converts “foundation models don’t help perturbation” from a verdict into a testable “…because they were pretrained observationally,” and Tahoe makes the counterfactual runnable.

3. A fold-change adapter so toxicogenomics data can enter at all

• Gap. scGPT ingests binned absolute counts; toxicogenomics is signed log-fold-change — a type mismatch, not a domain gap (11 §3, §22.7).
• How to. Add a small input adapter that maps a signed log-fold-change value into the model’s value-embedding space, and give the model a dual value head that distinguishes “absolute level” (native scRNA-seq) from “signed change vs control” (toxicogenomics), so a single backbone can consume both without pretending one is the other. This is a few-parameter change at the input/output, not a new architecture.
• Data & metric. DrugMatrix and Open TG-GATEs (rat + human, dose × time). Evaluate on cross-tissue direction prediction — given the fold-change profile in a source tissue, predict the direction (up/down/none) in a target tissue — against Funk-SVD / ToxCompl (the book’s own baselines, 04_TOXCOMPL.md, 07_TRANSTISSUEFORMER.md).
• Why it’s a contribution. It is the minimal bridge that lets a pretrained scFM touch toxicogenomics at all, and it makes the type-error claim falsifiable: if the adapter works, the gap was representational; if it doesn’t, the gap is deeper.

4. Tokenize genes by protein sequence for cross-species transfer

• Gap. The human-gene-ID vocabulary makes rat/mouse genes different tokens; cross-species transfer is weak by construction (§22.7, species row).
• How to. Replace (or augment) the learned gene-ID embedding table with protein-sequence embeddings: run each gene’s protein through a protein language model (ESM-2, Lin et al., Science 2023) and use that as ${\mathrm{emb}}_g$, the way UCE (Universal Cell Embeddings, Chapter 25) does. Orthologs — rat Cyp1a1 and human CYP1A1 — then start with nearly identical embeddings because their proteins are nearly identical, so knowledge transfers across species for free.
• Data & metric. rat → human direction transfer on the paired compounds in Open TG-GATEs (the same evaluation already sketched in 20_REASONING_DATASET_IDEA.md §20.6): train on rat, test on human, macro-F1 over up/down/none, stratified by ortholog conservation (expect strong transfer on conserved genes, weak on the divergent P450s).
• Why it’s a contribution. It imports the single most transferable idea from UCE/EVA into a value-carrying model like scGPT, and tests it on the species boundary that matters for replacing animal testing.

5. Batch handling that cannot backfire

• Gap. Zero-shot, scGPT can amplify batch signal (§22.7; Kedzierska).
• How to. Move batch correction from an afterthought to part of pretraining: keep DAB/ECS lightly active during pretraining (not just fine-tuning), or add an explicit batch-conditioned decoder so batch is modeled and subtracted rather than left latent in $h_{\text{cls}}$. The design target is a guarantee: the integrated embedding is never worse on batch than the raw data.
• Data & metric. The scIB benchmark (Luecken et al., Nature Methods 2022, doi:10.1038/s41592-021-01336-8): report the batch-removal vs bio-conservation trade-off head-to-head with Harmony and scVI, following the Kedzierska protocol, with the explicit success criterion “never below raw data on batch.”
• Why it’s a contribution. It turns a documented embarrassment (a foundation model losing to a 2019 method) into a measured fix with a falsifiable guarantee.

6. An objective the mean cannot game

• Gap. Masked-MSE is beatable by predicting per-gene means (§22.4, Shortcut 1).
• How to. Add a contrastive / ranking term to the loss: alongside reconstructing the masked values, require the model to score this cell’s true gene profile above a decoy (another cell’s profile, or a shuffled one). A mean-predictor assigns both the same score, so it cannot win the contrastive term — the model is forced to use cross-gene context. This is a light add-on to the existing loss, not a new model.
• Data & metric. Evaluate zero-shot (the setting where the current objective loses) on Kedzierska’s clustering + integration suite; success is beating HVG/scVI/Harmony without fine-tuning — the thing stock scGPT currently cannot do.
• Why it’s a contribution. It attacks the objective’s core shortcut directly and is measured on precisely the benchmark that exposed it.

A note on cost. Several of these (1, 3, 6) are small input/output/loss changes trainable on one GPU, especially if adapted with LoRA rather than full fine-tuning (scPEFT, Nature Machine Intelligence 2025, shows LoRA matches full fine-tuning on scGPT) — so they are graduate-student-scale experiments (18 §18.8), not data-center ones. Directions 2 and 4 are heavier (a pretraining run; a protein-embedding vocabulary) but reuse existing corpora and existing evaluation harnesses. In every case the metric and the baseline exist before the model is built — which, per 21 §21.11, is the line between a research contribution and a redesign with no way to be proven right.

This chapter is exposition, not a result. Architecture and objective details are cross-checked between the scGPT paper (Cui et al., Nature Methods 2024; bioRxiv 2023.04.30.538439v2) and the released configs/CZI model card (12 layers / 512 dim / 8 heads, $B=51$ bins, ~51–53M params, ECS threshold 0.3, $h_i={\mathrm{emb}}_g+{\mathrm{emb}}_x+{\mathrm{emb}}_c$); the pretraining mask ratio and the exact parameter count are flagged as unconfirmed. The zero-shot critique is Kedzierska et al., Genome Biology 2025; the perturbation-baseline critique is Ahlmann-Eltze et al., Nature Methods 2025 — the same findings anchored in 18_GENOMIC_FM_LANDSCAPE.md §18.4. Next: Chapter 23, Tahoe — a perturbation-first, 100-million-cell corpus that targets exactly the interventional gap identified in §22.7–22.8.