Single-Cell Foundation Models: What They Are, and How to Actually Use Them

Every claim here has a worked example with numbers you can check by hand. Run code/demo_scgpt_pipeline.py alongside this — it prints every step.

0. The 60-second version

The thing to take from all of them: the gene embedding table. Not the model.

The thing that will break: the value encoder. Every single time.

1. What a cell is, as data

Forget biology for a second. Here is a liver cell:

gene:    Alb    Cyp1a1   Actb    Tp53    Lcn2
count:   450       3      890      12       0

That’s it. A cell is a vector of counts. Real vectors are ~20,000 long; this is 5.

What the genes are, so the examples mean something:

The zero problem

Lcn2 = 0. Is that gene off, or did the assay just miss it?

You cannot tell. In real single-cell data, ~80% of genes read zero this way — you only capture 10–20% of a cell’s RNA. This is called dropout and it is the central pathology of scRNA-seq.

NLP analogy: every sentence has 80% of its tokens randomly deleted, and you don’t get told which positions were deleted. You have millions of sentences, each mostly holes.

Hold onto this. It comes back in §8 as an argument that toxicogenomics data is better.

2. scGPT, step by step

2.1 Gene → token

Exactly a vocabulary lookup. scGPT’s vocab is ~60,000 genes instead of ~50,000 wordpieces.

Alb      -> token id 4102
Cyp1a1   -> token id 88
Actb     -> token id 17
Tp53     -> token id 991
Lcn2     -> token id 2043

2.2 Value → bin

A count of 450 isn’t a token. So scGPT bins it. Bin edges are fit on the training data — which is counts, so everything is $\ge 0$:

bin edges:  [0, 1, 5, 20, 60, 150, 400, 1000, ∞]

Alb      count 450  -> bin 6
Cyp1a1   count   3  -> bin 1
Actb     count 890  -> bin 6
Tp53     count  12  -> bin 2
Lcn2     count   0  -> bin 0

2.3 Embed and add

Two lookup tables, both 512-dimensional:

$${\text{token}}_g=\underset{\text{which gene is this?}}{\underbrace{E_{\text{gene}}[\text{id}(g)]}}+\underset{\text{how much of it?}}{\underbrace{E_{\text{val}}[\text{bin}(x_g)]}}$$

$$E_{\text{gene}}\in {\mathbb{R}}^{60000\times 512},E_{\text{val}}\in {\mathbb{R}}^{|\text{bins}|\times 512}$$

This is literally BERT. token_embedding + position_embedding. Except “position” is replaced by “expression level” — because genes have no order. A cell is a set, not a sequence.

2.4 The rest

• 12 transformer layers, no positional encoding
• Pretraining: mask some genes’ values, predict them. Masked language modelling, with expression values instead of wordpieces
• 33M cells

Remember these two tables. They have completely different fates:

3. Where toxicogenomics breaks it — worked numerically

Here is a DrugMatrix profile. Same five genes. But these are ${\log }_{10}$ fold-change:

Alb       -0.02    no real change
Cyp1a1    +1.85    UP 71x vs untreated control   <-- the signal
Actb      +0.01    no real change
Tp53      +0.42    UP 2.6x
Lcn2      -0.71    DOWN 5.1x

Now push it through scGPT’s binner:

Alb      value  -0.02  ->  bin -1   <-- CRASHES: no such bin
Cyp1a1   value  +1.85  ->  bin  1   <-- same bin as a COUNT of 1-5
Actb     value  +0.01  ->  bin  0   <-- same bin as a COUNT of 0-1
Tp53     value  +0.42  ->  bin  0   <-- same bin as a COUNT of 0-1
Lcn2     value  -0.71  ->  bin -1   <-- CRASHES: no such bin

Two failures. The second is worse.

Failure 1 — negatives crash

$-0.71$ lands in bin $-1$. There is no bin $-1$. Bin edges start at 0 because you cannot have $-0.71$ RNA molecules. There is no embedding row for it. It errors — or silently wraps to the last row, which is worse.

Failure 2 — positives fail silently ⭐

Cyp1a1 = +1.85 means “this gene went up 71×, the loudest signal in the entire profile.”

It bins to 1. Bin 1 means “a count of 1 to 5 molecules” — i.e. “this gene is essentially off.”

No error. No warning. The most important signal in the profile has been silently relabelled as its opposite. And all 92% of DrugMatrix that sits in $(-0.3,0.3)$ collapses into bin 0 right next to it.

You would get a number out. The number would be garbage. Nothing would tell you.

Why this matters more than it sounds

This is a type error, not a domain gap.

A domain gap is “trained on news, you have tweets.” More training fixes it. A type error is “the function expects an int, you passed a list.” No amount of fine-tuning fixes a type error.

This is the concrete, technical answer to “why don’t you just fine-tune scGPT?” — and it’s much stronger than a shrug.

It also means: anyone who tried it and got mediocre results may simply have been feeding it nonsense. The failure would look like “the foundation model didn’t help,” not like “we made a units mistake.” Worth remembering when reading the critique literature.

4. What survives: the gene embeddings ⭐

The type error lives entirely in $E_{\text{val}}$. It does not touch $E_{\text{gene}}$.

$$E_{\text{gene}}[\text{Cyp1a1}]\text{answers:}\text{“what kind of gene is this? what does it co-express with? what pathway?”}$$

That question has nothing to do with how you encode expression values. It’s learned from co-occurrence statistics across 33M cells — the same way word2vec learns that king and queen are related, from co-occurrence, regardless of how you later count the words.

So $E_{\text{gene}}$ is:

• pretrained on 33M cells
• reusable
• modality-independent — doesn’t care about counts vs fold-change
• largely species-conserved — a gene’s function is mostly the same in rat and human

It is the part of scGPT that transfers. And nobody in the toxicogenomics literature has used it.

5. The connection: TransTissueFormer’s first layer is a gene table

TransTissueFormer’s first layer is a matrix $W\in {\mathbb{R}}^{8565\times 512}$. It computes:

$$\mathbf{h}=W^{\top }\mathbf{x}=\sum _g{x}_g\cdot W[g,:]$$

Written out on a toy ($n=5$ genes, $s=3$ slots so it fits on the page):

$$W=\left[\begin{array}{ccc}0.9& 0.1& 0.0\\ 0.8& 0.2& 0.1\\ 0.1& 0.9& 0.0\\ 0.0& 0.8& 0.2\\ 0.1& 0.0& 0.9\end{array}\right]\begin{array}{c}\leftarrow \text{Alb}\\ \leftarrow \text{Cyp1a1}\\ \leftarrow \text{Actb}\\ \leftarrow \text{Tp53}\\ \leftarrow \text{Lcn2}\end{array}\mathbf{x}=\left[\begin{array}{c}-0.02\\ 1.85\\ 0.01\\ 0.42\\ -0.71\end{array}\right]$$

$$\mathbf{h}=W^{\top }\mathbf{x}=[1.392,0.713,-0.370]$$

Read the rows. Row $g$ of $W$ is a vector describing gene $g$.

$$\boxed{\text{Row }g\text{ of }W\text{ IS a gene embedding.}}$$

And $\mathbf{h}={\sum }_g{x}_{g}W[g,:]$ is an expression-weighted sum of gene embeddings — word2vec document embedding, where the weights are fold-changes instead of term frequencies.

So:

$$\boxed{\text{Same shape. Load one into the other.}}$$

And it dodges the type error completely, because $W$ never touches $E_{\text{val}}$. TransTissueFormer has no value encoder — it multiplies the raw fold-change straight into the gene embedding. The broken part of scGPT is the part this architecture doesn’t have.

$W$ is 96.6% of TransTissueFormer’s parameters (07_TRANSTISSUEFORMER.md §7.6.1).

You’d never train word embeddings from scratch on 425 sentences. That is what this model does, on 96.6% of itself, and a matching pretrained table is sitting in a public checkpoint.

6. The other models, and what each one gives you

Geneformer — rank instead of value

Instead of binning the count, sort the genes by expression and feed the ranking:

counts:  Actb 890, Alb 450, Tp53 12, Cyp1a1 3, Lcn2 0
ranks:   Actb=1,   Alb=2,   Tp53=3,  Cyp1a1=4, Lcn2=5
tokens:  [Actb, Alb, Tp53, Cyp1a1, Lcn2]     <- position IS the value

Clever: normalization-free, robust to batch effects and sequencing depth. If one sample is sequenced twice as deep, all counts double — but the ranking doesn’t change.

The cost: magnitude is gone. Rank 4 vs rank 5 could be a 1.01× or a 100× difference.

For toxicogenomics: ranking a fold-change profile is almost meaningful — you’d get “which genes moved most.” But it destroys the up/down distinction unless you rank by signed value, and 92% of the profile is tied at ~0 so the ranking is mostly noise. Not a good fit.

scFoundation — skip the zeros

Only encodes the non-zero genes. An asymmetric encoder-decoder: the encoder reads ~2,000 non-zero genes, the decoder reconstructs all 20,000.

Why: if 80% of your input is uninformative zeros, don’t spend attention on them.

For toxicogenomics: ⭐ the architecture idea transfers, and it transfers better than it works in single-cell. See §8.

UCE — the species fix ⭐

The problem: scGPT and CellFM are human. DrugMatrix is rat. You’d need to map rat genes → human orthologs. That’s ~80% clean — and the missing 20% is concentrated in exactly the genes toxicology cares about. Cytochrome P450s (Cyp1a1 and its family) have expanded and diverged differently in rodents, because rats and humans eat different things and evolved against different toxins. The dictionary fails precisely where you need it.

UCE’s move: don’t use a vocabulary at all. Get a gene’s token by running its protein sequence through ESM2 (a protein language model):

scGPT:  Cyp1a1 -> look up "Cyp1a1" in a vocab -> row 88 -> 512-dim vector
                  (fails if the gene isn't in the vocab)

UCE:    Cyp1a1 -> get its protein sequence -> ESM2 -> 5120-dim vector
                  (works for ANY protein-coding gene, ANY species,
                   including species never seen in training)

Trained on 36M cells across 8 species.

NLP analogy: this is byte-level or subword tokenization for an unseen language, versus a fixed vocabulary that OOVs everything. UCE never needs the bilingual dictionary because it reads the “spelling” directly.

For toxicogenomics: ⭐ the right structural fit for rat data. And it makes a sharp prediction: if you compare scGPT-embeddings-via-orthologs against UCE-embeddings-no-mapping, UCE’s advantage should concentrate in the non-ortholog genes — the P450s. If it does, that’s a mechanistic result, not a leaderboard bump.

CellFM — more of the same, bigger

100M human cells, ~800M params, ERetNet backbone. Same value-encoding problem. Same gene table to steal.

STATE — the architecture worth noticing

Arc Institute. Trained on 167M observational + >100M perturbational cells, 70 human cell contexts. Two modules:

• SE (State Embedding) — where is this cell in state space?
• ST (State Transition) — how does it move when you push it?

Notice the decomposition. “Where am I” + “how do I move” is the same split as:

• CPA’s ${\mathbf{z}}_{\text{basal}}+{\mathbf{z}}_{\text{drug}}$
• ToxCompl’s $b_i+P_{i,:}\cdot Q_{:,j}$

Three literatures, one idea. That convergence is evidence it’s the right decomposition — and it means the toxicogenomics program has been building a linear STATE without calling it that.

7. Tahoe-100M and the pseudobulk bridge ⭐⭐

What Tahoe actually is

Not a model. A dataset. 100M single cells, ~1,100 drugs × 50 cancer cell lines, ~60,000 drug-cell combinations. Open source, on HuggingFace. Released Feb 2025 — ~50× larger than all previously public drug-perturbed single-cell data combined.

Crucially: interventional. Someone dosed the cells and measured what happened.

The problem

Wrong modality (single-cell counts) for toxicogenomics (bulk fold-change). Two mismatches at once.

The fix, worked numerically

Tahoe has DMSO controls for every cell line. DMSO is the solvent — “we ran the experiment with no drug in it.”

So: average the cells, then divide by the control average.

500 cells treated with drug X, 500 DMSO control cells.

gene       mean treated   mean control   log10 FC   meaning
--------------------------------------------------------------
Alb              455.0          449.6       0.01    no change
Cyp1a1            29.9            3.0       0.88    UP 7.6x
Actb             879.6          889.7      -0.00    no change
Tp53              39.8           12.0       0.50    UP 3.1x
Lcn2              12.2            1.0       0.81    UP 6.5x

$${\text{FC}}_g={\log }_{10}\frac{\frac{1}{N}{\sum }_{i\in \text{treated}}{c}_{ig}}{\frac{1}{M}{\sum }_{j\in \text{DMSO}}{c}_{jg}}$$

That last column is DrugMatrix’s format.

100,000,000 single cells
    → average within each (drug, cell line)      [pseudobulk]
    → divide by that cell line's DMSO control    [fold-change]
    → log10
    → ~60,000 bulk-equivalent fold-change signatures

Both mismatches solved by one operation. Single-cell → bulk (averaging). Counts → fold-change (dividing by control).

You deliberately threw away single-cell resolution. That’s the trade: lose per-cell detail, gain commensurability with every bulk toxicogenomics dataset ever collected.

And LINCS L1000 is already there

LINCS L1000 Level 5 ships z-scores — signed, centred at zero, structurally the same object as log fold-change. ~1.3M more signatures, free, already in the right modality.

Note the June 2026 paper that fine-tuned scGPT on 3M+ LINCS profiles used Level 3 (absolute expression). Nobody has used Level 5 as a native differential pretraining corpus.

The corpus that could exist

~1.4M interventional fold-change signatures. That corpus does not exist. Nothing prevents it existing.

8. The argument that toxicogenomics data is better ⭐

Both single-cell and fold-change data are ~90% zeros. The zeros mean completely different things.

Single-cell zeros are missing data. Fold-change zeros are signal.

The consequence: scFoundation’s “only encode non-zero genes” is a workaround for dropout — a hack around not knowing. In fold-change space the same trick is principled: you skip the 92% because nothing happened there, not because you’re uncertain.

So a fold-change-native foundation model gets scFoundation’s efficiency for a better reason, and gets a cleaner training signal, because its zeros are honest.

I have not seen this argued anywhere. It’s a small point. But it says the modality mismatch isn’t purely a disadvantage — on one axis, the tox data is better-conditioned.

9. How to actually use each one

Ordered by effort. All of these are concrete.

9.1 Steal the gene table 🟢 do this first

1. Download scGPT weights (public, HuggingFace)
2. Pull out E_gene  -> (~60000, 512)
3. Map rat genes -> human orthologs (biomaRt / Ensembl)
4. Look up each of DrugMatrix's 8,565 probes
5. Load the result as TransTissueFormer's W
6. Train as normal

Ablate against — this is the part that matters:

Both outcomes publish. Wins → the FM bridge. Loses to DrugMatrix’s own co-expression → consistent with Kedzierska and Souza & Mehta, and converts “we plan to explore in future work” into “we tested it; here’s the evidence.”

9.2 Build the fold-change corpus 🟡

Pseudobulk Tahoe + LINCS Level 5 + TG-GATEs + DrugMatrix → ~1.4M interventional differential signatures. Pretrain natively in fold-change space, with a signed value encoder.

Why bother — the mechanistic argument: Ahlmann-Eltze says FMs underperform because pretraining data is observational. LINCS, Tahoe, and DrugMatrix are all interventional.

9.3 Don’t 🔴

• Don’t feed fold-change to scGPT’s value encoder. §3. It fails silently.
• Don’t fine-tune scGPT end-to-end on DrugMatrix. Same reason.
• Don’t expect zero-shot scGPT embeddings to work. Kedzierska: beaten by highly-variable-gene selection, a 2010 heuristic with no learning in it.
• Don’t skip the baselines. The Virtual Cell Challenge ran 1,200 teams and reported models “not yet consistently outperforming naive baselines.”

10. Summary

1. A cell is a vector of counts. ~80% are zero and you can’t tell why.
2. scGPT is BERT over genes. token = gene_embedding + value_bin_embedding. No positional encoding, because a cell is a set.
3. Fold-change has negative values. There is no bin for those. And the positives get silently mis-binned — Cyp1a1 up 71× reads as “essentially off.” Type error, not domain gap.
4. Gene embeddings never touch values → they transfer.
5. TransTissueFormer’s $W$ (96.6% of it) is a gene embedding table, $8565\times 512$. scGPT’s is $60000\times 512$. Same shape. Load it.
6. UCE tokenizes by protein sequence → any species, no ortholog dictionary. The right fit for rat.
7. Pseudobulk + ÷ DMSO turns Tahoe-100M into DrugMatrix’s format. LINCS Level 5 is already there. ~1.4M signatures.
8. Fold-change zeros are signal; single-cell zeros are missing data. On that axis, tox data is better.

Toy examples runnable in code/demo_scgpt_pipeline.py. Bin edges, token IDs, and embedding tables in the demo are invented for illustration — the structure is real, the specific numbers are not. scGPT’s 512-dim gene embeddings and ~60k vocab are verified; other model details are flagged in 17_SOURCES.md.