Bridging Genomics Foundation Models and Toxicogenomics

Read this as a book: mdbook serve --open → http://localhost:3000. It live-reloads on every edit. See Building the book at the bottom.

A working set of notes on an open problem: the foundation model revolution has not reached toxicogenomics, and the reasons are structural rather than incidental.

Written for ML researchers — especially NLP — with no biology background. Built around the ORNL–NIEHS cross-tissue translation program (ToxCompl+, TransPlatformer, TransTissueFormer, GenTox) and the 2026 single-cell foundation model literature.

The short version

Three facts that together define the opportunity:

1. There is no toxicogenomics foundation model. Every genomics FM is single-cell, absolute-expression, mostly-observational, human, in vitro.

2. Scale has not solved perturbation prediction. The Virtual Cell Challenge 2025 ran 1,200+ teams on 100M+ cells and concluded models are “not yet consistently outperforming naive baselines across all metrics”, with hybrids of deep learning and classical statistics winning. Souza & Mehta (2026) show parameter-free linear methods beating FMs including out-of-distribution.

3. The leading diagnosis points at what toxicogenomics has. Ahlmann-Eltze et al. (2025): FMs underperform because their pretraining data is observational. Toxicogenomics data is small, bulk, rat — and 100% interventional.

The central technical obstacle, stated precisely: scGPT’s value encoder consumes binned absolute counts; toxicogenomics data is log fold-change. That is a type error, not a domain gap. No amount of fine-tuning fixes it. You either convert the data or change the encoder — and choosing which is the research question.

Documents

Suggested order — file number = chapter number = sidebar number:

01_BACKGROUND              the biology, from zero
02_TOXICOLOGY_CONTEXT      why the field exists — regulators, 2035, ETAP
03_TOXICOGENOMICS_RESEARCH what's being researched — and the competition
   ↓
04_TOXCOMPL                matrix factorization from zero — the engine
05_TRANSPLATFORMER         across machines
06_GENTOX                  inductive MF — new drugs
07_TRANSTISSUEFORMER       across organs — the architecture
08_AUGMENTATION            how the training data is manufactured
09_WALKTHROUGH             one matrix, four axes
   ↓
10_SOTA_LANDSCAPE          the ML field in 2026
11_SC_FOUNDATION_MODELS    scGPT, UCE, Tahoe — with toy numbers
12_TRANSLATION_TRANSFER    what MT offers this problem
   ↓
13_SYNTHESIS               all four papers connected      ← the payoff
14_RESEARCH_AGENDA         defensible vs speculative
15_FRONTIER                the research program
   ↓
16_MATH_NOTES              every derivation    } reference —
17_SOURCES                 provenance          } dip in as needed
code/                      runnable demos

What’s actually established here

Things tested rather than asserted (all in code/):

• The matrix-completion-augmented translation task is exactly affine. Ridge scores $\rho =1.0000$ / MAE $0.0000$ on augmented pairs; the analytic construction with no fitting also scores $1.0000$. Scoring well on augmented data proves you can invert a low-rank linear map — not that you learned biology. (16_MATH_NOTES.md §2)

• Row-wise PCC is undefined for a mean predictor. Constant rows, no denominator. This makes it the metric that exposes the baseline everyone should be running — and it is the metric GenTox §2.3 argues for from first principles, in a paper the same program wrote. (§3.5)

• The DrugMatrix pair table decomposes exactly into 24 study panels — residual 0.0, all 24 structural zeros reproduced. Missingness is study-design block structure, which means the data are plausibly Missing Not At Random, contra the stated MAR assumption. (§6)

• A model can score ${\rho }_{\text{col}}=1.0000$ and be useless. GenTox’s degenerate adversary, verified. (§3.4)

And one claim that testing refuted: the rank of the induced linear map is ~2, not ~301. $r+1$ is an upper bound, not the rank. The corrected version is stronger — the task can be nearly rank-one — but the original form was wrong. Kept visible in 17_SOURCES.md rather than quietly fixed.

The finding that surprised me most

TransTissueFormer’s input bottleneck is a matrix $W\in {\mathbb{R}}^{8565\times 512}$. It holds 96.6% of the model’s parameters — the 32-layer transformer stack is 3.4%. And row $g$ of $W$ is, by construction, a 512-dimensional embedding of gene $g$: the bottleneck computes $W^{\top }\mathbf{x}={\sum }_g{x}_{g}W[g,:]$, an expression-weighted sum of gene embeddings. It is word2vec-style document embedding.

scGPT’s gene token embeddings are also 512-dimensional.

Same shape. scGPT’s pretrained gene embedding table can be loaded directly as the initialization of $W$ — no adaptation layer, no dimension mismatch, no architecture change. And it sidesteps the fold-change/absolute-count type error entirely, because that error lives in scGPT’s value encoder, which this architecture doesn’t have. Gene embeddings never touch expression values.

So: 96.6% of a model currently initialized randomly and trained on 425 examples, versus a public checkpoint trained on 33M cells, with matching dimensions. See 07_TRANSTISSUEFORMER.md §7.7.1.

The finding I didn’t expect to find twice

ToxCompl’s $P$ matrix is $375,000\times 300$ — a gene embedding table, randomly initialized. Each gene row is fit from ~369 observed treatments.

Fitting a 300-dim vector from 369 observations is a ratio of 1.23. Marginal. At $r=500$ — which the paper tried — it’s 0.74: underdetermined, and the fit is arbitrary in ~131 directions. That explains the paper’s own reported failure (raising $r$ made MaxAE worse, 1.71 → 3.99) with a mechanism the paper doesn’t give. Brain genes, at ~65 observations for 300 parameters, are underdetermined by 5×.

A pretrained gene embedding is a prior — and a prior is the only thing that can fix an underdetermined system. That’s a much stronger argument for foundation models than “it might help.” See 04_TOXCOMPL.md §4.10.

The thread that runs through all of it

Three papers. Three different problems. The same object at the core of each — and it’s random:

And GenTox is the sharpest case, because it says so out loud:

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

GenTox went inductive on drugs — a function of molecular features, with a GNN pretrained on 1,000,000 compounds as the basis. That is a foundation model, built by this program, in 2024. Their own ablation proved learned representations beat hand-crafted ones.

They just never applied that lesson to the other axis of the same matrix.

The most under-rated open problem

The field is in an acknowledged metric crisis. “The Metric Picks the Winner” (June 2026) shows model rankings inverting end-to-end with metric choice on drug-response prediction.

GenTox proved the underlying pathology in 2024 — Theorem 1: unrelated transcriptomic profiles correlate at $\rho \to 1$ asymptotically; their own baseline scored ${\rho }_{\text{col}}=0.992$ with ${\rho }_{\text{row}}=0.0084$. It was early, and the field has independently caught up to it. It remains an unpublished draft with two empty sections.

The proposed replacement — enrichment-consistency, scoring predictions by whether they support the same biological conclusions as the truth — is robust to the correlation pathology, is the actual downstream task, and cannot be gamed by a mean predictor (which outputs identical enrichment for every treatment, hence zero discriminative power). See 15_FRONTIER.md F6.

Caveats

• The synthetic data reproduces the real statistics (92% normal, Table 3 pair counts to mean abs error 5.4, all 24 structural zeros) but is linear-Gaussian plus spikes. Real biology is not. It is a testbed for claims about algorithms, not about biology.
• The study-panel decomposition is not unique — 36 equations, 47 candidates. It is a valid decomposition, not the study design.
• Claims from unverified background knowledge are flagged in 17_SOURCES.md. Check before citing.
• Everything downstream is conditional on the mean baseline. If it matches the reported numbers on real data, the cross-tissue translation premise needs rethinking rather than extending. That is why it comes first.

Building the book

This folder renders as an mdBook — searchable, themed, navigable, live-reloading.

One-time setup

brew install mdbook          # or: cargo install mdbook
cargo install mdbook-katex   # STRONGLY recommended — see below

Then in book.toml: uncomment the [preprocessor.katex] block at the bottom and set mathjax-support = false.

Why KaTeX matters here. These docs are heavy on $$...$$ — all of 16_MATH_NOTES.md is derivations, and the papers’ equations appear throughout. MathJax works as a fallback but is slow and flaky. KaTeX is worth the one extra install.

Read it — one command

bash tools/book.sh

That’s it. It regenerates the table of contents, kills any stale server holding the port, builds, serves, and opens your browser at http://localhost:3000. Live-reloads on every edit — leave it running.

Make it one word. Add to ~/.zshrc:

alias book='bash ~/projects/single_cell/toxicogenomics_and_single_cell_fm_learning/tools/book.sh'

Then from anywhere: book.

Other modes:

bash tools/book.sh --build       # build to ./book/, don't serve
bash tools/book.sh --port 3001   # different port

Stop it with Ctrl-C, not Ctrl-Z. Ctrl-Z suspends — the process keeps holding port 3000 and the next run fails with “Address already in use”. (book.sh kills stragglers for you, but it’s a habit worth having.)

Why book.sh and not just mdbook serve

It handles the three things that bite:

It also checks that mdbook-katex actually runs before building — otherwise mdBook dies with a cryptic Exec format error and produces nothing.

Adding a new page

1. write the .md
2. add one line to ORDER in tools/gen_summary.py
3. run bash tools/book.sh — it regenerates the TOC

If you skip step 2 the script tells you. That’s the whole safety net: ORDER is the single source of truth for the book’s structure.

Why ORDER is explicit

File number is reading order now (01_BACKGROUND → 15_SOURCES), so a filename sort would work. But ORDER also carries the part groupings (“Background”, “The Program — Four Papers”, …) and the display titles, which a sort can’t infer. It doubles as the orphan check.

Note the ml_and_llm_learning generator won’t work here — that one globs NN_*/ directories; this folder is flat.

Publish (optional)

Same GitHub Actions workflow as ml_and_llm_learning — see that repo’s HOW_TO_BUILD_BOOK.md §4. Point path: ./book, set Pages source to “GitHub Actions”.

Gotchas

• Math not rendering → KaTeX isn’t installed, or [preprocessor.katex] is still commented out, or mathjax-support is still true.
• A page is missing → it’s not in SUMMARY.md. Run python3 tools/gen_summary.py and read the orphan warning.
• The .py demos → mdBook copies non-markdown files through to book/, so links from code/README.md to demo_*.py serve the raw source. They don’t render as pages.