Bridging Genomics Foundation Models and Toxicogenomics

Read this as a book: mdbook serve --open → http://localhost:3000. It live-reloads on every edit. See Building the book at the bottom.

A working set of notes on an open problem: the foundation model revolution has not reached toxicogenomics, and the reasons are structural rather than incidental.

Written for ML researchers — especially NLP — with no biology background. Built around the ORNL–NIEHS cross-tissue translation program (ToxCompl+, TransPlatformer, TransTissueFormer, GenTox) and the 2026 single-cell foundation model literature.

The short version

Three facts that together define the opportunity:

1. There is no toxicogenomics foundation model. Every genomics FM is single-cell, absolute-expression, mostly-observational, human, in vitro.

2. Scale has not solved perturbation prediction. The Virtual Cell Challenge 2025 ran 1,200+ teams on 100M+ cells and concluded models are “not yet consistently outperforming naive baselines across all metrics”, with hybrids of deep learning and classical statistics winning. Souza & Mehta (2026) show parameter-free linear methods beating FMs including out-of-distribution.

3. The leading diagnosis points at what toxicogenomics has. Ahlmann-Eltze et al. (2025): FMs underperform because their pretraining data is observational. Toxicogenomics data is small, bulk, rat — and 100% interventional.

The central technical obstacle, stated precisely: scGPT’s value encoder consumes binned absolute counts; toxicogenomics data is log fold-change. That is a type error, not a domain gap. No amount of fine-tuning fixes it. You either convert the data or change the encoder — and choosing which is the research question.

Documents

Suggested order — file number = chapter number = sidebar number:

01_BACKGROUND              the biology, from zero
02_TOXICOLOGY_CONTEXT      why the field exists — regulators, 2035, ETAP
03_TOXICOGENOMICS_RESEARCH what's being researched — and the competition
   ↓
04_TOXCOMPL                matrix factorization from zero — the engine
05_TRANSPLATFORMER         across machines
06_GENTOX                  inductive MF — new drugs
07_TRANSTISSUEFORMER       across organs — the architecture
08_AUGMENTATION            how the training data is manufactured
09_WALKTHROUGH             one matrix, four axes
   ↓
10_SOTA_LANDSCAPE          the ML field in 2026
11_SC_FOUNDATION_MODELS    scGPT, UCE, Tahoe — with toy numbers
12_TRANSLATION_TRANSFER    what MT offers this problem
   ↓
13_SYNTHESIS               all four papers connected      ← the payoff
14_RESEARCH_AGENDA         defensible vs speculative
15_FRONTIER                the research program
   ↓
16_MATH_NOTES              every derivation    } reference —
17_SOURCES                 provenance          } dip in as needed
code/                      runnable demos

What’s actually established here

Things tested rather than asserted (all in code/):

• The matrix-completion-augmented translation task is exactly affine. Ridge scores $\rho =1.0000$ / MAE $0.0000$ on augmented pairs; the analytic construction with no fitting also scores $1.0000$. Scoring well on augmented data proves you can invert a low-rank linear map — not that you learned biology. (16_MATH_NOTES.md §2)

• Row-wise PCC is undefined for a mean predictor. Constant rows, no denominator. This makes it the metric that exposes the baseline everyone should be running — and it is the metric GenTox §2.3 argues for from first principles, in a paper the same program wrote. (§3.5)

• The DrugMatrix pair table decomposes exactly into 24 study panels — residual 0.0, all 24 structural zeros reproduced. Missingness is study-design block structure, which means the data are plausibly Missing Not At Random, contra the stated MAR assumption. (§6)

• A model can score ${\rho }_{\text{col}}=1.0000$ and be useless. GenTox’s degenerate adversary, verified. (§3.4)

And one claim that testing refuted: the rank of the induced linear map is ~2, not ~301. $r+1$ is an upper bound, not the rank. The corrected version is stronger — the task can be nearly rank-one — but the original form was wrong. Kept visible in 17_SOURCES.md rather than quietly fixed.

The finding that surprised me most

TransTissueFormer’s input bottleneck is a matrix $W\in {\mathbb{R}}^{8565\times 512}$. It holds 96.6% of the model’s parameters — the 32-layer transformer stack is 3.4%. And row $g$ of $W$ is, by construction, a 512-dimensional embedding of gene $g$: the bottleneck computes $W^{\top }\mathbf{x}={\sum }_g{x}_{g}W[g,:]$, an expression-weighted sum of gene embeddings. It is word2vec-style document embedding.

scGPT’s gene token embeddings are also 512-dimensional.

Same shape. scGPT’s pretrained gene embedding table can be loaded directly as the initialization of $W$ — no adaptation layer, no dimension mismatch, no architecture change. And it sidesteps the fold-change/absolute-count type error entirely, because that error lives in scGPT’s value encoder, which this architecture doesn’t have. Gene embeddings never touch expression values.

So: 96.6% of a model currently initialized randomly and trained on 425 examples, versus a public checkpoint trained on 33M cells, with matching dimensions. See 07_TRANSTISSUEFORMER.md §7.7.1.

The finding I didn’t expect to find twice

ToxCompl’s $P$ matrix is $375,000\times 300$ — a gene embedding table, randomly initialized. Each gene row is fit from ~369 observed treatments.

Fitting a 300-dim vector from 369 observations is a ratio of 1.23. Marginal. At $r=500$ — which the paper tried — it’s 0.74: underdetermined, and the fit is arbitrary in ~131 directions. That explains the paper’s own reported failure (raising $r$ made MaxAE worse, 1.71 → 3.99) with a mechanism the paper doesn’t give. Brain genes, at ~65 observations for 300 parameters, are underdetermined by 5×.

A pretrained gene embedding is a prior — and a prior is the only thing that can fix an underdetermined system. That’s a much stronger argument for foundation models than “it might help.” See 04_TOXCOMPL.md §4.10.

The thread that runs through all of it

Three papers. Three different problems. The same object at the core of each — and it’s random:

And GenTox is the sharpest case, because it says so out loud:

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

GenTox went inductive on drugs — a function of molecular features, with a GNN pretrained on 1,000,000 compounds as the basis. That is a foundation model, built by this program, in 2024. Their own ablation proved learned representations beat hand-crafted ones.

They just never applied that lesson to the other axis of the same matrix.

The most under-rated open problem

The field is in an acknowledged metric crisis. “The Metric Picks the Winner” (June 2026) shows model rankings inverting end-to-end with metric choice on drug-response prediction.

GenTox proved the underlying pathology in 2024 — Theorem 1: unrelated transcriptomic profiles correlate at $\rho \to 1$ asymptotically; their own baseline scored ${\rho }_{\text{col}}=0.992$ with ${\rho }_{\text{row}}=0.0084$. It was early, and the field has independently caught up to it. It remains an unpublished draft with two empty sections.

The proposed replacement — enrichment-consistency, scoring predictions by whether they support the same biological conclusions as the truth — is robust to the correlation pathology, is the actual downstream task, and cannot be gamed by a mean predictor (which outputs identical enrichment for every treatment, hence zero discriminative power). See 15_FRONTIER.md F6.

Caveats

• The synthetic data reproduces the real statistics (92% normal, Table 3 pair counts to mean abs error 5.4, all 24 structural zeros) but is linear-Gaussian plus spikes. Real biology is not. It is a testbed for claims about algorithms, not about biology.
• The study-panel decomposition is not unique — 36 equations, 47 candidates. It is a valid decomposition, not the study design.
• Claims from unverified background knowledge are flagged in 17_SOURCES.md. Check before citing.
• Everything downstream is conditional on the mean baseline. If it matches the reported numbers on real data, the cross-tissue translation premise needs rethinking rather than extending. That is why it comes first.

Building the book

This folder renders as an mdBook — searchable, themed, navigable, live-reloading.

One-time setup

brew install mdbook          # or: cargo install mdbook
cargo install mdbook-katex   # STRONGLY recommended — see below

Then in book.toml: uncomment the [preprocessor.katex] block at the bottom and set mathjax-support = false.

Why KaTeX matters here. These docs are heavy on $$...$$ — all of 16_MATH_NOTES.md is derivations, and the papers’ equations appear throughout. MathJax works as a fallback but is slow and flaky. KaTeX is worth the one extra install.

Read it — one command

bash tools/book.sh

That’s it. It regenerates the table of contents, kills any stale server holding the port, builds, serves, and opens your browser at http://localhost:3000. Live-reloads on every edit — leave it running.

Make it one word. Add to ~/.zshrc:

alias book='bash ~/projects/single_cell/toxicogenomics_and_single_cell_fm_learning/tools/book.sh'

Then from anywhere: book.

Other modes:

bash tools/book.sh --build       # build to ./book/, don't serve
bash tools/book.sh --port 3001   # different port

Stop it with Ctrl-C, not Ctrl-Z. Ctrl-Z suspends — the process keeps holding port 3000 and the next run fails with “Address already in use”. (book.sh kills stragglers for you, but it’s a habit worth having.)

Why book.sh and not just mdbook serve

It handles the three things that bite:

It also checks that mdbook-katex actually runs before building — otherwise mdBook dies with a cryptic Exec format error and produces nothing.

Adding a new page

1. write the .md
2. add one line to ORDER in tools/gen_summary.py
3. run bash tools/book.sh — it regenerates the TOC

If you skip step 2 the script tells you. That’s the whole safety net: ORDER is the single source of truth for the book’s structure.

Why ORDER is explicit

File number is reading order now (01_BACKGROUND → 15_SOURCES), so a filename sort would work. But ORDER also carries the part groupings (“Background”, “The Program — Four Papers”, …) and the display titles, which a sort can’t infer. It doubles as the orphan check.

Note the ml_and_llm_learning generator won’t work here — that one globs NN_*/ directories; this folder is flat.

Publish (optional)

Same GitHub Actions workflow as ml_and_llm_learning — see that repo’s HOW_TO_BUILD_BOOK.md §4. Point path: ./book, set Pages source to “GitHub Actions”.

Gotchas

• Math not rendering → KaTeX isn’t installed, or [preprocessor.katex] is still commented out, or mathjax-support is still true.
• A page is missing → it’s not in SUMMARY.md. Run python3 tools/gen_summary.py and read the orphan warning.
• The .py demos → mdBook copies non-markdown files through to book/, so links from code/README.md to demo_*.py serve the raw source. They don’t render as pages.

Background — everything you need, from zero

Audience: ML researchers — especially NLP — with no biology background. Promise: by the end you can read any paper in this field without getting stuck on vocabulary. Method: build the biology from first principles, anchoring each concept to something an ML reader already knows.

Read this before 10_SOTA_LANDSCAPE.md.

Part 1: What a gene actually is

1.1 The central dogma

$$\text{DNA}\stackrel{\text{transcription}}{\to }\text{RNA}\stackrel{\text{translation}}{\to }\text{protein}$$

• DNA — the stored program. A string over $\{A,C,G,T\}$. ~3 billion characters in a human, ~2.7 billion in a rat. Identical in every cell of the organism.
• Gene — a substring of DNA that codes for something. ~20,000 protein-coding genes. (Plus tens of thousands of non-coding ones that do regulatory things. Ignore those for now.)
• RNA — a working copy of one gene. Disposable. Made on demand.
• Protein — the thing that actually does work. Enzymes, structure, signalling.

The thing that confuses everyone at first: if every cell has identical DNA, why is a liver cell different from a brain cell?

Because they transcribe different subsets of it, at different rates. Same program, different runtime behaviour. A liver cell runs the metabolism modules hard and leaves the neurotransmitter modules idle. Brain does the reverse.

NLP framing: DNA is the codebase. Cell type is which functions actually get called, and how often. Gene expression is the profiler output.

1.2 Gene expression

Expression of gene $g$ = how much RNA for $g$ exists right now. It’s a proxy for “how hard is the cell running module $g$.”

Measure it for all ~20,000 genes at once and you get a transcriptomic profile:

$$\mathbf{x}\in {\mathbb{R}}^{20000},x_g=\text{abundance of RNA for gene }g$$

NLP framing: a dense embedding of the cell’s current state. But — and this is the important difference — every dimension is named and interpretable. Dimension 4,412 is Cyp1a1, and biologists have opinions about it. When they say “the model got Atf3 wrong,” they mean coordinate 8,301 specifically.

This is why interpretability pressure in this field is so much higher than in NLP. Nobody asks what dimension 412 of a BERT embedding means. Here it’s the entire point.

1.3 Why “expression” is a slippery word

Three different things get called expression:

These are not interchangeable, and conflating them is the single most common way to misread this literature. See Part 4.

Part 2: How you measure it

2.1 Microarrays (the old way)

A glass chip with millions of short DNA probes stuck to it in known positions. Each probe is complementary to a bit of one gene.

1. Extract RNA from tissue, convert to cDNA, tag it with fluorescent dye
2. Wash it over the chip
3. cDNA sticks (“hybridizes”) to matching probes
4. Shine a laser, photograph the chip
5. Brightness of spot $g$ ∝ abundance of gene $g$

Analog. Continuous fluorescence intensity. Properties that matter:

• saturation — very bright spots max out; you can’t distinguish 100× from 1000×
• background — the chip glows a bit even with nothing bound
• cross-hybridization — similar sequences stick to the wrong probe
• fixed vocabulary — the chip only has probes for genes it was designed with

In DrugMatrix: CodeLink (8,565 probes, discontinued 1st-gen) and Affymetrix GeneChip Rat 230 2.0 (31,042 probes, 2nd-gen, still used).

NLP framing: a closed-vocabulary bag-of-words counter with lossy analog readout and OOV problems. You can only count words that were in the dictionary when the chip was printed.

2.2 RNA-seq (the new way)

Don’t hybridize. Just read the RNA, letter by letter, using a sequencer. Then count how many reads map to each gene.

Digital. Read counts. Properties:

• huge dynamic range (no saturation)
• open vocabulary — you find genes you weren’t looking for
• Poisson-ish counting noise at low abundance
• depth-dependent — sequence deeper, get more counts for everything

2.3 Targeted sequencing: S1500+ / TempO-Seq

Sequencing everything is expensive. So: pick ~2,700 well-chosen landmark genes, measure only those, then predict the rest.

This works because gene expression is massively redundant — genes move in coordinated modules, so a few hundred well-chosen ones carry most of the information.

In DrugMatrix: BioSpyder S1500+ measures the landmarks; a tool called GeniE extrapolates to ~20,000 genes → “BioSpyderWT” (22,794 probes).

⚠️ File this away. A chunk of “measured” BioSpyderWT data is itself a model prediction. Nobody in the four papers dwells on this. When ToxCompl+ imputes DSMatrix, it is partly imputing from imputations. That’s a soft spot, and an open question — quantifying how much error the extrapolation injects is F2 in 15_FRONTIER.md.

NLP framing: landmark genes are like measuring a document’s embedding from 300 anchor words and predicting the rest. Same trick LINCS L1000 uses (978 landmarks) — which is why L1000 and S1500+ are architecturally cousins. Remember that; it matters in 15_FRONTIER.md.

2.4 The comparability problem

You cannot put a fluorescence intensity and a read count on the same axis. They aren’t the same kind of number.

This is what TransPlatformer exists to solve, and it’s why fold-change matters so much.

Part 3: Single-cell vs bulk

3.1 Bulk

Grind up a whole liver. Extract all the RNA. Measure. You get one vector per sample.

But a liver contains hepatocytes, immune cells, endothelial cells, bile duct cells… So your vector is a weighted average over cell types:

$${\mathbf{x}}_{\text{bulk}}=\sum _k{\pi }_k{\mathbf{x}}_k,\sum _k{\pi }_k=1$$

where ${\pi }_k$ is the proportion of cell type $k$.

The consequence, and it’s ugly: if a drug kills 10% of one cell type, $\mathbf{\pi }$ changes and ${\mathbf{x}}_{\text{bulk}}$ shifts — even if no surviving cell changed its expression at all. You cannot distinguish “cells changed behaviour” from “the population changed composition.”

NLP framing: you’re reading the mean embedding of a document collection. If the topic mixture shifts, the mean moves, and you can’t tell that from every document changing.

3.2 Single-cell (scRNA-seq)

Physically separate the cells first (droplets, wells, barcodes), then sequence each one. Instead of one vector per liver you get 10,000 vectors, one per cell.

The cost is brutal. Each cell contains very little RNA. You capture maybe 10–20% of it. So per cell you detect ~1,000–5,000 genes out of 20,000. The rest are zero — and you cannot tell “not expressed” from “we missed it.”

That’s dropout, and it’s the defining pathology of single-cell data.

NLP framing: every sentence is 70–90% masked, at random, and you don’t get the mask positions. You have millions of sentences but each is mostly holes.

3.3 The trade

Why this matters for you: the entire foundation model literature is single-cell, because that’s where the millions of samples are. the DrugMatrix data is bulk. That is not a small mismatch — it’s one of the two central gaps you’d be bridging. (The other is fold-change vs absolute; Part 4.)

Part 4: Fold change — the concept that decides everything

4.1 The definition

$${\text{FC}}_g=\frac{\text{expression of }g\text{ in treated animals}}{\text{expression of }g\text{ in control animals}},\text{value stored}={\log }_{10}{\text{FC}}_g$$

Log because effects are multiplicative and you want symmetry: doubling is $+0.301$, halving is $-0.301$.

4.2 Why they do it

Reason 1 — it kills the baseline. Raw liver expression is dominated by “this is a liver.” Every profile looks the same. The drug effect is a small perturbation on a huge constant. Dividing by control subtracts the constant and leaves the signal.

Reason 2 — it makes platforms comparable. Fluorescence and read counts are different units. But a ratio is unitless:

$$\frac{{\text{intensity}}_{\text{treated}}}{{\text{intensity}}_{\text{control}}}\text{and}\frac{{\text{counts}}_{\text{treated}}}{{\text{counts}}_{\text{control}}}$$

are both “how much did this gene move,” and the platform-specific scale factor cancels — to first order. It doesn’t fully cancel, because saturation and background are nonlinear. That residual is exactly what TransPlatformer models.

4.3 The three consequences you must hold onto

(a) 92% of the matrix is ≈ 0. Most drugs don’t touch most genes. From TransTissue Table 1:

The ~8% that isn’t zero is the entire biological content. This is class imbalance wearing a regression costume. Keep asking: what does the all-zeros predictor score?

(b) It changes what correlation means. GenTox proves both directions (see 16_MATH_NOTES.md §3):

• On absolute data, two unrelated profiles share the baseline ⟹ $\rho \to 1$. High PCC is meaningless.
• On fold-change, the baseline is gone ⟹ $\rho \to 0$ under noise. High PCC is hard-won.

(c) It breaks foundation models. ⭐ This is the single most important fact in this document.

scGPT’s input is binned absolute expression. Its value encoder takes a count, bins it, embeds the bin. DrugMatrix is already-differenced log ratios.

There is no sensible way to feed a fold-change into a value encoder trained on counts. $-0.7$ is not a count. It isn’t in any bin. The model has never seen a negative number in that slot.

This is the honest, technical, concrete answer to “why don’t you just fine-tune scGPT?” It’s not a shrug. It’s an architectural incompatibility, and naming it precisely is what turns a weak future-work paragraph into a real argument.

NLP framing: it’s like trying to feed word deltas to a model whose embedding layer expects word IDs. The type signature is wrong.

4.4 Levels of processing (a table you’ll want later)

Note L1000 Level 5. Z-scores are signed and centred at zero — structurally like fold-change. That’s not a coincidence, and it’s why L1000 is the natural bridge. See 15_FRONTIER.md F1.

Part 5: Biological structure you need to know exists

5.1 Pathways

Genes don’t act alone. They form pathways — chains where gene A’s protein activates gene B, which regulates C.

Example: a toxicant damages DNA → p53 (encoded by TP53) activates → p53 turns on ~100 downstream genes → cell either repairs itself or dies.

So a “p53 signature” is a coordinated pattern across ~100 genes. You don’t look at one gene; you look for the fingerprint.

This is why every one of these papers insists genes are “not a sequence.” There’s no linear order — there’s a graph. TransPlatformer §2.2 makes this argument to reject Seq2Seq. They’re right that there’s no order; whether that rules out attention is a separate question (it argues for no positional encoding, i.e. a set transformer — which is close to what they built).

5.2 Enrichment analysis — how biologists check your work

Given a predicted profile:

1. Take the top 100 up-regulated genes
2. Ask: are they enriched for any known gene set? (Hypergeometric test.)
3. If “PPARα activation” comes back at $p<0.05$, your prediction is consistent with PPARα activation.

Enrichr is the standard tool. This is how TransTissue §6 validated its predicted kidney profiles — and it’s worth more than any PCC, because a toxicologist looked at the output and said yes, that’s what lead poisoning looks like.

NLP framing: it’s like checking that your generated text has the right topic distribution rather than just low perplexity. A downstream, semantic, human-meaningful check.

5.3 Gene regulatory networks

Build a graph: nodes = genes, edges = co-expression across conditions. Find hubs (master regulators) and modules (functional units).

Crucially: the edges are correlations between rows of $G$ — gene $i_1$ across all treatments vs gene $i_2$ across all treatments.

This is why GenTox §2.3’s row-wise argument is load-bearing. A model perfect column-wise and garbage row-wise gives you profiles that score beautifully and are useless for the analysis biologists actually run.

5.4 Toxicology vocabulary

Genes that keep appearing, and why:

TransTissue’s validation used exactly these: gemfibrozil → PPARα ✓, cisplatin → TP53 ✓, lead → p53 + oxidative stress ✓.

5.5 Why cross-tissue translation is a coherent idea

Dose a rat. The compound enters the bloodstream and reaches every organ. Liver, kidney, heart, brain all see it. Each responds — differently, but to the same systemic event.

So there’s genuinely shared structure:

$${\mathbf{x}}_c^{\tau }=\underset{\text{shared across }\tau }{\underbrace{\text{common systemic response}}}+\underset{\text{depends on }\tau }{\underbrace{\text{organ-specific program}}}$$

If you can measure liver (easy, always done) and infer kidney (harder), you save an animal and a lot of money.

⚠️ And here’s the catch that the whole research agenda turns on. That decomposition has two terms. The common term is the same shape for every drug — only its magnitude changes with severity. A model that learns only the common term looks like it’s translating and has learned nothing drug-specific.

The mean-predictor baseline measures exactly this. That’s why 14_RESEARCH_AGENDA.md A0.1 is first.

Whether translation is possible is open, and TransTissue says so — §5, verbatim: “it is possible that there are simply no (or sufficient) signals for cross-tissue translation.” That honesty is a feature.

Part 6: Cross-species — orthology

Rat and human genes descend from a common ancestor ~90M years ago. Corresponding genes are orthologs.

• ~80% of rat protein-coding genes have a clean 1:1 human ortholog (approximate — verify before citing)
• The rest: many-to-many (gene families expanded differently), or no ortholog at all

The specific problem for toxicology: cytochrome P450s have expanded and diverged differently in rodents. Rats have P450s humans don’t, with different substrate specificities. So the genes that matter most for drug metabolism are the ones where orthology is worst. That’s not bad luck; it’s because those genes are under strong species-specific selection (different diets, different toxins).

NLP framing: orthology is a bilingual dictionary. ~80% coverage, and the missing 20% is concentrated in exactly the domain-specific terminology you need.

And this is why UCE is interesting. It tokenizes genes via ESM2 protein embeddings — a gene’s token comes from its protein sequence, not a vocabulary lookup. So it embeds any protein-coding gene from any species, zero-shot, no dictionary. That’s byte-level/subword tokenization for an unseen language versus a fixed vocab that OOVs everything.

Related, and directly relevant: people have already done rat→human translation of drug-induced expression with deep nets — PLOS One 2020 used a CNN and a bottleneck DNN to translate rat→human primary hepatocytes, explicitly “circumventing the current reliance on orthologs”, and beat classical ML. A 2023 follow-up added transfer learning for rat in vitro → human in vivo. Neither is cited in the four papers, and both are the cross-species analogue of TransTissueFormer. Worth reading and worth raising.

Part 7: The datasets

DrugMatrix’s unique selling point: eight tissues, in vivo. TG-GATEs has two. LINCS and Tahoe are cell lines in dishes — no organs, no systemic exposure, no inter-organ communication.

That is the moat. Cross-tissue in vivo translation cannot be studied on LINCS or Tahoe at all. Whatever else is true, the program has data nobody else has for this specific question.

Part 8: ML concepts specific to this field

8.1 “Foundation model” here

Same pitch as BERT: pretrain self-supervised on lots of unlabelled cells, fine-tune on your small labelled task.

The cell-as-sentence metaphor drives all of it:

The load-bearing weirdness: there is no word order. A cell is a set of (gene, value) pairs. So every one of these models is a set transformer with no positional encoding, and the real design problem is how to encode the value. That’s the main axis of variation between models — see 10_SOTA_LANDSCAPE.md.

8.2 Batch effects

Same biological sample, two labs, two days, two kits → measurably different numbers. Batch effects are often larger than the biological signal you’re chasing.

Standard fixes: ComBat, Harmony, quantile normalization, scVI.

This matters more than you’d think: several 2026 benchmark papers find that FM embeddings encode batch as strongly as biology, and that plain HVG selection beats them at integration. A model can look great and be reading the sequencer’s serial number.

8.3 Observational vs interventional ⭐

This is the deepest idea in the whole field, and it’s the one to actually internalize.

• Observational data: you watched cells sit there. CELLxGENE, most atlases. Tells you what states exist.
• Interventional data: you did something and measured the response. LINCS, Tahoe, DrugMatrix. Tells you what happens when you push.

Ahlmann-Eltze et al.’s explanation for why foundation models underperform on perturbation:

pretraining data is observational. You cannot learn what happens when you push a system by only watching it sit still.

This is Pearl’s ladder of causation, and it’s the single best argument for why scaling observational cell atlases might not deliver perturbation prediction — and why DrugMatrix, which is 100% interventional, is more valuable per sample than its size suggests.

8.4 Matrix completion / collaborative filtering

Netflix Prize. Assume the matrix is low-rank, factor $M\approx PQ$, fit on observed entries only.

• Transductive: can only fill in cells whose row and column were seen. Vanilla Funk-SVD.
• Inductive: can handle a new row/column by using features. GenTox’s Row NN / Col NN.

MAR vs MNAR:

• Missing At Random — missingness independent of the values. Recovery theory assumes this.
• Missing Not At Random — missingness depends on the values. Everything breaks.

TransTissue §5 explicitly assumes MAR. The panel analysis in 16_MATH_NOTES.md §6 suggests that’s false — Table 3 decomposes exactly into study panels, with structural zeros (BR–LI = 0) that no random model produces. This is an open, testable, previously-unstated concern.

8.5 The metrics

Report all five. Always. GenTox §2 argues this from first principles; nobody, including GenTox, consistently does it.

Part 9: The dictionary

Keep this open.

Part 10: Where the two worlds don’t meet

The four gaps between the DrugMatrix data and the FM literature. Everything in 15_FRONTIER.md is an attack on one of these.

Read the last two rows again. On the axes that Ahlmann-Eltze identifies as why FMs fail, DrugMatrix has what the FMs lack. It is small, but it is in vivo and it is interventional.

That asymmetry is the thesis. It’s developed in 15_FRONTIER.md.

Next: 10_SOTA_LANDSCAPE.md — what the field actually looks like in 2026, and why the news is stranger than you’d expect.

The Field: What Toxicogenomics Is Actually For

Who pays for this, what they need, and what would count as a contribution.

Everything else in this book is about methods. This chapter is about why anyone cares. Read it before deciding what to work on — because “improve PCC by 5%” is not a contribution to this field, and knowing why is the difference between a paper that lands and one that doesn’t.

All 2025–2026 claims sourced; links inline and in 17_SOURCES.md.

2.1 The problem, in one number

~350,000 chemicals are on the global market. Most are “data-poor” — no clear information on their mechanisms or health effects.

Not “under-studied.” Data-poor. Nobody knows what they do.

And the traditional way to find out:

Do the arithmetic. At 8 years per chemical, 350,000 chemicals is not a backlog — it’s a mathematical impossibility. The field is not slowly catching up. It is permanently, structurally behind, and falling further behind every year as new chemicals enter commerce.

This is the problem toxicogenomics exists to solve. Not “understand biology better” — clear an impossible queue.

2.2 The deadline that changes everything

This is no longer aspirational. There are dates.

EPA

The EPA’s goal: eliminate ALL mammalian animal testing by 2035.

On June 2, 2026, EPA announced two major actions to replace animal-based testing with alternatives for chemical assessments under TSCA (Toxic Substances Control Act) and FIFRA (pesticides). It also introduced a streamlined process to nominate NAMs for use in pesticide and chemical assessments. (source)

FDA

April 2025: a roadmap for a stepwise transition from animal testing to validated New Approach Methodologies — beginning with monoclonal antibodies, expanding to other biologics, then new chemical entities.

March 2026: draft guidance, “General Considerations for the Use of New Approach Methodologies in Drug Development” — a validation framework and regulatory expectations for replacing animal toxicology studies. (FDA)

What “NAMs” means

New Approach Methodologies — FDA’s definition spans complex in vitro, 2D in vitro, in chemico, and in silico studies.

Read that last one again. In silico is a NAM.

A model that predicts a kidney profile from a liver profile is a candidate NAM. That is not a metaphor or a stretch — it is the regulatory category this work falls into, and there is now a formal process to nominate things into it.

The gap this creates

Animal testing is being phased out on a deadline. The replacements must exist. They must be validated. They must be accepted by regulators.

That’s the demand side. This is a field with a legal mandate and a shortage of supply. That is an unusual and favourable place to be doing methods research.

2.3 What regulators actually want (and it isn’t a gene profile)

This is the single most important thing for an ML person to understand, and it’s where most ML papers in this space miss.

A regulator does not want your predicted 8,565-dimensional vector. They want one number:

The Point of Departure (POD)

The dose below which nothing bad happens.

Everything downstream — safety factors, reference doses, exposure limits, whether a chemical can be sold — derives from that number.

Traditionally the POD is an apical POD: dose animals at several levels for months or years, look for actual damage (liver necrosis, tumours, death), find the dose where damage starts. This is what takes 8 years.

NLP framing: you spent a year building a beautiful sequence-to-sequence model, and the customer wants a scalar. Everything you optimize should be judged by whether it improves that scalar’s accuracy or the confidence around it. PCC on the vector is at best a proxy, and 16_MATH_NOTES.md §3 shows it’s a bad proxy.

2.4 The transcriptomic POD — why this field exists at all

Here is the finding the entire enterprise rests on:

Apply benchmark-dose analysis to gene expression from a SHORT-term exposure, and you get a POD that is concordant with the apical POD from a LONG-term study — including chronic toxicity and cancer.

Read that carefully. Days of exposure predict years of outcome.

Why it works, mechanistically: damage doesn’t appear from nowhere. Before a tumour, before necrosis, the cell’s transcriptional program shifts — stress response, repair, proliferation. The transcriptome is the early warning. Gene expression moves first; pathology follows.

NLP framing: it’s a leading indicator. You don’t wait for the user to churn; you read the signal that precedes churn.

This is the whole bet of toxicogenomics, and it has held up well enough that regulators are building products on it.

2.5 ETAP — the actual regulatory product

Not a proposal. A thing EPA is building.

EPA Transcriptomic Assessment Product (ETAP): a human health assessment for chemicals lacking traditional toxicity data, using a standardized short-term in vivo study + standardized transcriptomic analysis to derive a reference value. (EPA)

Reviewed by EPA’s Board of Scientific Counselors in July 2023. Real assessments exist — e.g. for Perfluoro-3-Methoxypropanoic Acid.

This is the destination. A standardized short in-vivo study → transcriptomics → a number a regulator can act on, in months rather than a decade.

Everything in this book is upstream of that pipe.

2.6 Where DrugMatrix fits — and who is actually behind these papers

This is worth knowing because it explains the program’s real objective.

Scott S. Auerbach, Ph.D., DABT leads the Toxicoinformatics Group in the Predictive Toxicology Branch, Division of Translational Toxicology (DTT), NIEHS. He:

• oversees the NTP DrugMatrix resource — the dataset all four papers are built on
• built BMDExpress 2 and 3, the standard tools for genomic dose-response analysis
• co-authored S1500+, the landmark gene set (01_BACKGROUND.md §2.3)
• is corresponding author on both TransPlatformer and TransTissueFormer

(NIEHS profile)

So this is not an academic ML program that happens to use tox data.

It is a regulatory toxicology program that is using ML. The people who own DrugMatrix, who built the dose-response tooling, and who are driving transcriptomic PODs toward regulatory acceptance are the authors.

That should change what you consider a contribution. A method that improves MAE but produces profiles a toxicologist can’t act on is worth less than an honest negative result that de-risks a regulatory submission.

2.7 The keystone result — and the uncomfortable question it raises ⭐⭐

Johnson, Auerbach & Costa (2020), Toxicological Sciences 176(1):86 — “A Rat Liver Transcriptomic Point of Departure Predicts a Prospective Liver or Non-liver Apical Point of Departure”

The setup: 79 molecules from Open TG-GATEs. Derive a liver transcriptomic POD from short-term exposure. Compare it to the systemic apical POD.

The finding:

A liver tPOD predicted the systemic apical POD within 10× — even when that apical POD came from a non-liver endpoint.

Following subacute (29-day) dosing, liver BEPOD and systemic apical POD agreed within 10×.

Note the author list. Scott Auerbach — corresponding author on TransTissueFormer.

The uncomfortable question ⭐

If the liver transcriptome already predicts non-liver toxicity within 10×, what does cross-tissue translation add?

This is the question a reviewer will ask, and it deserves a real answer rather than a dodge. As far as I can tell there are two, and they’re different in kind:

Answer 1 — different products. The tPOD is a dose: “below 5 mg/kg, nothing bad happens anywhere.” Protective, conservative, and deliberately silent on mechanism. Cross-tissue translation produces a profile: “at 50 mg/kg the kidney shows p53 activation and oxidative stress.” You cannot get the second from the first. POD answers how much; the profile answers what and where.

Answer 2 — the 10× is doing a lot of work. Ten-fold is a wide band. For a protective screening threshold that’s fine — err low, nobody gets hurt. For deciding which organ to look at or what the mechanism is, it’s useless.

And the honest caveat: if what you need is a protective threshold, the 2020 result already delivers it, cheaply, from liver alone — and cross-tissue translation is solving a problem that is already solved. The contribution has to be on the mechanism side, or it isn’t a contribution.

This is the framing question for the whole program, and I’d want it answered before committing a year. It’s also exactly the kind of thing the corresponding author can answer in five minutes, since he wrote the 2020 paper.

2.8 The fifth paper you should know about

There is a ToxCompl successor that isn’t in your four PDFs:

Nguyen & Cong (2025), BioKDD’25 — “Completion of the DrugMatrix Toxicogenomics Database using 3-Dimensional Tensors” — ToxiTenCompl.

The idea, and it’s a good one: DrugMatrix isn’t naturally a matrix. “each probe measurement is made with a compound, a dose, a duration, for a gene within a tissue” — that’s a 3D or 4D tensor. The 2D view is a flattening that throws away structure.

ToxCompl (2D):     rows = (platform, tissue, gene)   x   cols = treatments
ToxiTenCompl (3D): tissue  x  treatment  x  gene

⭐ Why this matters for everything in Chapters 2–5

The tensor formulation partly fixes a problem I spent three chapters complaining about.

In ToxCompl’s 2D layout, Cyp1a1-in-liver and Cyp1a1-in-kidney are different rows with independent parameters. The model has no idea they’re the same gene.

In the 3D tensor, gene is its own mode. The gene factor is shared across tissues by construction. That’s structurally the right move, and it’s the same instinct as “make the gene side inductive” (06_GENTOX.md §6.7) — arrived at from a different direction.

It also changes my obs/param analysis (04_TOXCOMPL.md §4.6). CP rank-$R$ on a $T\times C\times G$ tensor has $R(T+C+G)$ parameters, not $r(n_{\text{rows}}+m)$ where $n_{\text{rows}}=T\times G$. The tensor is far more parameter-efficient — which is likely why it wins, and it’s a better explanation than the paper gives.

Read this paper. It is the current state of the program’s completion work, and it partly pre-empts the criticism in Chapter 4.

2.9 What the field actually struggles with

The honest list, from the literature. These are the open problems — pick from here, not from the ML leaderboard.

Two terms you’ll hit constantly

AOP (Adverse Outcome Pathway) — a structured causal chain: molecular initiating event → key events → adverse outcome. E.g. “binds receptor → activates transcription → cell proliferation → tumour.” It’s how toxicologists formalize mechanism. Regulators like the idea and barely use it. That gap is a research opportunity.

Read-across — “this new chemical looks like that known chemical, so assume similar toxicity.” It’s the default for data-poor chemicals because there’s nothing better. It’s structure-based guessing with expert judgment on top.

⭐ Notice that GenTox’s compound GNN is a learned read-across. Pretrain on 1M molecules, embed, predict response. That is exactly the read-across problem, done with representation learning instead of expert judgment — and their own ablation shows learned beats hand-crafted (06_GENTOX.md §6.3).

That framing would land much better with a tox audience than “inductive matrix factorization” does. Same method, different sentence, completely different reception.

2.10 What counts as a contribution — and to whom

You are choosing between three audiences. They want different things, publish in different places, and reward different work.

Audience A — the ML community (NeurIPS, ICML, ICLR, KDD)

Audience B — the toxicology community (Toxicological Sciences, Arch Toxicol, SOT)

Audience C — the regulatory/NAM community (EPA ORD, NIEHS DTT, ICCVAM)

The strategic read

The program’s authors sit in B and C. DrugMatrix’s owner is the corresponding author. The destination is ETAP-shaped.

You arrive with A’s toolkit. That’s the arbitrage — and it cuts both ways. Your NMT knowledge is genuinely rare here. But if you optimize for A’s metrics you’ll produce something B and C cannot use, and the people who’d have to champion it are the ones you’d be failing.

2.11 The contributions that would actually matter

Re-derived from the field’s priorities rather than the ML leaderboard. Ordered by how much a toxicologist would care.

1. Enrichment-consistency as the metric ⭐⭐ — the best fit between your skills and their needs

15_FRONTIER.md F6 argued for this on ML grounds. The field argument is stronger.

Toxicologists do not evaluate profiles by correlation. They run the profile through enrichment analysis, get a mechanism, and act on the mechanism. That is the actual downstream task. TransTissueFormer did it by hand for 3 of 44 compounds and never automated it.

“Our model recovers the correct mechanism of action for 31 of 44 compounds” is a sentence a regulator can act on. “PCC 0.793” is a sentence a regulator will ignore.

And it’s the field’s own live problem: “The Metric Picks the Winner” (June 2026) shows rankings invert with metric choice, and GenTox proved the pathology in 2024. You’d be closing a loop the program itself opened.

2. Predict the tPOD, not the profile ⭐⭐ — the reframing

Everything in this book optimizes the wrong target.

The regulatory product is a dose. So evaluate cross-tissue translation on the question that matters:

Does a predicted kidney profile yield the same kidney tPOD as the measured one?

That’s a single number, comparable to apical data, directly ETAP-shaped, and BMDExpress already computes it — built by the corresponding author.

If predicted profiles give tPODs within 2–3× of measured ones, that is a NAM candidate, and there is a formal nomination process. If they don’t, the model isn’t fit for the purpose it exists for, and that’s worth knowing in a week rather than a year.

Nobody has done this. It’s the natural bridge between the ML and the regulation, it uses existing tooling, and it’s the experiment I’d want run before any of 15_FRONTIER.md.

3. Frame the compound GNN as learned read-across 🟢 — free

Same method, a sentence a tox audience recognizes. Read-across is the current regulatory crutch for data-poor chemicals. GenTox already beats hand-crafted descriptors with it. That’s a Tox Sciences paper written in the framing, not the method.

4. Cross-species — the field’s own named gap 🟡

“Limited cross-species applicability” is listed as a key gap. The tensor paper names it as the goal. UCE’s ESM2 tokenization is the principled route (11_SC_FOUNDATION_MODELS.md §6). And PLOS One 2020 already did rat→human hepatocytes with a bottleneck DNN — “circumventing the current reliance on orthologs” — and isn’t cited in any of the four papers.

Rat data is only a proxy for humans. Everything in this program is one translation short of the thing anyone actually wants.

5. Uncertainty 🟡 — what regulators need and models don’t give

A regulator cannot act on a point estimate with no error bar. Every model here outputs bare numbers. A calibrated interval on a predicted tPOD is worth more than a better mean, because it’s the difference between “usable in an assessment” and “interesting.”

This also connects to the MaxAE/sign-flip failure (04_TOXCOMPL.md §4.7): a model that’s confidently wrong about direction is worse than one that says it doesn’t know.

6. The baselines 🟢 — still first

Unchanged from 14_RESEARCH_AGENDA.md. Everything above is conditional on the mean predictor not matching 0.793. The field’s own flagship competition ran 1,200 teams and found models not consistently beating naive baselines. This isn’t pedantry — it’s the field’s central methodological problem.

2.12 What to read, in order

The field context (none of this is ML — read it anyway)

1. ETAP — About — the destination, 3 pages
2. Johnson, Auerbach & Costa (2020), Tox Sci 176(1):86 — liver tPOD predicts non-liver apical. The keystone. Read §2.7’s question while you do.
3. FDA NAMs page — the regulatory frame, and note in silico is a NAM
4. Bioinformatic workflows for deriving tPODs (Tox Sci 2024) — current status, data gaps, research priorities. A ready-made list of open problems.
5. An AOP primer — PMC5805086, concise, for toxicologists

The program’s own work you don’t have

6. ToxiTenCompl (BioKDD’25) — the tensor successor to ToxCompl. §2.8.
7. ToxCompl on bioRxiv — the published version
8. BMDExpress 2/3 — the tool that turns your profile into the number regulators want. If you do contribution #2, this is the tool.

2.13 The summary

THE PROBLEM ~350,000 chemicals, most data-poor. Traditional assessment: 8+ years each. That queue can never clear.

THE DEADLINE EPA: eliminate all mammalian animal testing by 2035, with actions announced June 2026 and a formal NAM nomination process. FDA: NAM guidance March 2026. In silico methods are NAMs. A cross-tissue model is a candidate NAM — a regulatory category, not a metaphor.

WHAT REGULATORS WANT Not your vector. One number — the Point of Departure, the dose below which nothing bad happens.

WHY THE FIELD EXISTS Benchmark-dose analysis on short-term gene expression gives a POD concordant (often within 3×) with chronic apical PODs. Days predict years. The transcriptome moves before the pathology.

THE DESTINATION ETAP — short in-vivo study → transcriptomics → a regulatory reference value in months instead of 8 years, for data-poor chemicals. Real, reviewed, deployed.

WHO’S BEHIND THIS Scott Auerbach — leads Toxicoinformatics at NIEHS DTT, oversees DrugMatrix, built BMDExpress, corresponding author on two of your four papers. This is a regulatory toxicology program using ML, not an ML program using tox data.

THE UNCOMFORTABLE QUESTION His own 2020 paper shows a liver tPOD already predicts non-liver apical PODs within 10×. So what does cross-tissue translation add? Best answer: the POD is a dose, the profile is a mechanism — different products. But if a protective threshold is all you need, that’s already solved from liver alone. The contribution has to be on the mechanism side. Ask before committing a year.

THE FIFTH PAPER ToxiTenCompl (BioKDD’25) — 3D tensor completion, beats ToxCompl and CP, non-negative version gives interpretable tissue factors, explicitly aims at rat→human. It partly fixes the shared-gene problem Chapters 2–5 complain about, because gene becomes its own tensor mode.

WHAT WOULD ACTUALLY COUNT

1. Enrichment-consistency as the metric — the real downstream task, and the mean predictor can’t game it
2. Predict the tPOD, not the profile — the regulatory product is a dose; BMDExpress already computes it; nobody has tried
3. Reframe the compound GNN as learned read-across — free, and it’s the field’s current crutch
4. Cross-species — the field’s own named gap; rat is only a proxy for human
5. Uncertainty — regulators can’t act on a bare point estimate
6. The baselines — still first, still gating everything

Regulatory and field claims verified by search July 2026; links inline. The §2.7 question and the §2.8 parameter-efficiency reading are mine. Audience analysis in §2.10 is judgment, not fact — weigh it against what you learn from the people actually in the room.

The Toxicogenomics Research Landscape

What people are actually working on, who they are, and where the four papers sit among them.

Chapter 2 was why the field exists — the regulatory pull. This is what the field does — the research threads, the datasets, the groups, and the competition.

The headline finding, and you need to know it before writing anything: there is a method from a different federal agency, published a year earlier in a Nature-portfolio journal, doing the exact same task as TransTissueFormer. It’s cited — in one clause, generically. See §3.5.

All 2024–2026 claims sourced.

3.1 The map

Toxicogenomics has four research arms. They’re funded separately, publish separately, and mostly don’t talk to each other. Knowing which one you’re in determines who reviews you.

Arm 1 is the smallest and the one with a regulatory product (ETAP, Ch 2 §2.5). It’s also where the data is scarcest — which is why matrix completion, augmentation, and translation are the methods that dominate. The methods in this book are a response to arm 1’s data poverty.

⭐ The interesting structural fact: GenTox is the only one of the four that touches arm 4 (chemical structure, via the compound GNN). That makes it the bridge between the two arms — and, as Ch 2 §2.9 notes, a learned read-across. That framing is worth more than “inductive matrix factorization” to this audience.

3.2 The datasets — the real infrastructure

Toxicogenomics is dataset-driven to an unusual degree. Knowing which is which tells you what a paper can and cannot claim.

Two things to notice:

DrugMatrix is unique in one respect and one only — eight organs, in vivo, same animal. TG-GATEs has two. LINCS/DRUG-Seq/sci-Plex are cells in dishes: no organs, no systemic exposure, no inter-organ communication. That is the moat (15_FRONTIER.md), and it’s real.

ArrayTrack matters because it’s what TransTox trains on — so TransTox and TransTissueFormer solve the same problem on different data, which makes them complementary evidence rather than a head-to-head. That’s a useful thing to be able to say.

3.3 Thread 1 — transcriptomic points of departure

Covered in Ch 2 §2.4–16.5. The dominant thread in arm 1, and the one with a regulatory product.

Active research questions, from Tox Sci 2024, “Bioinformatic workflows for deriving tPODs” — this paper is a ready-made list of open problems:

• which BMD method? distribution-based vs gene-set-based
• how do you filter which genes respond?
• how few animals / how short an exposure can you get away with?
• how reproducible is the whole workflow across labs?

Standard tool: BMDExpress 2/3 — built by Scott Auerbach, who is corresponding author on two of your four papers.

This is the thread your work would feed. Ch 2 §2.11’s contribution #2 — predict the tPOD, not the profile — lives here.

3.4 Thread 2 — organ toxicity prediction (DILI and friends)

DILI — drug-induced liver injury — is the field’s canonical prediction task. It’s the single biggest cause of drug withdrawal, so industry cares enormously.

The pattern is always the same:

transcriptomic profile  →  classifier  →  "is this hepatotoxic?"

Representative work:

• Open TG-GATEs + FAERS (FDA Adverse Event Reporting System) + deep nets → predict adverse drug reactions. A nice trick: link molecular profiles to real-world human outcomes.
• Att-RethinkNet — multilabel, memory + attention, correlating hepatic and renal pathology from TG-GATEs
• ToxAssay (Bioinformatics 2025) — R package over TG-GATEs and DrugMatrix
• ML for DILI to assist medicinal chemistry (J Med Chem)

Why this matters to you: TransPlatformer’s best result (F1 0.636 → 0.718, 05_TRANSPLATFORMER.md §5.8) is exactly this task. It’s the one number in the four papers that a tox audience will immediately understand and value — because it’s their benchmark, not an ML one.

3.5 Thread 3 — multi-organ, and the competitor ⭐⭐⭐

This is the most important section in this chapter.

TransTox

Li, Chen & Tong (2024), npj Digital Medicine 7(1):310 — “Bridging organ transcriptomics for advancing multiple organ toxicity assessment with a generative AI approach”. FDA National Center for Toxicological Research.

From the abstract:

“TransTox, an innovative AI model using a Generative Adversarial Network (GAN) method to facilitate bidirectional translation of transcriptomic profiles between the liver and kidney under drug treatment.”

That is TransTissueFormer’s task. Liver ↔ kidney. Drug treatment. Transcriptomic profiles.

How TransTissueFormer cites it

Reference [26]. One clause, in the Introduction:

“Beyond predictive performance, generative AI models have been proposed to produce synthetic transcriptomic profiles that preserve biological signals, facilitating downstream applications such as toxicity assessment and biomarker discovery [26].”

That describes it as generic “generative AI.” It does not say: this is a method that already does bidirectional liver–kidney translation. There’s no comparison, no discussion of how the approaches differ, no positioning.

A reviewer will find this. It is the first thing a reviewer will find.

Not because it’s hidden — it’s in npj Digital Medicine, from Weida Tong, who is one of the most visible people in computational toxicology, at the FDA.

And there’s a second tension

TransTissue §5 states:

“The very limited amount of common measurements between tissues and the vast drug space in DrugMatrix prevent the application of generative models for creating new profiles.”

TransTox is a generative model creating new profiles, for liver↔kidney, and it works.

The claim may still be defensible — on DrugMatrix specifically. DrugMatrix has 425 LI–KI pairs; ArrayTrack may have far more, and TG-GATEs-derived liver/kidney data is denser. So “generative models don’t work at our sparsity” could be exactly right.

But it has to be argued, not asserted — especially with [26] sitting in the bibliography.

What this means for you — and it’s not bad news

The strongest move: stop treating TransTox as a citation and start treating it as the baseline. “TransTox (GAN, 2 organs, ArrayTrack) vs TransTissueFormer (transformer, 8 organs, DrugMatrix) — here’s what each does better, and here’s the first head-to-head” is a better paper than either alone.

And note TransTox reports mechanism concordance, not PCC. They already evaluate the way Ch 2 §2.11 argues you should. That’s not a coincidence — it’s what an FDA audience demands.

3.6 Thread 4 — the in vitro arm (ToxCast / Tox21 / IVIVE)

Not transcriptomics, but it’s where the federal money is, and you’ll be asked about it.

Tox21 — a collaboration across four US federal agencies (EPA, NIH/NCATS, NIEHS/NTP, FDA). ToxCast is EPA’s in vitro screening programme: a large chemical library × assay panels covering signalling pathways and cellular processes.

The scale is the point: ~10,000 chemicals, run at throughput impossible for animals.

The problem is IVIVE — in vitro to in vivo extrapolation. A dish is not an animal. So:

A concentration that does something in a well → what dose in a human?

Current state: “predictive adult IVIVE models have produced fold differences less than 10 between the modelled human Administered Equivalent Dose and the measured in vivo adjusted points of departure.”

Within 10×. Same ballpark as the liver-tPOD→apical result (Ch 2 §2.7).

⭐ Notice the pattern. Every bridge in this field lands at “within 10×”:

• in vitro → in vivo: ~10×
• liver tPOD → systemic apical POD: ~10×
• tPOD → apical POD: ~3×

10× is what this field accepts as useful for a protective threshold. If your cross-tissue model can’t beat 10× on something that matters, it isn’t competitive with methods that already exist. That’s a much more honest bar than PCC, and it’s the one the field actually uses.

3.7 Thread 5 — AOPs, and why they’re stalled

Adverse Outcome Pathway: molecular initiating event → key events → adverse outcome.

The status is the interesting part:

• “the implementation of omics-based approaches in AOPs and their acceptance by the risk assessment community is still a challenge”
• “regulatory agencies show limited formal adoption of AOPs despite their potential”
• “there is a disconnect between AOP research and regulatory implementation”

Twenty years of effort, universal agreement it’s the right idea, and it’s barely used.

Why? An AOP is hand-curated. Someone reads the literature and writes down a causal chain. It doesn’t scale to 350,000 chemicals.

The gap: AOPs are the field’s lingua franca for mechanism, and there’s no automated way to go from data to AOP. A model that outputs “which key events fired” rather than “here’s a vector” would speak the field’s language. That’s adjacent to Ch 2 §2.11’s enrichment-consistency idea, and possibly the more ambitious version of it.

3.8 What the ML-in-tox literature says about itself

There’s a substantial self-critical literature. Read it before you write anything, because reviewers have.

Archives of Toxicology 2026 — “A review of machine learning in toxicology: current practices and reporting gaps” and companions list six recurring failures:

#6 is worth dwelling on. The field is explicitly complaining about unshared code and data. DrugMatrix is public; ToxCompl’s completed matrix is published; ToxiTenCompl is on arXiv. But no model code.

Releasing runnable code would be a contribution the field has explicitly asked for, costs nothing intellectually, and is the kind of thing that gets a method adopted rather than admired.

Also emerging: “Potential Role of Agentic Artificial Intelligence in Toxicologic Pathology” (arXiv 2026) — the LLM-agent wave is arriving. Note it, don’t chase it.

3.9 Who’s who

You will be reviewed by these people. It’s worth knowing the map.

Read the author lists on your four PDFs again. ORNL + NIEHS + Sciome. That’s a national-lab / regulatory-agency / contractor triangle, not a university group. It tells you the work is aimed at deployment inside a federal assessment pipeline — which is exactly what Ch 2 §2.10 says about audiences B and C.

3.10 Where the field is going in 2026

That last row is the thesis of this whole book. The FM wave has hit single-cell biology hard (10_SOTA_LANDSCAPE.md) and has not reached toxicogenomics. There is no toxicogenomics foundation model. Given a 2035 deadline and 350,000 data-poor chemicals, that’s a gap with a clock on it.

3.11 What this changes about your plan

Three concrete updates to 13_SYNTHESIS.md–15_FRONTIER.md:

1. TransTox is a baseline, not a citation. ⭐ The most valuable single experiment just became: run TransTox’s approach on DrugMatrix, or TransTissueFormer on ArrayTrack, and compare. GAN vs transformer, 2 organs vs 8, and the first head-to-head on the task. Nobody has done this, and the two agencies clearly aren’t talking.

2. “Within 10×” is the bar, not PCC. IVIVE lands at 10×. Liver-tPOD→apical lands at 10×. tPOD→apical at 3×. That’s the currency. Evaluate cross-tissue translation on whether it beats 10× on a decision someone makes — which routes straight back to Ch 2 §2.11’s predict the tPOD idea.

3. Release the code. The field’s own review literature names unshared code as a top-six failure. None of the four papers ship any. It’s free, and it’s what makes a method get used.

And one thing that got easier: TransTox de-risks the premise. You are no longer arguing that cross-organ translation is worth doing — the FDA published that argument in a Nature journal in 2024. You’re arguing about how, on more organs, with better evaluation. That’s a much easier paper to write.

3.12 Summary

THE FOUR ARMS In vivo transcriptomics (where your papers are, smallest, has the regulatory product) · high-throughput in vitro (ToxCast/Tox21, where the money is) · AOPs (the mechanism language, stalled) · chemical structure (QSAR/read-across, oldest). GenTox’s compound GNN is the only bridge between arms — and it’s a learned read-across.

THE DATASETS DrugMatrix is unique for eight organs, in vivo, same animal. Everything else is two organs, or cells in dishes. That’s the moat.

THE COMPETITOR ⭐ TransTox (Li, Chen & Tong, FDA NCTR, npj Digital Medicine, Nov 2024): a GAN for bidirectional liver↔kidney translation, validated across labs, evaluated on mechanism concordance. TransTissueFormer cites it as generic “generative AI” and never compares. A reviewer will find this immediately. It’s also the best available baseline, and it validates the premise.

THE TENSION TransTissue §5 says sparsity “prevent[s] the application of generative models.” TransTox is a generative model doing exactly this. Possibly true on DrugMatrix specifically — but it must be argued.

THE BAR Everything in this field lands at within 10× — IVIVE, liver-tPOD→apical. tPOD→apical at 3×. That’s the currency, not correlation.

THE SELF-CRITIQUE Six named failures. Three apply here: overfitting (425 samples, 4.4M params), low-quality data (92% noise), and unshared code — which none of the four papers release.

THE GAP WITH A CLOCK ON IT Foundation models are essentially absent from toxicology. EPA wants all mammalian animal testing gone by 2035. 350,000 data-poor chemicals. That is the opening.

TransTox details are from its abstract, author list, and the citation context in TransTissue (1).pdf — I have not read its full body. Do that before building on §3.5. Group/personnel mapping is from public profiles and paper author lists. §3.6’s “everything lands at 10×” observation is mine.

Chapter 4 — ToxCompl and ToxCompl+, and Matrix Factorization From Zero

This is the engine under the whole program. It assumes nothing. Run code/demo_matrix_factorization.py next to this chapter — it prints every number below.

Read this before Chapters 1–3. ToxCompl is the thing TransTissueFormer’s augmentation runs on. It’s what GenTox generalizes. And it’s where a foundation model plugs in most naturally.

⚠️ Two papers, one lineage. Get the order right.

This confused me at first, so here it is plainly:

		what it adds
1. ToxCompl	bioRxiv 2024, Auerbach co-author	the original: matrix completion, plus hybrid sampling and Bayesian optimization, plus biological validation. Deployed publicly by NIEHS.
2. ToxCompl+	the sequel	built on top of ToxCompl. Adds side-information and attention (§4.8–4.9).

ToxCompl came first. ToxCompl+ is the follow-up. §4.8–4.9 are the sequel’s additions. §4.12 is what the original does — hybrid sampling, a proper rare-signal metric, a real baseline, and biological validation.

The matrix-factorization core (§4.1–4.6) is shared by both, and it’s the part that lasts. Read §4.12 before repeating any criticism — two critiques in this chapter don’t hold against the original paper.

4.1 The magic trick

Four people, four movies, ratings 1–10:

Look at Bob’s row: [4, 1, 4.5, 0.5]. Now Alice’s: [8, 2, 9, 1].

Bob is exactly Alice divided by 2. Every single entry.

Carol is Alice times 1.5. Dave is Alice divided by 4. Exactly, every entry.

So every row is just a multiple of Alice’s row. That is what “rank 1” means. Nothing more.

Write it compactly. Give each person one number, and each movie one number:

$$\mathbf{p}=[1,0.5,1.5,0.25]\text{one number per person}$$ $$\mathbf{q}=[8,2,9,1]\text{one number per movie}$$ $$M_{ij}=p_i\cdot q_j$$

Check: Carol’s RoboCop rating is $p_3\cdot q_3=1.5\times 9=13.5$. ✓

Now count the numbers:

Eight numbers describe sixteen. So you don’t need to see all sixteen to know them.

4.2 So you can fill in holes

Say Bob rated only one movie:

Bob rated Alien 4. Alice rated Alien 8. So Bob’s scale is $4/8=0.5$.

That fixes his whole row: $0.5\times$ Alice’s row = [4, 1, 4.5, 0.5].

predicted: [4.  1.  4.5 0.5]
truth:     [4.  1.  4.5 0.5]
error:     0.0000000000

One observation. Three holes filled. Exactly right.

That is matrix completion. That’s the whole idea. Everything after this is just making it work when the structure is only approximate, not exact.

4.3 Rank 2, rank $r$

Rank 1 is too rigid. It says everyone has the same taste and differs only in how generous they are. But real people differ in kind, not just in degree.

Fix: give each person two numbers, and each movie two numbers.

$${\mathbf{p}}_i=[\text{likes ACTION},\text{likes ROMANCE}],{\mathbf{q}}_j=[\text{has ACTION},\text{has ROMANCE}]$$

$$M_{ij}=p_{i,1}{q}_{j,1}+p_{i,2}{q}_{j,2}\leftarrow \text{a dot product}$$

• Alice on Alien: $9(9)+1(1)=82$ — action fan, action movie. High. ✓
• Bob on Alien: $2(9)+8(1)=26$ — romance fan, action movie. Low. ✓

Rank 1 could not do this. It forced Bob to like whatever Alice liked, only weaker. Rank 2 lets Bob like different things.

The general form

$$\boxed{M_{ij}=\sum _{f=1}^r{P}_{if}{Q}_{jf}=⟨P_{i,:},Q_{j,:}⟩}$$

If $r$ is small, then $(n+m)r$ is far less than $nm$. Fewer unknowns than entries. That gap is what lets you fill holes.

This is the entire mathematical content of ToxCompl.

4.4 Swap the nouns: this is DrugMatrix

Change the words, keep the math:

Here is ToxCompl’s model, straight from the paper:

$$G_{ij}\approx b_i+b_j+\sum _{f=1}^r{P}_{if}{Q}_{jf},r=300$$

Two extra terms:

• $b_i$ is the gene bias — “this gene is generally jumpy.”
• $b_j$ is the treatment bias — “this drug is generally toxic.”

Why believe low rank here?

The paper’s argument, and it’s a good one:

“there are similarities between the drugs, for example, Doxorubicin and Epirubicin, and the genes form co-expression networks”

In plain terms:

• There are 3,000 columns, but only 636 distinct drugs. Many columns repeat.
• Doxorubicin and Epirubicin are nearly the same molecule. So their columns are nearly the same. Redundant.
• Genes move in co-regulated groups. So rows are redundant too.
• The same gene is measured in 8 organs. More redundancy.

Redundant rows and columns are exactly what “low rank” means.

4.5 How you actually fit it

There’s no formula that solves this in one step. The problem is non-convex. So you go downhill:

1. Start $P$ and $Q$ at random values.
2. Measure the error on the entries you did observe.
3. Nudge $P$ and $Q$ in the direction that lowers that error.
4. Repeat.

$$\underset{P,Q,b}{\min }\sum _{(i,j)\in \Omega }(G_{ij}-(b_i+b_j+⟨P_{i,:},Q_{j,:}⟩){)}^2$$

⚠️ The sum runs only over observed entries ($\Omega$). Missing entries add nothing to the loss. They get filled in purely because the low-rank structure forces the whole table to stay consistent. That’s the trick.

Optimizer: Adam, learning rate $10^{-3}$, weight decay $5\times 10^{-4}$.

4.6 The counting that decides everything

Take a 12×10 table of true rank 2. Change how much of it we observe:

That table is the whole lesson. There’s a sharp jump right around obs/param = 1.

Matrix completion isn’t magic. It’s just solving a system of equations. Each observation is an equation. Each entry of $P$ and $Q$ is an unknown.

• Many more equations than unknowns → clean recovery.
• Barely more → fragile.
• Fewer → arbitrary. Garbage.

Now count the real matrix

DSMatrix is 375,000 rows × 3,000 columns = 1.12 billion entries. 12.3% are observed = 138 million.

$$\text{observations per gene row}=\frac{138,000,000}{375,000}\approx 369$$

To fit one gene’s vector $P_{i,:}\in {\mathbb{R}}^r$, you need at least $r$ observations in that row:

This explains a failure the paper reports but doesn’t explain

ToxCompl+ Figure 2 shows something odd: going from $r=300$ to $r=500$ made MAE better but MaxAE worse. The paper calls this surprising and moves on.

Here’s why it happens.

At $r=500$, each gene row has about 369 observations but 500 free parameters. The row is underdetermined. The fit is arbitrary in roughly 131 directions.

And that’s the average row. Brain shows up in only ~65 treatments. So a brain gene has 65 observations for 500 parameters — underdetermined by nearly 8×. Its latent vector is essentially random in most directions.

Even $r=300$ is only marginal (ratio 1.23). ToxCompl is running at the edge of identifiability, and the tissues with the least data fall off that edge.

Remember this. It’s exactly what a foundation model fixes (§4.10).

4.7 Where ToxCompl breaks: the sign flip

ToxCompl+ Figure 2 reports this, moving from DrugMatrix to the bigger DSMatrix:

And at the worst single point:

$$\text{target}=-0.77\text{(gene SUPPRESSED)}\text{predicted}=+0.94\text{(gene ACTIVATED)}$$

The sign flipped. This isn’t “a little off.” It’s the opposite conclusion.

Why it happens

Low rank assumes everything is a mix of $r$ shared patterns. But a rare extreme signal is, by definition, not a shared pattern. It’s the part of the table that isn’t low-rank at all. So the model can’t represent it.

The demo reproduces this. A 200×150 matrix, true rank 4, plus rare spikes on 1% of the entries:

typical |value| in the smooth part : 1.48
spike magnitudes                   : 6 to 10   ← rare, large, NOT low-rank

held-out MAE on NORMAL entries : 0.178
held-out MAE ON SPIKES         : 7.845   ← 44x worse
MaxAE                          : 10.099
SIGN FLIPS on held-out spikes  : 49%     ← the ToxCompl failure

49% sign flips on the rare signals. A coin toss. The average looks fine, but the only entries with toxicological meaning are wrecked.

This is why MAE alone can fool you. The average hides the failure.

✅ But credit where it’s due: the ToxCompl authors saw this exact problem and built for it. The published paper (§4.12) doesn’t hide behind MAE. It reports Mean F1 across the five categories precisely to check whether the rare up/down signals survive, and its main contribution — hybrid sampling — exists to preserve them. So “MAE hides the failure” is a fair point about the metric in general, but not a fair criticism of this paper, which measures the right thing and reports it.

4.8 ToxCompl+ fix 1 — side information

The idea: we know things about the rows and columns that the plain model ignores.

Two treatments labeled “Doxorubicin at 100mg” and “Doxorubicin at 200mg” should end up with similar latent vectors. But plain ToxCompl doesn’t know they share a drug. To it, they’re just column 47 and column 112.

So embed each feature and add the interactions:

$${\hat{G}}_{ij}=b_i+b_j+P_{i,:}\cdot Q_{:,j}+\underset{\text{row}\times \text{col-features}}{\underbrace{\sum _{t=1}^k{P}_{i,:}\cdot C_{:,j}^t}}+\underset{\text{row-features}\times \text{col}}{\underbrace{\sum _{t=1}^k{R}_{:,i}^t\cdot Q_{j,:}}}$$

Result (Figure 3): MaxAE 3.99 → 3.27, but MAE 0.03 → 0.05.

That’s a trade, not a clean win. Better on the extremes, worse on the average.

4.9 ToxCompl+ fix 2 — attention (the signature move)

Plain factorization:

$${\hat{G}}_{ij}=P_{i,:}\cdot Q_{j,:}=\sum _f{P}_{if}{Q}_{jf}$$

Every one of the $r=300$ dimensions counts equally. That’s just what a dot product is — an unweighted sum.

But maybe dimension 7 matters a lot for this gene-treatment pair and not at all for another one. A plain dot product can’t say that.

So ToxCompl+ learns two extra tables, $P^{\prime }$ and $Q^{\prime }$. Their only job is to decide how much each dimension counts:

$$\boxed{{\hat{G}}_{ij}=(P_{i,:}\circ Q_{j,:})\cdot \sigma (P_{i,:}^{\prime }\circ Q_{j,:}^{\prime })}$$

Here $\circ$ is the elementwise product — multiply position by position, don’t sum. And $\sigma$ is softmax.

Worked out, with $r=4$

P[i,:]           = [ 2.0  -1.0   3.0   0.5]
Q[j,:]           = [ 1.0   2.0  -1.0   4.0]
P ∘ Q  (values)  = [ 2.0  -2.0  -3.0   2.0]     ← "what each dim says"

P'[i,:]          = [ 0.1   0.2   3.0   0.1]
Q'[j,:]          = [ 0.5   0.3   2.5   0.2]
P' ∘ Q'          = [0.05  0.06   7.5  0.02]
softmax(P' ∘ Q') = [0.00  0.00  1.00  0.00]     ← "how much each dim counts"

PLAIN dot product  : sum(P ∘ Q)        = -1.000
ATTENTION-weighted : (P ∘ Q) · softmax = -2.994

The plain version gives every dimension weight 1. The attention version gives dimension 3 weight ≈ 1.00 and the rest ≈ 0. So the answer is dominated by dimension 3’s value, which is −3.0.

Why this really is attention

values   = P ∘ Q                (r of them)
weights  = softmax(P' ∘ Q')     (sum to 1)
output   = Σ values × weights

That’s attention with $r$ “positions” and one query. Same three lines as a transformer. The paper is right to call it attention.

Result: MaxAE 3.27 → 0.83, MAE → 0.02. A large win.

The fair objection

Is the win from attention, or just from doubling the parameters ($P,Q\to P,P^{\prime },Q,Q^{\prime }$)?

The paper answers this in part. Raising $r$ from 300 to 500 also adds parameters, and that made things worse. So extra capacity alone doesn’t explain it. Decent evidence.

Not airtight, though. The clean test is to run $P^{\prime },Q^{\prime }$ with $\sigma =\mathrm{id}$ (identity instead of softmax). That would isolate the nonlinearity from the extra capacity. Worth asking whether it was run.

(Note from §4.6: the $r$=500 comparison is confounded anyway — it went underdetermined, which is a different failure from “more capacity didn’t help.” So the ablation is still needed.)

4.10 Where a foundation model plugs in

Look at $P$ again.

$$P\in {\mathbb{R}}^{375,000\times 300},\text{row }i\text{ is a 300-dim vector describing gene }i$$

$$\boxed{\text{P IS A GENE EMBEDDING TABLE.}}$$

Same realization as TransTissueFormer’s bottleneck (07_TRANSTISSUEFORMER.md). And right now it is randomly initialized and learned from about 369 observations per row.

From §4.6: at $r=300$ that’s a ratio of 1.23 — marginal. At $r=500$ it’s underdetermined. For brain genes (~65 observations, 300 parameters) it’s hopeless.

A pretrained gene embedding is a prior. And a prior is the only thing that can fix an underdetermined system.

This isn’t a nice-to-have. When you have 65 observations and 300 unknowns, you must bring in outside information, or the answer is arbitrary. scGPT’s gene embeddings are outside information — learned from 33 million cells.

Demonstrated

Set up genes in three co-regulated modules. Starve the factorization of data so the system is genuinely underdetermined, like the real low-data tissues:

matrix 60x50, true rank 4, observed 10%
observations: 286   free parameters: 440   ratio: 0.65   ← UNDERDETERMINED

                                    held-out MAE     MaxAE
------------------------------------------------------------
random init  (ToxCompl today)             1.9851   13.7905
PRETRAINED init (scGPT-style)             0.6886    5.4533

Better MAE and better MaxAE — and MaxAE is the metric ToxCompl+ was built to fix. The prior does the job the data can’t, because the data isn’t there.

⚠️ The control that actually matters

The real comparison is not “pretrained vs random.” That one is rigged — of course a prior helps when the system is underdetermined.

The real question is: does scGPT beat the co-expression structure you can compute from DrugMatrix itself? DrugMatrix already tells you which genes move together. That’s a free prior, with no foundation model involved.

If scGPT loses to DrugMatrix’s own co-expression, that’s a result too. It’s what Kedzierska et al. (2025) and Souza & Mehta (2026) would predict. It turns “we plan to explore this in future work” into “we tested it, here’s the evidence.”

Either outcome publishes. That’s what makes it the right first experiment.

4.11 ToxCompl vs ToxCompl+ side by side

The last row is the opening. Both versions learn a 375,000 × 300 gene embedding table from scratch, from about 369 observations per row, in a system that’s marginally determined at best.

4.12 What the published paper actually does

§4.8–4.9 covered ToxCompl+, the sequel. This section is the original ToxCompl it was built on: “Completion of the DrugMatrix Toxicogenomics Database using ToxCompl”, Cong, Patton, … Auerbach. It’s the foundational paper of the whole program, and it’s more careful than the chapter above might suggest. Here’s what it really does, and what it corrects.

The real fix: hybrid sampling, not attention

The published version’s answer to the rare-signal problem (§4.7) is not side-information-plus-attention. It’s hybrid sampling guided by Bayesian optimization:

• The 5 categories are wildly imbalanced. 92% of values sit in category 0 (“no change”).
• Left alone, the model just learns to predict 0 and ignores the rare extremes.
• So they over-sample and under-sample the minority categories, and use Bayesian optimization to find the best mix.

Result (their Figure 2): Mean F1 about 2.6× better than plain matrix completion at recovering the rare up/down signals, while keeping MAE about the same.

That’s the same instinct as “class weights for an imbalanced classifier” (12_TRANSLATION_TRANSFER.md §12.8), reached from the completion side.

They do report a real metric — and a baseline

And there is a baseline: Nearest-Neighbor-Mean (NNM, $k=10$). For each missing entry, average its 10 nearest measured neighbours in the same row. ToxCompl beats it per-organ (their Figure 1).

⚠️ Correction to the rest of this book. Two critiques I make elsewhere are too harsh against this paper:

• “they only use MAE” — false. They report F1 and Mean F1, and Mean F1 is the number the method exists to move.
• “no baselines” — partly false. NNM is a real baseline for the completion task. (It is not the cross-tissue mean-predictor from 14_RESEARCH_AGENDA.md §A0 — that’s a different task — so that critique still stands for TransTissueFormer. But for ToxCompl completion, a baseline exists.)

They did the biological validation — the gold standard

Not just numbers. The published paper checks the completed matrix against biology:

• connectivity analysis — do predicted profiles cluster with measured ones the way real biology does?
• pathway / MOA analysis — do the differentially-expressed gene lists point at the right mechanisms?
• transcriptional biomarkers of tissue toxicity
• predicted apical endpoints — does the filled-in data predict actual organ damage?

This is exactly the enrichment-consistency direction 15_FRONTIER.md F6 argues the field should move toward — and ToxCompl is already doing it. It just hasn’t propagated to TransTissueFormer.

And they name their own weakness — which matters to a regulator

The paper flags a positivity bias: when predicting histopathology, the model over-predicts damage. It cries “toxic” too often.

That’s the honest, important kind of failure. A false alarm (“this safe chemical looks toxic”) wastes a real animal study. And in the phase-out world of 02_TOXICOLOGY_CONTEXT.md, a screening tool that over-flags is a specific, named problem that regulators care about. That’s worth more attention than any MAE number.

It’s a deployed resource, not just a paper

The completed DrugMatrix is released publicly by NIEHS: rstudio.niehs.nih.gov/complete_drugmatrix. Gene expression, clinical chemistry, hematology, and histopathology — filled in and browsable.

This confirms 02_TOXICOLOGY_CONTEXT.md’s central point: this is a regulatory toxicology program that uses ML, not an ML program that happens to use tox data. The output is a government resource, not a leaderboard entry.

They know about the GAN competitors

The paper cites ToxGAN and TransOrGAN — the FDA NCTR generative work behind TransTox (03_TOXICOGENOMICS_RESEARCH.md §3.5). So the ORNL group is aware of the FDA group’s methods. That softens the “they ignore TransTox” worry somewhat — though TransTissueFormer specifically still under-cites it.

What still stands

The durable critiques survive:

• The obs/param counting (§4.6) is still the best explanation for the $r$=500 MaxAE failure, and the paper still doesn’t give it.
• The gene table is random-init (§4.10). A pretrained prior is still the natural fix.
• Row-wise metrics. F1 is a real metric, but it’s still per-entry. The gene-network / row-structure argument (16_MATH_NOTES.md §3.5) is untouched.

4.13 Summary

Matrix factorization. Every row is a combination of $r$ shared patterns. So $(n+m)r$ numbers describe $nm$ entries. Fewer unknowns than entries means you can fill holes. ToxCompl: $G_{ij}\approx b_i+b_j+⟨P_{i,:},Q_{j,:}⟩$, $r=300$, fit by Adam on observed entries only.

Why it breaks. Rare extreme signals are the part that isn’t low-rank. MAE improves while MaxAE degrades, and signs flip: target −0.77, predicted +0.94. Reproduced: 49% sign flips on held-out rare signals.

A quantitative reason the paper doesn’t give. About 369 observations per gene row. At $r=300$ the ratio is 1.23 (marginal). At $r=500$ it’s 0.74 (underdetermined). That’s why raising $r$ made MaxAE worse. Brain genes (~65 obs, 300 params) are hopeless.

ToxCompl+ fix 1 — side information. Use drug name, dose, organ. MaxAE 3.99 → 3.27, MAE 0.03 → 0.05. A trade.

ToxCompl+ fix 2 — attention. $(P\circ Q)\cdot \sigma (P^{\prime }\circ Q^{\prime })$. Values from one pair of tables, weights from another. Literally attention with $r$ positions and one query. MaxAE 3.27 → 0.83. A large win.

Where the foundation model goes. $P$ is a gene embedding table, randomly initialized, fit from ~369 observations per row for 300 parameters. That system is barely determined, and for low-data tissues it’s not determined at all. A pretrained gene embedding is a prior — the only thing that can fix an underdetermined system. Control against DrugMatrix’s own co-expression, or the comparison is rigged.

Every number here comes from code/demo_matrix_factorization.py. The obs/param analysis uses the paper’s stated DSMatrix dimensions (375,000 × 3,000, 12.3% observed) and its stated $r$ values. The underdetermination explanation for the $r$=500 failure is my reading, not the paper’s — it follows from the counting, but should be checked against the implementation.

Chapter 5 — TransPlatformer: Translating Between Generations of Machines

Why it’s needed, what it does, and the finding buried in §5.5 that matters more than the headline.

Paper: “TransPlatformer – Translating Toxicogenomic Profiles Between Generations of Platforms” (ORNL / Sciome / NIEHS).

Read 04_TOXCOMPL.md first. This is the sibling of TransTissueFormer — same architecture, different axis.

5.1 Why this exists: a very expensive problem

Over 20 years, three generations of machines measured gene expression:

Here’s the problem in one sentence:

Decades of toxicology were run on CodeLink and Affymetrix. Modern work is BioSpyderWT. The old data can’t be pooled with the new. So it sits there, paid for, unusable.

Why can’t they be pooled? The two technologies produce different kinds of number:

• A microarray is a glass chip. Probes glow, and you measure brightness. That’s analog. It saturates, and it has background noise.
• A sequencer reads the RNA letter by letter and counts reads. That’s digital, with a huge dynamic range.

You can’t put a brightness and a count on the same axis. They aren’t the same kind of number.

NLP analogy: it’s the same language, transcribed by three different systems with different phoneme inventories and different error profiles. You have a huge corpus in the old transcription, and your model expects the new one. TransPlatformer is a normalizer.

But wait — doesn’t fold-change fix this?

Partly. And that’s the whole point. From 01_BACKGROUND.md §4.2: dividing treated by control makes the numbers unitless, so the platform’s scale factor cancels — to first order.

It doesn’t cancel all the way, because saturation and background are nonlinear. A gene at the top of CodeLink’s range is clipped; the same gene on BioSpyderWT isn’t. Fold-change doesn’t undo clipping.

That leftover part — the bit fold-change can’t remove — is exactly what TransPlatformer models. Everything else was already handled by the ratio.

5.2 What it actually does

$${\mathbf{x}}^{\text{CodeLink}}\in {\mathbb{R}}^{8,565}⟼{\mathbf{x}}^{\text{BioSpyderWT}}\in {\mathbb{R}}^{22,794}$$

Same tissue. Same treatment. Different machine.

⚠️ Notice the dimensions don’t match. $n=8,565\to m=22,794$. This is not a square problem. TransTissueFormer maps $n\to n$; TransPlatformer maps $n\to m$. Same architecture, but the output side has to be sized differently — hence 45 decoders ($\lceil 22794/512\rceil =45$) instead of TransTissueFormer’s 17.

The architecture — identical to TransTissueFormer

(B, n)  →  [n×s bottleneck]  →  (B, s)  →  [1×r]  →  (B, s, r)
        →  [L = 32 layers]   →  [⌈m/s⌉ = 45 decoders]  →  (B, m, r)  →  [r×1]  →  (B, m)

$s=512$, $r=16$, $L=32$, dropout 0.1, Adam $10^{-3}$, batch 16.

Everything in 07_TRANSTISSUEFORMER.md §7.3–7.6 applies word for word, including:

• The bottleneck is a gene embedding table — row $g$ of $W$ is a 512-dim vector for gene $g$.
• The bottleneck holds the overwhelming majority of the parameters. It’s worse here: on Affymetrix it’s $31,042\times 512=15.9\mathbf{million}$ parameters, against a ~150k transformer stack.
• The transformer never sees a gene.
• The 1×r projection makes the tokens collinear at layer 0.

5.3 The Seq2Seq ablation — a number worth remembering

They tried a plain Seq2Seq transformer. Treat the profile as a sentence, decode it one token at a time, <EOS> and all.

Here’s the result, on an NVIDIA Tesla V100S 32GB, sequence length 8,565, batch 8:

The number of layers that fit: 2.

Two. Against TransPlatformer’s 32.

And on the 2 layers that fit, TransPlatformer is 2.74× better on overall MAE and 2.27× better on rare-signal MAE. Seq2Seq loses to the MLP too.

This is why “we can’t use standard transformers” isn’t hand-waving. It’s not that they’re slow. At $n=31,042$ (Affymetrix) a standard transformer doesn’t run at all — see 07_TRANSTISSUEFORMER.md §7.4: about 2 TB of attention scores.

Their three reasons for rejecting Seq2Seq (their §2.2), all sound:

1. Memory. $O(n^2)$ attention on 31,042 tokens. Dead.
2. Genes aren’t ordered. “the first $s$ genes as tokens may not necessarily hint at the $(s+1)$th token.” Correct — there’s no order; there’s a graph.
3. Profiles are ~90% identical to each other. “two distinct transcriptomic profiles for two different treatments can share more than 90% of their content.” Feed those in as “sentences” and the model drowns.

Point 2 is right that there’s no order, which means no positional encoding — a set transformer. But that doesn’t mean no attention. Their conclusion overshoots their premise a little. Point 1 alone settles it, though.

5.4 Three training strategies — and this is the interesting part

The data is skewed (liver everywhere, brain almost nowhere). So how do you train?

Sample counts: CodeLink→BioSpyderWT = 2,225 pairs. Affymetrix→BioSpyderWT = 888.

Mixed-tissue — the headline

MAE cut by more than half versus ToxCompl (0.043 vs 0.09). PCC doubled versus the MLP (0.71 vs 0.37). Rare-signal MAE under 0.22.

Why compare against ToxCompl at all? Because ToxCompl can also fill in the missing entries — but it’s “agnostic to the relationship between data entries and the platforms.” So the comparison asks one precise question:

“whether additional side information, in this case the correspondence of genes across platforms, can help improve prediction performance.”

It does — by more than half. That’s a clean, well-posed ablation, and it’s the best-designed comparison in the four papers.

The mixed-tissue assumption, stated plainly:

“It assumes that the tissue-identifying signals are latent in the vectors, and with a powerful architecture and plenty of [data]…”

And here’s the thing: that assumption is correct — for this task. Hold that thought; §5.6 is about why.

5.5 Zero-shot transfer, and nobody is talking about it ⭐⭐⭐

This is the most important section in the paper, and it’s not in the abstract.

The problem: brain and intestine have almost no data. Single-tissue models for them fail.

So they tried this:

Train on $S=\{\text{LI, KI, HE, SM}\}$. Test on $T=\{\text{BR}\}$ or $\{\text{IN}\}$.

“Note that data from the target tissues are completely withheld from training.”

Zero brain data. Zero intestine data. Predict brain and intestine.

The result

A model that had never seen a brain beat the model trained on brain’s own data — by 2.42×.

What this means

Read it slowly, because it’s counterintuitive:

• The single-tissue brain model had brain data. It was trained on the actual target.
• The cross-tissue model had zero brain data. It saw liver, kidney, heart, muscle.
• The one with no target data won, by a factor of 2.4.

Why? Because brain’s own data is so scarce that training on it is worse than useless — you overfit ~65 samples. But the translation function itself — “how does CodeLink’s brightness map to BioSpyder’s counts?” — is largely tissue-independent. It’s a property of the machines, not the organ. So liver data teaches you the machine, and the machine is the same for brain.

Their own framing:

“Cross-tissue mode assumes that some fundamental relationships among genes are preserved across tissues and can be leveraged for translation.”

“Figure 6 suggests that the translation mechanism learned from one set of tissues may be applied to another set.”

And then they did few-shot

“To further improve the performance of the model, we include a very small fraction of the data (e.g., one data sample) from the target tissue in training.”

One sample. They call it Cross-tissue*.

NLP framing, and it’s exact:

This is massively multilingual NMT’s central finding. A low-resource language does better inside a multilingual model trained mostly on other languages than in its own bilingual model. Its own corpus is too small; the shared structure carries it.

§5.5 is zero-shot cross-lingual transfer, plus one-shot fine-tuning, in a toxicogenomics paper — reported as a paragraph and a bar chart.

5.6 The contradiction — and why it isn’t one ⭐⭐

Put two facts from the same program side by side:

Same group. Same architecture. Same data. Opposite outcomes. What gives?

The tasks are structurally different, and the difference is exactly conditioning:

TransPlatformer:
    input:   CodeLink profile of tissue X
    output:  BioSpyder profile of tissue X          ← SAME tissue
    → the tissue is a NUISANCE VARIABLE. It's implicit in the input.
    → the model never has to CHOOSE. Mixing tissues just adds data.
    → mixed-tissue WORKS.

TransTissueFormer:
    input:   liver profile
    output:  kidney? heart? spleen?                 ← DIFFERENT tissue, YOUR CHOICE
    → the tissue is a FREE PARAMETER. It is not in the input.
    → the model MUST choose and is never told which.
    → multi-task COLLAPSES.

The two results are perfectly consistent. Together they prove the conditioning argument from inside the program.

TransPlatformer doesn’t need a target token — the target tissue equals the source tissue, so it’s already in the input. Mixing tissues is free data.

TransTissueFormer does need one — the target tissue is a choice the model is asked to make blind. Under squared loss it correctly returns $\mathbb{E}[{\mathbf{t}}_c\mid {\mathbf{s}}_c]$, the average over organs (16_MATH_NOTES.md §4.2). Mush is the right answer to an ill-posed question.

So this is no longer my argument from the NMT literature. It’s the program’s own two results, read together. TransPlatformer proves the shared trunk works. TransTissueFormer proves you need the target token. Add ${\mathbf{e}}_{\tau }$ and you get both.

And §5.5 makes the zero-shot claim in 12_TRANSLATION_TRANSFER.md §12.4 far less speculative than I first rated it. Zero-shot to brain already worked once, on the platform axis, in this program. The tissue axis needs conditioning — but the transfer mechanism is demonstrated.

5.7 Attention interpretability — a real bonus

Translating CodeLink → BioSpyderWT, the model attends most to:

Cmya1, Ca3, Cyp1a1, Ctsh, Sds, Cited4, Atf3, Lcn2, Stac3

Several are well-known toxicology genes:

And per-tissue (heart): Arntl, Timm8a, Rufy1, Kap, Atf3, Sln, Gal, Alas2, Fos — Arntl is a circadian clock gene, Fos an immediate-early stress gene. Plausible for heart.

This is a genuine sanity check. The model attends to genes a toxicologist would name. It’s weak evidence — attention weights aren’t explanations, and that’s a whole literature — but it’s the right kind of evidence, and it costs nothing.

⚠️ Be careful here. “Attention is not explanation” (Jain & Wallace 2019) is a well-known result in your field, and it applies. High attention does not mean high importance. But “the model attends to Cyp1a1 when translating toxicogenomics” is still a reassuring smell test, and they present it as one rather than over-claiming.

5.8 The downstream result — the most convincing number in all four papers ⭐

Everything above is intrinsic evaluation. This is the one that matters.

Task: predict liver necrosis (does this compound damage the liver?) from a BioSpyderWT profile. A simple MLP classifier.

Experiment: train it on BioSpyderWT only. Then translate the legacy Affymetrix data into BioSpyderWT format with TransPlatformer, add it to the training set (nearly doubling it), and retrain the same model with the same hyperparameters.

That’s +12.9% relative F1. All four metrics improve.

Why this is worth more than every PCC in the four papers

PCC 0.71 tells you the predicted vector correlates with the true vector. So what? Nobody deploys a predicted vector.

This says something else: translated legacy data made a real classifier measurably better at a real task. The old CodeLink and Affymetrix data, sitting unusable for 20 years, became useful. That’s the paper’s actual claim, tested end to end.

It’s the same kind of evidence as TransTissueFormer’s enrichment validation (gemfibrozil → PPARα ✓), and it’s the direction 15_FRONTIER.md F6 argues the whole field should move: score models by whether they help the downstream task, not by whether the vectors correlate.

⚠️ One small reporting inconsistency: the abstract says F1 “improves by approximately 8%”; the results section says “12.9% improvement.” Both are right — 8.2 percentage points (0.636 → 0.718) versus 12.9% relative. Worth picking one convention and stating it consistently.

5.9 What’s still open

5.10 Summary

Why it’s needed. Three generations of machines. Microarrays measure brightness; sequencers count. You can’t pool them. Decades of paid-for toxicology data sit unusable. Fold-change cancels the platform to first order — but saturation and background are nonlinear and survive. That leftover is what TransPlatformer models.

What it does. $8,565\to 22,794$. Same tissue, same treatment, different machine. Same architecture as TransTissueFormer, 45 output decoders instead of 17.

The headline. MAE 0.043 vs ToxCompl’s 0.09 (>50% better); PCC 0.71 vs the MLP’s 0.37 (doubled). The ToxCompl comparison is well-posed — it isolates “whether the correspondence of genes across platforms helps.” It does.

The Seq2Seq number. On a 32GB V100S at length 8,565: 2 layers fit, versus 32. TransPlatformer is 2.74× better on MAE. The compute argument is not hand-waving.

The finding nobody mentions (§5.5). Train on {liver, kidney, heart, muscle}, withhold brain entirely, test on brain: 2.42× better than the model trained on brain’s own data. Then one brain sample improves it further. That is zero-shot and one-shot cross-lingual transfer. It’s not in the abstract.

The contradiction that isn’t. TransPlatformer’s mixed-tissue works; TransTissueFormer’s multi-task collapses. Consistent: TransPlatformer’s target tissue equals its source tissue (implicit in the input, no choice to make), while TransTissueFormer’s is a free choice it’s never told. The program’s own two results prove the target-token argument from the inside.

The best number in all four papers. Harmonize legacy Affymetrix into BioSpyderWT, add it to a liver-necrosis classifier: F1 0.636 → 0.718, all metrics up. Twenty years of unusable data, made useful, measured on a real task.

Numbers are verbatim from the paper. §5.6’s reading of the TransPlatformer/TransTissueFormer relationship is mine — it follows from the two task definitions, but I’d want the authors to confirm the framing. Architecture claims inherit the caveats in 07_TRANSTISSUEFORMER.md §7.6.

Chapter 6 — GenTox: Deep Inductive Matrix Factorization

The three components, from zero, with numbers. Run code/demo_inductive_mf.py alongside. Read 04_TOXCOMPL.md first — this builds on plain MF.

Paper: “Predictive Transcriptomics with Attention-Augmented Inductive Matrix Factorization and Graph Mutual Information for Small Molecule Drugs” (GenTox).

The paper has three components:

This chapter is where the whole document set converges. The punchline is in §6.6.

6.1 The problem plain MF cannot solve

Recall ToxCompl:

$$G_{ij}=⟨P_{i,:},Q_{j,:}⟩$$

$P$ and $Q$ are lookup tables. To get $Q_{j,:}$ you look up row $j$ of a big array of learned numbers.

Now a chemist synthesizes a brand-new compound. Predict its profile.

Which row of $Q$ do you look up?

There isn’t one. The table has 2,238 rows, and this drug is number 2,239.

You can’t even ask the question. The model has no input slot for it. This isn’t “poor accuracy” — the function is undefined.

This is the cold-start problem. Netflix has it too: a movie released today has zero ratings, so no learned vector, so no recommendation.

That’s why GenTox exists. ToxCompl fills holes in a matrix. GenTox predicts a whole new column.

Two different verbs

6.2 §3.1 — the fix: replace the lookup with a function

The whole idea in one line:

$$\text{TRANSDUCTIVE:}{Q}_{j,:}=\text{lookup}[j]⟵\text{dies on new }j$$ $$\text{INDUCTIVE:}{Q}_{j,:}=f(\text{features}(j))⟵\text{works on any }j$$

A new drug has no row in a table. But it has features — it’s a molecule, and you can measure things about it. Feed the features to $f$, get a vector out.

GenTox’s Figure 3:

  row features (gene, tissue)          ──► Row NN ──┐
                                                     ├──► MIXER ──► G'[i,j]
  col features (compound, dose, time)  ──► Col NN ──┘

The paper’s own note on why this is a generalization, not a new thing:

“The model degenerates to a standard matrix factorization approach when the outputs of the row NN and the column NN have the same dimensions and the mixer network is simply a dot product.”

So plain MF is just inductive MF with $f$ = lookup and mixer = dot product. GenTox is strictly more general.

Watch the difference

A toy: 40 genes × 60 drugs. Each drug has 12 molecular descriptors that cause the response. Train on 50 drugs; the last 10 are new compounds — never seen, not one measurement.

---- TRANSDUCTIVE (plain MF, ToxCompl-style) ----
    on SEEN drugs : relative error 0.0000   ← excellent
    on NEW drugs  : *** CANNOT RUN ***
                    Q has 50 rows. The new drugs are rows 50-59.
                    There is nothing to look up. The function is undefined.

---- INDUCTIVE (GenTox-style) ----
    on SEEN drugs : relative error 0.0040
    on NEW drugs  : relative error 0.1656   ← *** IT RAN ***

It predicted profiles for 10 compounds it had never seen, from their molecular descriptors alone.

Nothing clever happened. We replaced one lookup with one small MLP. That’s the entire content of “deep inductive matrix factorization.”

One more thing §3.1 does

“As during training the samples in each minibatch are drawn uniformly at random from the whole G, the model in GenTox with three networks is expected to learn to not only predict the profile for a certain treatment but also preserve the relationships between endpoints in different regions of the matrix.”

Sampling random cells (not whole columns) is what forces the model to preserve row structure as well as column structure. That directly addresses the rank-degeneracy argument in 16_MATH_NOTES.md §3.4. It’s a deliberate, well-motivated design choice.

6.3 §3.2 — everything now depends on the features

The inductive model is only as good as $\text{features}(j)$.

The paper calls $\text{features}(j)$ the induction basis — the thing that makes induction possible at all.

Feature quality decides everything

induction basis                   SEEN drugs     NEW drugs
----------------------------------------------------------
informative descriptors               0.0020        0.1664
descriptors + heavy noise             0.0050        1.2549
random features                       0.0002        1.8598   ← !!
half the descriptors                  0.0049        1.0792

⭐ Read the random features row. It has the best in-distribution fit of all four (0.0002 — the MLP simply memorises) and the worst generalization (1.86).

A model can look perfect in-distribution and be worthless inductively. Only the new-drug column reveals it. That gap is the whole game.

Option A — hand-crafted features (the classical answer)

Mordred descriptors: about 1,826 numbers a chemist decided are meaningful. Molecular weight, logP, ring count, polar surface area. Deterministic — run the molecule through a fixed algorithm.

Morgan / ECFP fingerprints: a bit vector. Bit 4,412 = “does this molecule contain –C(=O)NH– ?” One bit per substructure.

In GenTox: $s=1826+1+1=1828$ — Mordred, plus dose, plus duration.

NLP analogy: this is a bag of hand-designed features. It’s what you did before word embeddings — someone decides in advance what properties matter, and you count them.

Option B — learned features (GenTox’s contribution)

Pretrain a graph neural network on about 1 million compounds, self-supervised.

A molecule is a graph: atoms are nodes, bonds are edges.

B1 — Contrastive (positive/negative pairs)

Take a molecule. Corrupt it twice, differently. Both corruptions came from the same molecule, so they’re a positive pair. A corruption of a different molecule is a negative pair. Train so positives end up close and negatives far apart.

Their corruptions: atom (node) masking plus bond (edge) deletion, “taking care not to select a node whose removal will result in a disconnected graph.”

$$\mathcal{L}=-\sum _i\log \frac{\exp (\mathrm{sim}(z_i,z_i^{+})/\tau )}{\exp (\mathrm{sim}(z_i,z_i^{+})/\tau )+{\sum }_{j\ne i}\exp (\mathrm{sim}(z_i,z_j^{-})/\tau )}$$

This is SimCLR. Identical objective. Molecules instead of images, atom-masking instead of cropping.

B2 — Graph InfoMax (mutual information)

Different idea: make each atom’s embedding predictive of the whole molecule’s embedding.

$$\underset{\theta }{\max }\sum _G\sum _{v\in V(G)}{I}_{\theta }(z_v;z_G)$$

Mutual information is intractable, so train a discriminator $D_{\theta }$ to tell real (atom, its-own-molecule) pairs from fake ones:

$$\mathcal{L}=-\sum _{v\in V}\log D_{\theta }(z_v,z_G)-\sum _{v^{\prime }\in V^{\prime }}\log (1-D_{\theta }(z_{v^{\prime }},z_{G^{\prime }}))$$

Intuition: if an atom’s neighbourhood tells you which molecule you’re in, the embedding has captured real structure.

Encoder: 2-layer GCN, hidden 64, output FC 64×300.

$$X^{\prime }={\hat{D}}^{-1/2}\hat{A}{\hat{D}}^{-1/2}X\Theta ,\hat{A}=A+I$$

Graph embedding = mean of node embeddings.

Their finding ⭐

graph InfoMax > contrastive pairs ≫ Mordred ≈ Morgan

“Both mutual information based approaches perform slightly better than the positive-negative pair approach, and much better than traditional fingerprinting (e.g., Morgan fingerprint) or descriptor (Mordred) approaches.”

Learned representations beat hand-crafted ones.

That sentence should sound familiar. It’s word2vec beating bag-of-words. BERT beating TF-IDF. Your field’s entire 2013–2018.

And note what this means ⭐⭐

GenTox pretrains a GNN on 1,000,000 compounds, self-supervised, freezes it, and uses it as a feature extractor.

That is a foundation model.

This program already built one — on the chemistry side. It did so before the TransTissue paper wrote “we plan to explore the adaptation of these models in future work.”

6.4 §3.3 — attention-based aggregation (the Mixer)

Now you have $R_i$ (gene factor) and $C_j$ (drug factor). Combine them.

The plain version: $G_{ij}=R_i\cdot C_j={\sum }_f{R}_{if}{C}_{jf}$. Every one of the 300 dimensions counts equally, because that’s what a dot product is. But maybe dimension 7 is decisive for this gene–drug pair and irrelevant for another one. A dot product can’t say that.

So GenTox learns two extra factors, $R_i^{\prime }$ and $C_j^{\prime }$, whose only job is to decide how much each dimension counts:

$$\boxed{G_{ij}=(R_i\circ C_j)\cdot \sigma (R_i^{\prime }\circ C_j^{\prime })}$$

$\circ$ is the Hadamard (elementwise) product — multiply position by position, don’t sum. $\sigma$ is softmax.

Worked out, $r=5$

R_i           = [ 3.0  -2.0   4.0   1.0  -0.5]
C_j           = [ 2.0   1.0  -1.0   3.0   2.0]
R_i ∘ C_j     = [ 6.0  -2.0  -4.0   3.0  -1.0]    ← WHAT each dim says

R'_i          = [ 0.2   0.1   2.8   0.3   0.1]
C'_j          = [ 0.4   0.5   2.6   0.2   0.3]
R'_i ∘ C'_j   = [0.08  0.05  7.28  0.06  0.03]
softmax(...)  = [0.00  0.00  1.00  0.00  0.00]    ← HOW MUCH each dim counts

plain dot product  : sum(R ∘ C)        =  2.000
attention-weighted : (R ∘ C) · softmax = -3.984

The plain version gives every dimension weight 1.0. Attention gives dimension 3 weight ≈ 1.00 and the rest ≈ 0, so the answer is dominated by dimension 3’s value (−4.0). The sign flipped, because the plain sum was averaging away the dimension that actually mattered.

Why this really is attention

values  = R ∘ C               (r of them)
weights = softmax(R' ∘ C')    (sum to 1)
output  = Σ values × weights

Query, keys, values — with $r$ “positions” and a single query. The same three lines as a transformer.

Their claim to novelty:

“To our knowledge no prior study has directly introduced attention to the low-rank factorization approach for matrix completion.”

It’s the same mechanism as ToxCompl+ (04_TOXCOMPL.md §4.9). One idea, two papers — worth knowing they’re the same thing.

6.5 A drafting bug worth reporting

The paper contradicts itself inside one paragraph.

§3.1 says:

“From the columns $c_1,c_2,c_3$ GenTox produces column (treatment) features” → columns = treatments

But the Results section says, in consecutive sentences:

“the fully connected layers in the column NN are $s\times 300$ … if the compound is represented by a Mordred descriptor, $s=1828$” → column NN handles compounds

“The col NN is a look-up table of $31099\times 300$ as there are 31099 genes/probesets in AffyMetrix” → col NN handles genes

The same network is described as both a 31,099-gene lookup table and an $s=1828$ compound MLP. The row/column labels are swapped somewhere.

The sensible reading, and what the model must actually be:

Easy fix. Worth flagging because §3.1 is the paper’s central contribution, and the labels are inverted in the very section that implements it.

(The paper is also a draft — broken Figure ?? refs, [?] citations, and §5 “Gene Network Analysis” / §6 “Out-of-distribution validation” are empty section headers.)

6.6 The asymmetry nobody has acted on ⭐⭐⭐

Read this sentence from the implementation section:

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

So here’s the split:

They built a foundation model for the columns and left the rows random.

And they proved, in their own ablation, that learned representations beat hand-crafted ones — for molecules. They never applied that lesson to genes.

This is the same gap in all three papers

Three papers. One randomly-initialized gene embedding table at the core of each. And a public checkpoint trained on 33 million cells that has exactly that object, already learned.

6.7 The fix: be inductive on genes too

$$\text{TODAY:}\text{gene\_factor}(i)=\text{lookup}[i]\text{(transductive)}$$ $$\text{PROPOSED:}\text{gene\_factor}(i)=\text{RowNN}(\text{gene\_features}(i))\text{(inductive)}$$

Where do gene features come from? The same place drug features come from: a pretrained model.

The symmetry is exact. It’s the argument they already made and won, applied to the other axis of the same matrix.

What inductive-on-genes buys — more than a leaderboard bump

1. New platforms. A CodeLink probe and an Affymetrix probe for the same gene are different lookup rows today — the model has no idea they’re the same gene. With gene features they get the same features. That’s TransPlatformer’s entire problem, solved structurally.

2. New species. Rat → human. A lookup table can’t do this, ever. UCE tokenizes genes by their protein sequence, so a rat gene and its human ortholog get similar features automatically, with no dictionary needed (11_SC_FOUNDATION_MODELS.md §6).

3. The low-data tissues. ⭐ This is the big one. From 04_TOXCOMPL.md §4.6: a brain gene has ~65 observations and 300 free parameters. The lookup row is underdetermined — arbitrary in most directions. An inductive gene factor doesn’t need 300 observations. It’s a function of features shared with every other gene, so every gene’s data helps every other gene.

4. Unmeasured genes. A gene never on any array still has a protein sequence → still has features → still predictable.

Demonstrated — the underdetermined regime

45 genes x 40 drugs, observed 11%
-> 197 observations vs 340 parameters, ratio 0.58   (UNDERDETERMINED)
   (values scaled so mean |G| = 1.00, so errors read as fractions)

gene factor                               held-out MAE     MaxAE
----------------------------------------------------------------
LOOKUP TABLE (GenTox today)                      0.990     9.399
MLP on gene features (proposed)                  0.268     3.050

-> MAE 73% lower,  MaxAE 68% lower

Both improve — and MaxAE is the metric ToxCompl+ was built to fix.

⚠️ The control that matters

The comparison is not “features vs lookup.” That’s rigged when the system is underdetermined — of course a function beats 300 free parameters fit from 65 observations.

The real question: do scGPT/UCE gene features beat the co-expression structure you can compute from DrugMatrix itself? That’s a free basis, no foundation model required.

If scGPT loses to DrugMatrix’s own co-expression, that’s a result too. It’s what Kedzierska (2025) and Souza & Mehta (2026) would predict. Either outcome publishes.

6.8 Summary

The problem. Plain MF uses lookup tables. A new drug has no row, so you can’t even ask the question. Cold start.

§3.1 — deep inductive matrix factorization. Replace the lookup with a function of features: $Q_{j,:}=f(\text{features}(j))$. Now a new drug works — it has features even though it has no row. Degenerates to plain MF when $f$ = lookup and mixer = dot product. Random-cell minibatch sampling forces row and column structure to be preserved.

§3.2 — representation learning for the induction basis. Everything depends on $\text{features}(j)$. Random features give the best in-distribution fit and the worst generalization — only the new-drug column reveals it.

• hand-crafted: Mordred (1,826 numbers), Morgan (substructure bits)
• learned: GNN on 1M compounds — contrastive (= SimCLR for molecules) or graph InfoMax

Finding: InfoMax > contrastive ≫ Mordred ≈ Morgan. Learned beats hand-crafted. That’s word2vec vs bag-of-words. And a GNN pretrained on 1M compounds is a foundation model — they built one.

§3.3 — attention-based aggregation. $G_{ij}=(R\circ C)\cdot \sigma (R^{\prime }\circ C^{\prime })$. Values from one pair of factors, weights from another. Attention with $r$ positions and one query. Same mechanism as ToxCompl+.

The opening.

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

They went inductive on drugs with a 1M-compound foundation model, and left genes as a random $31,099\times 300$ lookup table. All three papers have the same random gene table at their core.

Being inductive on genes buys new platforms, new species, the low-data tissues, and genes never measured. The argument is the one they already made and won — applied to the other axis of the same matrix.

Every number here is produced by code/demo_inductive_mf.py. Quotes are verbatim from the GenTox draft. The row/column inconsistency in §6.5 is in the paper as written and may be a labelling slip rather than a modelling error — check the implementation.

Chapter 7 — TransTissueFormer, In Depth

What it is, why it’s fast, why it’s the right shape, where it breaks, and what to do about it.

7.1 The problem it solves

Given the transcriptomic profile of a source tissue under a treatment, predict the target tissue profile for the same treatment.

$${\mathbf{s}}_c\in {\mathbb{R}}^n⟼{\mathbf{t}}_c\in {\mathbb{R}}^n,n=8,565\text{ (CodeLink)}$$

Both are ${\log }_{10}$ fold-change vs. control. A treatment $c$ is a (chemical, dose, duration) triple.

Concretely: Nitrofurantoin, 100 mg/kg, 24h. We measured the liver. Predict the heart.

Why it’s hard: 8,565 → 8,565, from 425 training examples on the best-resourced pair. And 92% of both vectors are ≈ 0.

7.2 The architecture

      (B, n)                          n = 8,565 source genes
        │
        │   [1]  bottleneck FC:  n × s          s = 512
        ▼
      (B, s)                          the profile, compressed to 512 numbers
        │
        │   [2]  projection:  1 × r             r = 16
        ▼
      (B, s, r)                       512 "slots", each a 16-dim token
        │
        │   [3]  L transformer layers ("TP attention")    L = 32
        ▼
      (B, s, r)
        │
        │   [4]  ⌈n/s⌉ = 17 PARALLEL decoders
        ▼
      (B, n, r)
        │
        │   [5]  projection:  r × 1
        ▼
      (B, n)                          predicted target profile

Hyperparameters, from the paper: $n=8565$, $s=512$, $r=16$, $L=32$, $\lceil n/s\rceil =17$ concurrent decoders. Adam, $\eta =10^{-3}$, weight decay $5\times 10^{-4}$, batch 16, 40 epochs.

The three claimed departures from Vaswani

1. The $n\times s$ bottleneck compresses 8,565 genes into 512 slots before attention. Attention then costs $O(s^2)$, not $O(n^2)$. Same spirit as Linformer/Performer — project the sequence dimension down.
2. Every output gene attends to the entire input profile. No context window. Any liver gene can in principle influence any kidney gene.
3. 17 parallel decoders. Not autoregressive — all segments emit at once.

7.3 Worked example: what the bottleneck actually does

Take $n=6$ genes, $s=3$ slots. The bottleneck is $W\in {\mathbb{R}}^{6\times 3}$:

$$W=\left[\begin{array}{ccc}0.9& 0.1& 0.0\\ 0.8& 0.2& 0.1\\ 0.1& 0.9& 0.0\\ 0.0& 0.8& 0.2\\ 0.1& 0.0& 0.9\\ 0.0& 0.1& 0.8\end{array}\right]\mathbf{x}=\left[\begin{array}{c}1.2\\ 0.9\\ 0.0\\ 0.1\\ -0.8\\ -0.6\end{array}\right]$$

The compressed representation is $\mathbf{h}=W^{\top }\mathbf{x}$:

$$h_1=0.9(1.2)+0.8(0.9)+0.1(0)+0(0.1)+0.1(-0.8)+0(-0.6)=1.72$$ $$h_2=0.1(1.2)+0.2(0.9)+0.9(0)+0.8(0.1)+0(-0.8)+0.1(-0.6)=0.32$$ $$h_3=0(1.2)+0.1(0.9)+0(0)+0.2(0.1)+0.9(-0.8)+0.8(-0.6)=-1.09$$

Read the columns of $W$. Column 1 loads on genes 1–2; column 2 on genes 3–4; column 3 on genes 5–6. Those are gene modules — co-regulated programs. Slot 1 says “module A is up (+1.72),” slot 3 says “module C is down (−1.09).”

Now read the rows. Row $g$ of $W$ is a vector in ${\mathbb{R}}^s$ describing how gene $g$ participates in each module.

$$\boxed{\text{Row }g\text{ of }W\text{ is a }512\text{-dimensional embedding of gene }g.}$$

And the operation $\mathbf{h}=W^{\top }\mathbf{x}={\sum }_g{x}_g\cdot W[g,:]$ is:

$$\boxed{\text{an expression-weighted sum of gene embeddings.}}$$

This is word2vec-style document embedding. A document is the weighted sum of its word vectors, with weights = term frequency. Here a profile is the weighted sum of its gene vectors, with weights = fold-change.

So the bottleneck isn’t a compression trick bolted on for compute reasons. It’s a gene embedding table, and the model is doing bag-of-embeddings pooling. That reading is what makes §7.7.1 possible.

7.4 Why it’s fast — the numbers

Attention memory

Standard self-attention over $n$ tokens materializes an $n\times n$ score matrix per head per layer. At fp32, batch 16, $L=32$ layers:

Standard attention on Affymetrix needs about 2 terabytes for score matrices alone. This isn’t “slow.” It doesn’t run, on any hardware that exists.

The bottleneck makes attention memory independent of $n$. It’s always $s^2=262,144$ entries — 0.54 GB across the whole stack. That’s the entire architectural argument in one row of a table.

Decode latency

Autoregressive decoding of an 8,565-token output is 8,565 forward passes, each attending to a growing KV cache. Parallel decoding is one pass. This is the larger practical win, and it’s the one the complexity table doesn’t show.

Complexity table

The paper’s Table 6:

⚠️ The “Standard / $O(s^2)$” entry is a typo. Standard attention’s score matrix is $O(n^2)$; $s$ isn’t even defined for it. Minor, but it’s the row that carries the paper’s central claim, so it should be fixed.

The real difference from Linformer: Linformer projects keys and values to $s$ but keeps $n$ queries, so its score matrix is $O(ns)$. TransTissueFormer projects the input itself to $s$ before attention, so its score matrix is $O(s^2)$ — strictly smaller. The cost is that it can never do gene-level attention again (see §7.6.1).

7.5 Why the shape is right

Three things the design gets genuinely right:

1. No positional encoding. Genes have no linear order — they form pathways and networks. Refusing to impose sequence order is correct, and the reasoning in TransPlatformer §2.2 is sound.

2. Full-profile context. Every output gene sees the whole input. In biology, any liver gene can in principle influence any kidney gene through systemic signalling. A context window would be a lie about the domain.

3. Parallel decoding. No autoregressive error accumulation, no arbitrary generation order. For a set-valued output, generating left-to-right would impose structure that doesn’t exist.

And the compute argument is unanswerable. At $n=31,042$, standard attention needs about 2 TB. There is no version of this project that uses vanilla self-attention over genes.

The honest framing: this is Perceiver-shaped. Cross-attend a huge input into a small latent array, compute in latent space, broadcast back out. The papers cite Linformer and Performer; Perceiver IO is the closer relative and isn’t mentioned. Worth knowing, because Perceiver’s literature — especially on latent-array sizing — applies directly.

7.6 Where it breaks

7.6.1 96.6% of the model is a linear projection ⭐

Parameter budget, CodeLink, from the paper’s stated hyperparameters:

(Standard transformer layer accounting: QKV + output + LayerNorm + a 4× MLP. Decoder depth is ambiguous in the paper — assumed ~1 layer each. Even at 32 layers each, the transformer stack reaches only ~1.8M, and the bottleneck still dominates.)

The transformer stack is 3.4% of the model. The “transformer” in TransTissueFormer is a rounding error on a 4.4M-parameter linear projection.

Three consequences follow.

(a) It explains the results. TransTissueFormer’s MAE is 0.081; the MLP’s is 0.086. A 6% gap. That’s not what you’d expect from a 32-layer transformer beating a 4-layer MLP. But it’s exactly what you’d expect if both models are mostly a big linear map with a small nonlinearity on top.

(b) It explains exp02. The Funk-SVD-augmented task is exactly affine of rank ≤ 301, and empirically rank ~2 (16_MATH_NOTES.md §2). A model that is 96.6% linear projection is a matched filter for a linear task. That’s why it wins Figure 7’s pretrain row (0.90 vs MLP 0.64 vs RF 0.51) — not because it captures “the intricate relationships between transcriptomic profiles from different tissues,” but because its inductive bias is the process that generated the data it’s being scored on.

(c) It’s 10,000 parameters per training sample. 4.4M parameters, 425 examples. The bottleneck is trained from scratch, from random init, on 425 samples.

7.6.2 There is no gene-gene attention

Follow the tensor. After step [1], the profile is 512 numbers. Every gene-level distinction that isn’t preserved by $W^{\top }\mathbf{x}$ is gone before attention runs.

So the 32 transformer layers attend over 512 pooled latent dimensions, not over genes. The claim that “each gene in the target tissue profile can be influenced by any gene in the source” is true — but it’s delivered by the linear pooling, not by the attention. All gene-gene interaction in this model is linear.

Attention here plays the role it plays in ToxCompl+ — re-weighting latent factors — not the role it plays in a transformer. That’s consistent with the program’s style, and it’s a defensible design. But it means the model can’t represent “gene A’s effect on gene B depends on gene C,” which is what pathway biology is made of.

✅ Confirmed against Figure 1. The figure shows (B,n) → [n×s] → (B,s) → [1×r] → (B,s,r) → L stacked layers. The transformer stack operates on (B, s, r) = 512 slots × 16 dims. Genes are gone before layer 1. This is no longer an inference — it’s what the figure draws.

7.6.3 The first attention layer is provably inert

Step [2] maps $(B,s)\to (B,s,r)$ via a $1\times r$ projection. Read literally — and Figure 1 labels the operation once — slot $j$’s token is

$${\text{token}}_j=h_j\cdot \mathbf{v},\mathbf{v}\in {\mathbb{R}}^{16}\text{ shared across all }s\text{ slots.}$$

Every token is then a scalar multiple of the same vector. Verified numerically in code/demo_transtissueformer.py:

Why: $Q_i=W_Q(h_i\mathbf{v})$, $K_j=W_K(h_j\mathbf{v})$, so $S_{ij}\propto h_i{h}_j({\mathbf{v}}^{\top }{W}_Q^{\top }{W}_K\mathbf{v})$ — an outer product, rank 1. A rank-1 score matrix means every query attends with the same pattern up to a scalar. Block 1’s attention contributes nothing.

But this is not fatal, and the distinction matters. The MLP’s nonlinearity breaks the symmetry:

So the model recovers. The cost is waste, not incorrectness:

• block 1’s attention is provably doing nothing
• the model spends its first block manufacturing diversity that a slot embedding would supply for free at step 0
• with $r=16$ there’s very little room to manufacture it in

Say it precisely. “The architecture is broken” is wrong. “The first attention layer is provably inert, and the fix costs 0.18% of the parameters” is right, and it’s a better thing to say.

⚠️ Check the implementation before acting. Figure 1 draws one trapezoid per slot but labels the operation once. If the $1\times r$ is actually per-slot, this is already handled and there’s nothing to fix. The paper alone can’t settle it.

Also: $r=16$ per token, against BERT’s 768. That’s a very small representation to run 32 layers over.

7.6.4 The gaps that follow from everything above

7.7 What to do — actionable, ordered by effort

7.7.1 Initialize the bottleneck from scGPT ⭐⭐ — the best idea in this document

The observation:

• TransTissueFormer’s bottleneck is $W\in {\mathbb{R}}^{n\times 512}$. Row $g$ is a 512-dim embedding of gene $g$ (§7.3).
• scGPT’s gene token embeddings are 512-dimensional (confirmed: scGPT’s $d_{\text{model}}=512$, and gene embeddings are 512-dim vectors from its encoder layer).

$$\boxed{\text{Same shape. }W\in {\mathbb{R}}^{n\times 512}\text{ and scGPT’s gene embedding table }E\in {\mathbb{R}}^{|V|\times 512}.}$$

So you can drop scGPT’s gene embeddings straight in as the initialization of $W$. No adaptation layer. No dimension mismatch. No architecture change. Map rat genes → human orthologs → look up → initialize.

Why this is the right way in, and not a hack:

1. It initializes 96.6% of the model. Not a side input, not an auxiliary loss — the overwhelming majority of the parameters, currently random, on 425 samples.
2. It completely dodges the type error. The fold-change/absolute-count incompatibility (01_BACKGROUND.md §4.3) lives in scGPT’s value encoder. This uses only the gene embeddings. Gene embeddings never touch expression values — they encode which genes are functionally related, which is modality-independent and largely species-conserved.
3. It’s the answer to the future-work paragraph. TransTissue §7 says these models “may nevertheless be finetuned… We plan to explore the adaptation of these models in future work.” The adaptation is one tensor load.
4. Both outcomes are publishable. If it helps, that’s the FM bridge. If it doesn’t — consistent with Kedzierska, Souza & Mehta, and “one PCA still rules them all” — it converts a hedge into “we tested it; here’s the evidence.”

The ablation, designed so it can’t fail to inform:

That last row is the interesting one. Rat P450s — precisely the genes toxicology cares about — have no clean human orthologs. If ESM2’s advantage concentrates in the non-ortholog genes, that’s a mechanistic result, not a leaderboard bump, and it’s exactly what the theory predicts.

Effort: days. This should be experiment #1.

7.7.2 Add a target-tissue embedding

One vector ${\mathbf{e}}_{\tau }\in {\mathbb{R}}^{512}$, added to $\mathbf{h}$ after the bottleneck:

$$\mathbf{h}=W^{\top }\mathbf{x}+{\mathbf{e}}_{\tau }$$

This fixes the ρ=0.23 multi-task collapse by construction — the conditional-mean argument (16_MATH_NOTES.md §4.2) no longer applies once the target is specified. And it’s the prerequisite for zero-shot (Chapter 12 §12.4).

Effort: a day. Risk: at 425 pairs the conditioning may be swamped. Untested.

7.7.3 Tag the synthetic data

We know — provably — that the augmented pretraining data is exactly affine and nearly rank-2. Tagged back-translation (Caswell et al. 2019) is the way to use synthetic data without absorbing its artifacts:

$$\mathbf{h}=W^{\top }\mathbf{x}+{\mathbf{e}}_{\tau }+{\mathbf{e}}_{\text{synthetic}}$$

Set it to the “real” tag at fine-tuning. Effort: hours.

7.7.4 Noise the augmentation

Funk-SVD output is noiselessly linear — the extreme of the pathology Edunov et al. (2018) identified. Sample from a probabilistic factorization’s posterior instead of taking the point estimate, so each epoch sees a different draw.

Prediction: noised augmentation beats clean augmentation on real held-out data while scoring worse on augmented data. Effort: days.

7.7.5 Add a slot embedding

$${\text{token}}_j=h_j\cdot \mathbf{v}+{\mathbf{p}}_j,P\in {\mathbb{R}}^{s\times r}$$

Verified effect (code/demo_transtissueformer.py):

Cost: $512\times 16=8,192$ parameters — 0.18% of the model. Benefit: block 1’s attention starts working.

This is a positional encoding — but over latent slots, not genes, so it doesn’t violate the (correct) no-gene-order principle. Check the implementation first (§7.6.3). Effort: hours.

7.7.6 Raise $r$, lower $s$

$r=16$ is tiny; the transformer has 3.4% of the parameters. Rebalance:

The third row is a real transformer. And attention gets cheaper — $s^2$ shrinks faster than $r$ grows. This is a free axis nobody has swept.

Hypothesis: if performance is flat across this sweep, the transformer isn’t contributing, and the honest model is a low-rank linear map plus a small MLP — consistent with everything in 10_SOTA_LANDSCAPE.md §3.

Effort: a sweep. Value: high either way.

7.7.7 Report the metrics that discriminate

• row-wise PCC — undefined for a mean predictor, so it’s the metric that exposes one (16_MATH_NOTES.md §3.5). GenTox §2.3 argues for it from first principles; TransTissue doesn’t report it.
• macro-F1 over the 5 categories — the 92%/8% split is a classification problem dressed as regression (Chapter 12 §12.8b). The all-zeros predictor can’t game macro-F1.
• enrichment-consistency — do the predicted and true profiles support the same biological conclusions? (15_FRONTIER.md F6.) The mean predictor scores zero here by construction: identical enrichment for every treatment means no discriminative power.

Effort: a week. This is the highest-value non-modelling contribution available.

7.8 The upgrade path

Each step composes with the last. Nothing here requires abandoning the architecture.

v0  TransTissueFormer as published
      32 separate models, random init, column-wise PCC

v1  + scGPT/UCE bottleneck init          [7.7.1]  ← 96.6% of params, days
    + row-wise PCC, macro-F1             [7.7.7]  ← the metrics that discriminate
    + slot embedding if needed           [7.7.5]

v2  + target-tissue embedding            [7.7.2]  ← one model, all pairs
    + tagged synthetic data              [7.7.3]  ← quarantine the linearity artifact
    + noised augmentation                [7.7.4]

v3  + tissue adapters                    [Ch 12 §12.7]  ← shared trunk, per-organ capacity
    + compound embedding (GenTox's GNN)  ← unifies GenTox and TransTissueFormer
    + unpaired profiles via denoising    ← the 1,249 liver profiles on the floor

v4  + zero-shot to the 24 empty pairs    [Ch 12 §12.4]
    + off-target detection               [Ch 12 §12.4]  ← "is this actually a kidney?"
    + unsupervised alignment             [Ch 12 §12.5]  ← the bet

v1 is days of work and touches 96.6% of the parameters. Start there.

7.9 The honest summary

What’s right. The compute argument is unanswerable — vanilla attention on 31,042 genes needs about 2 TB. Refusing positional encoding is correct. Parallel decoding is correct. Full-profile context is correct. The biological validation (§6 of the paper: gemfibrozil→PPARα, cisplatin→TP53, lead→p53) is worth more than every PCC in the four papers, and it’s the part that deserves to be automated and scaled.

What’s overstated. The transformer is 3.4% of the model. There’s no gene-gene attention — gene interactions are entirely linear. The 6% MAE gap over an MLP is what you’d expect from two mostly-linear models. And Figure 7’s pretrain row scores a nearly-linear model on a provably-linear task, which is a matched filter, not a finding.

What’s missing. Tissue conditioning. Compound representation. The unpaired profiles. Parameter sharing across pairs. Row-wise metrics. And a pretrained gene embedding table of exactly the right shape, sitting in a public checkpoint.

The one-sentence version:

TransTissueFormer is a 4.4M-parameter gene embedding table with a small transformer attached, trained from random initialization on 425 examples — and scGPT ships a pretrained gene embedding table with precisely matching dimensions, whose fold-change incompatibility lives entirely in a value encoder this architecture doesn’t have.

Parameter counts computed from the paper’s stated hyperparameters; decoder depth is ambiguous and assumed shallow. Architecture read from the paper’s prose and figure — §7.6.2 and §7.6.3 in particular should be checked against the implementation. The linearity result is verified in code/experiments/exp02_rank_test.py. scGPT’s 512-dim gene embeddings verified against its documentation.

Chapter 8 — The Augmentation: What Is Done, Why, and What Needs Doing On Top

Walked through on the paper’s own Figure 4 example, with real numbers. Run code/demo_augmentation.py alongside — it prints every number here.

8.1 The problem, in one table

We want a liver → kidney translation model. We need treatments where both organs were measured.

Here is the paper’s own toy: 4 tissues, 7 treatments.

        1    2    3    4    5    6    7
  LI   ##   ##   ..   ##   ..   ##   ..
  BR   ##   ##   ..   ..   ##   ##   ##
  KI   ##   ##   ..   ..   ..   ##   ..
  HE   ..   ##   ##   ##   ..   ##   ..

       ## = measured      .. = missing

LI measured on:  1, 2, 4, 6
KI measured on:  1, 2, 6
─────────────────────────────
LI–KI PAIRS:     1, 2, 6        ← THREE. That's the whole training set.

At real scale that number is 425, against 8,565-dimensional inputs and outputs. 88% of DrugMatrix is empty.

Section 5 exists to fix this.

(The sparsity pattern above is reconstructed from the paper’s own arithmetic — see §8.5. It reproduces their stated 43% figure exactly, which is how we know it’s right.)

8.2 The idea: manufacture the missing data

The paper’s reasoning, and it’s sound:

“The scarcity of our data samples for translation is due to many of the missing endpoints in DrugMatrix. Had all entries contained observed measurement values, for all cross-tissue translation models there would be orders of magnitude more training samples.”

So: fill in the holes, then train on the filled-in matrix.

Why not the usual augmentation tricks?

They’re right to rule them out, and the reasoning is worth understanding:

“Unfortunately none of the existing augmentation techniques are suitable in training cross-tissue translation models.”