Walkthrough — One Matrix, Four Axes

Written for: someone with a transformers/multilinguality background and no biology. Core claim of this document: you already know most of this. It’s machine translation with low-resource language pairs. The vocabulary is different; the problems are the same.

Part 0: The one-paragraph version

the ORNL–NIEHS program has one data matrix and four papers that each attack a different axis of it.

The matrix is called DrugMatrix. Rows = genes (~375,000 of them, because the same gene appears once per tissue per measurement platform). Columns = treatments (~2,700–3,000; a “treatment” is a (chemical, dose, duration) triple). Each cell is a number: how much did this gene’s activity change in this tissue under this treatment, relative to an untreated control animal.

88% of the matrix is empty. Nobody ran every drug on every tissue on every platform. That single fact is what all four papers are about.

Read them in that order. ToxCompl+ is the foundation; the other three are built on its ideas.

Part 1: The biology, from zero

You need about six concepts. That’s genuinely it.

1.1 Genes and expression

Every cell in a rat contains the same DNA — the full instruction set, ~20,000-ish genes. But a liver cell and a brain cell behave completely differently. Why? Because they use different subsets of those instructions at different intensities.

The mechanism: DNA gets transcribed into mRNA, and mRNA gets translated into protein. Protein is what actually does things. So how much mRNA exists for gene X right now is a decent proxy for “how hard is the cell running instruction X.”

That quantity is called gene expression. Measure it for all ~20,000 genes at once and you get a transcriptomic profile — a vector in R^20000.

NLP intuition: think of a profile as a dense document embedding of the cell’s current state. Except each dimension is named and interpretable — dimension 4,412 is the gene Cyp1a1, and it means something specific. You have a 20,000-dimensional vector where every axis has a biological meaning. That interpretability is why biologists care about individual dimensions in a way NLP people usually don’t.

1.2 Fold change — the single most important preprocessing fact

You could report raw expression: “there were 5,000 copies of Cyp1a1 mRNA.” But raw numbers are dominated by boring stuff — liver cells always express liver genes highly, and that swamps any drug effect.

So DrugMatrix reports log10 fold change:

value = log10( expression_in_treated_animals / expression_in_control_animals )

• 0 → the drug did nothing to this gene
• +1 → gene went up 10×
• -1 → gene went down 10×

This differencing step is doing enormous work, and you must hold onto it:

1. It’s why cross-platform comparison is even possible. CodeLink measures fluorescence brightness; BioSpyder counts sequencing reads. Totally incompatible units. But a ratio is unitless. Dividing by the control cancels the platform out — partially. TransPlatformer exists because it only cancels partially.
2. It’s why single-cell foundation models don’t plug in. scGPT eats binned absolute expression. DrugMatrix is already-differenced ratios. There is no sensible way to feed a fold-change into scGPT’s value encoder. This is the concrete, technical reason “just fine-tune scGPT” is not a five-minute job. Remember this — it comes back in Part 5.
3. It’s why 92% of the matrix is ≈ 0. Most drugs don’t touch most genes. The signal is sparse.

1.3 The 92% problem

From Table 1 of TransTissue:

92% of every profile is noise around zero. The ~4% that isn’t zero is the entire biological content.

NLP intuition: this is a severe class imbalance problem wearing a regression costume. It’s why every one of these papers reports “rare MAE” separately from overall MAE. A model that outputs all zeros gets a great overall MAE and is completely worthless. Keep asking: what does the all-zeros predictor score?

1.4 Platforms (the “dialects”)

Three generations of measurement technology, all present in DrugMatrix:

Microarrays: little DNA probes stuck to a chip, your sample sticks to matching probes, you measure how bright each spot glows. Analog, saturates, noisy background.

Sequencing: you literally read the mRNA letter by letter and count how many reads map to each gene. Digital, huge dynamic range.

BioSpyder S1500+ has a twist: it only physically measures ~2,700 well-chosen “landmark” genes, then a tool called GeniE extrapolates to the full ~20,000. So a chunk of “BioSpyderWT” data is itself a model prediction. (Worth filing away. It’s a soft spot nobody in these papers dwells on.)

NLP intuition: same language, three different transcription conventions with different phoneme inventories and different amounts of transcription error. TransPlatformer is a normalizer.

1.5 Tissues (the “languages”)

Eight organs: liver (LI), kidney (KI), heart (HE), bone marrow (BM), thigh muscle (TM), spleen (SP), intestine (IN), brain (BR).

Coverage, in measured endpoints (TransTissue Table 2):

Platform       LI     KI     HE     BM     TM     BR      IN      SP
CodeLink     14.2M   7.7M   5.3M   0.2M   2.7M   0.55M   0.17M   1.5M
Affymetrix   20.3M  11.3M   6.5M   1.3M     0      0       0       0
BioSpyderWT  17.6M  15.9M  10.7M   6.5M   0.09M  0.07M   0.5M    1.6M

Liver is enormous. Brain and intestine are almost nothing. Why? Because liver is where drugs get metabolized, so toxicologists always look there. Brain is hard to sample and rarely the primary target.

NLP intuition: liver is English. It’s the pivot. It’s over-represented in every corpus because that’s where the field’s attention has been. Brain and intestine are your genuinely low-resource languages — and, exactly as in NMT, they’re the ones you actually want to translate into, because getting real data for them is expensive and invasive.

1.6 Why cross-tissue translation is a sane idea at all

You dose a rat with a compound. It goes everywhere in the bloodstream. Liver, kidney, heart, brain all see it. Each organ responds — differently, but they’re all responding to the same systemic event.

So there is genuinely shared latent structure: a common “the animal was poisoned” signal, plus organ-specific programs. If you can measure liver (easy) and infer kidney (harder), you save an animal, a biopsy, and a lot of money.

Whether that inference is actually possible is an open question, and — read this carefully — the TransTissue paper does not claim to have settled it. Section 5, verbatim: “it is possible that there are simply no (or sufficient) signals for cross-tissue translation.” That honesty is a feature. Don’t let anyone, including you, oversell the result.

Part 2: The shape of the data (internalize this and everything else follows)

                    treatments (m ≈ 2,700–3,000)
                    ─────────────────────────────►
                    (chemical, dose, duration)

    g   ┌───────────────────────────────────────┐
    e   │  CodeLink   × LI  ░░░░████████░░░░██  │  ← 8,565 rows
    n   │  CodeLink   × KI  ░░░████░░░░░░████░  │
    e   │  CodeLink   × BR  ░░░░░░░░░░█░░░░░░░  │  ← almost empty
    s   │  ...                                   │
        │  Affymetrix × LI  ██████░░░████████░  │  ← 31,042 rows
    n   │  ...                                   │
    ≈   │  BioSpyder  × LI  ░░████████░░░░░░██  │  ← 22,794 rows
   375k │  ...                                   │
        └───────────────────────────────────────┘
              █ = measured (12%)   ░ = missing (88%)

A row is (platform, tissue, gene). That’s why there are 375,000 of them and not 20,000 — the same gene Cyp1a1 appears as a separate row for every (platform, tissue) combination.

Now the four papers are just four different questions about this picture:

• ToxCompl+: fill in the ░ cells. Anywhere.
• TransPlatformer: given a CodeLink row-block, produce the BioSpyderWT row-block. Same column.
• TransTissueFormer: given the LI row-block, produce the KI row-block. Same column.
• GenTox: here’s a brand-new column (a drug never tested). Produce it from scratch.

That’s the whole research program.

Part 3: The four papers

3.1 ToxCompl / ToxCompl+ — “Improved Completion of DrugMatrix…”

Question: 88% of the matrix is missing. Can we just… fill it in?

Method — start with Funk-SVD. This is the Netflix Prize algorithm, and the analogy is exact:

Assume the matrix is low-rank. Factor M ≈ P × Q where P is (n_genes × r) and Q is (r × n_treatments), with r = 300. Fit by SGD on observed entries only:

min Σ_{i,j observed}  ( M[i,j] − ( b_i + b_j + P[i,:] · Q[:,j] ) )²

b_i, b_j are per-gene and per-treatment bias terms. Adam, lr=1e-3.

Why low-rank is a reasonable assumption (their argument, and it’s a good one): 3,000 columns but only 636 distinct drugs. Doxorubicin and Epirubicin are nearly the same molecule and do nearly the same thing. Genes come in co-regulated modules. The same gene is measured in 8 organs. There’s massive redundancy. Rank 300 out of 3,000 is plausible.

What went wrong. They added the new BioSpyderWT data, doubling the matrix (→ “DSMatrix”, 375k × 3k). Mean absolute error improved (0.05 → 0.03). But maximum absolute error got worse (1.58 → 3.99). And here’s the killer detail:

at that worst point: target = −0.77, prediction = +0.94

The gene was suppressed. The model said it was activated. The sign flipped. In toxicology that’s not a small error, it’s the opposite conclusion. More data made the average better and the tails worse — and the tails are the whole point (remember §1.3). Raising r to 500 and training longer made MAE better and MaxAE worse still. That’s the low-rank assumption cracking: rare extreme signals are exactly the part of the matrix that isn’t low-rank.

ToxCompl+’s two fixes:

(a) Side information. Rows have features (platform, marker, organ), columns have features (drug, duration, dosage). Embed each into R^300, add their interactions:

M[i,j] ≈ b_i + b_j + P[i,:]·Q[:,j] + Σ_t P[i,:]·C^t[:,j] + Σ_t R^t[:,i]·Q[j,:]

Result: MaxAE 3.99 → 3.27, but MAE went 0.03 → 0.05. A trade, not a win.

(b) Attention-augmented aggregation. This is their signature move — it recurs in GenTox, so learn it once here. Vanilla MF predicts a plain dot product P[i,:] · Q[:,j], which weights all r=300 latent dimensions equally. Instead, learn two extra factor matrices P' and Q', and let them decide how much each latent dimension matters for this particular cell:

M[i,j] ≈ b_i + b_j + ( P[i,:] ∘ Q[:,j] ) · σ( P'[i,:] ∘ Q'[:,j] )

∘ is Hadamard (elementwise) product, σ is softmax.

NLP intuition: P ∘ Q is your value vector; σ(P' ∘ Q') is a learned, input-dependent attention distribution over latent dimensions; the dot product is the weighted sum. It’s attention with r “positions” and a single query. Small idea, and it works: MaxAE 3.27 → 0.83, MAE → 0.02.

A fair question to hold: is the win from attention specifically, or from doubling the parameters? They anticipate this — they note that raising r from 300→500 (also more parameters) hurt, which is decent evidence the nonlinearity is doing real work. Not airtight, but decent. It’s a good thing to have an opinion about.

Take-away: ToxCompl+ is the engine. TransTissueFormer’s data augmentation is ToxCompl. Everything else sits on this.

3.2 TransPlatformer — “Translating Toxicogenomic Profiles Between Generations of Platforms”

Question: decades of legacy CodeLink and Affymetrix data exist. Modern work is BioSpyderWT. Can we translate the old to the new and reuse it?

Why they rejected Seq2Seq (their §2.2 — this argument recurs in TransTissue and you should be able to state it cold):

1. Compute. Self-attention is O(n²) and n = 31,042 for Affymetrix. That’s a ~10⁹ attention matrix. Dead on arrival.
2. Genes aren’t a sequence. In language, tokens 1..s predict token s+1 — there’s a real ordering. Genes have no meaningful linear order; they form pathways and networks. Autoregressive left-to-right decoding imposes a structure that doesn’t exist.
3. Profiles are ~90% identical to each other. Two completely different drugs produce profiles sharing >90% of their content (because 92% of both is ≈0). Feed those to a Seq2Seq model as “sentences” and it drowns.

This is the most important intellectual move in the whole program, and it’s the one your background makes you best-placed to evaluate. They are saying: transformers, yes; the NLP transformer’s inductive biases, no. Point 1 is unarguable. Point 2 is right that there’s no linear order — though “not a sequence” doesn’t imply “can’t use attention,” it implies “use it without positional encoding,” which is basically a set transformer, and is closer to what they built than they say. Point 3 is the interesting one and I’d push on it: high baseline similarity between inputs is an argument about signal-to-noise, not about architecture. Worth asking the original authors whether these are three independent arguments or really one (compute) plus two rationalizations.

The architecture (shared with TransTissueFormer, so learn it once):

input (B, n)                      n = 8,565 source genes
    │
    │  bottleneck FC:  n × s      s = 512      ← the key move
    ▼
   (B, s)
    │
    │  projection:  1 × r         r = 16       ← give each of the s slots an embedding
    ▼
   (B, s, r)
    │
    │  L = 32 transformer layers ("TP attention")
    ▼
   (B, s, r)
    │
    │  ⌈n/s⌉ = 17 PARALLEL decoders, one per output segment
    ▼
   (B, n, r)
    │
    │  projection: r × 1
    ▼
   output (B, n)

Three deliberate departures from Vaswani:

1. The n×s bottleneck compresses 8,565 genes into 512 “slots” before attention. Attention then costs O(s²) = O(512²) instead of O(8565²). Same spirit as Linformer/Performer — project the sequence dimension down. Their complexity table: standard O(n²d), Linformer O(nsd) with an O(ns) score matrix, theirs O(nsd) with an O(s²) score matrix.
2. Every output gene attends to the entire input profile. No context window. Any gene in liver can in principle influence any gene in kidney.
3. ⌈n/s⌉ = 17 parallel decoders. Not autoregressive — all 17 output segments emit simultaneously. This is what kills the O(n) sequential decode.

NLP intuition: it’s Perceiver-shaped. Cross-attend a huge input into a small latent array, do the work in latent space, broadcast back out. If you’ve read Perceiver IO, you already know this architecture. That’s a genuinely useful connection to bring up with the original authors — the papers cite Linformer and Performer but not Perceiver, and Perceiver IO is arguably the closer relative.

Results: mixed-tissue mode, MAE 0.043 vs 0.09 baseline (>50% reduction), PCC ≈0.71 vs ≈0.37 (doubled), rare-signal MAE <0.22. Downstream: they used it to translate legacy data into BioSpyderWT format, added it to a liver-necrosis classifier’s training set, and got ~8% F1 improvement. That downstream number is the most convincing result in the entire four-paper set — it’s the only one where a translated profile demonstrably helped a task someone actually cares about. Note it.

Bonus: attention weights are interpretable. Translating CodeLink→BioSpyderWT, the model attends heavily to Cmya1, Ca3, Cyp1a1, Ctsh, Sds, Cited4, Atf3, Lcn2, Stac3. Several of those (Cyp1a1 — drug metabolism; Atf3 — stress response; Lcn2 — injury marker) are well-known toxicology genes. That’s a nice sanity check that it learned something real.

3.3 TransTissueFormer — “Translating Transcriptomic Profiles Between Tissues”

Question: given the liver profile for a treatment, predict the kidney profile for the same treatment.

Same architecture as TransPlatformer. The paper is about data scarcity, not architecture.

The pair counts (CodeLink, Table 3) — this table is the paper:

        BM   BR   HE   IN   KI    LI   SP   TM
   BM  325    0   24    0  158   223  161    0
   BR    0   65   19    0    3     0    0    0
   HE   24   19  629    4  159   176   14    7
   IN    0    0    4   20    0     0    0    0
   KI  158    3  159    0  906   425   84   24
   LI  223    0  176    0  425  1674  164   28
   SP  161    0   14    0   84   164  180    0
   TM    0    0    7    0   24    28    0   29

NLP intuition — this is a parallel corpus size table and you have seen it a hundred times. LI–KI has 425 sentence pairs. That’s it. BR–LI has zero — a language pair with no parallel data at all. Of 64 possible directions, only 40 have any data, and only 32 are genuinely cross-tissue.

425 examples. Your models have 8,565-dimensional inputs and outputs. Sit with that for a second. The entire paper is a response to that number.

Direct training results:

TransTissueFormer wins, but PCC 0.53 is weak-to-moderate. Even LI–KI, the best-resourced pair, only hits 0.40. The paper says so plainly: “no definite conclusion can be drawn whether cross-tissue translation is possible.”

The interesting biology: HE–TM gets PCC 0.7 on only 7 pairs. LI–KI gets 0.4 on 425 pairs. Data volume isn’t destiny — biological relatedness matters more. Heart and thigh muscle are both striated muscle, both mitochondria-dense, both oxidative-phosphorylation-dependent — so they respond similarly. Liver and kidney both handle xenobiotics but by different mechanisms (liver = Phase I/II metabolism, cytochrome P450; kidney = transporter-mediated uptake, tubular stress), so they decouple.

NLP intuition: Spanish→Portuguese works on tiny corpora. English→Japanese doesn’t work on large ones. Typological distance beats corpus size. Same phenomenon, and it’s the single best cross-domain insight in the paper. There’s a real research question hiding here: can you predict translation quality from a tissue-relatedness prior? That’s “language similarity for transfer selection,” a well-studied problem in your own field.

The main contribution — AugmentTrain&Transfer (Algorithm 1):

1: split treatments m into train set a, test set b
2: F ← G − G[t, b]              # hold out TARGET rows at TEST columns
3: G' ← Funk_SVD(F)             # ToxCompl! impute the whole matrix
4: G' ← G'[:, s ∪ t]            # keep source+target tissue rows
5: split G' column-wise → train / validation
6: train model M on augmented data
7: fine-tune M on real measured data G[s ∪ t, a]
8: test M on real measured data G[s ∪ t, b]

NLP intuition: this is back-translation. Real parallel data is tiny (425 pairs), so manufacture synthetic parallel data (2,711 pairs for HE–TM, up from 7), pretrain on synthetic, fine-tune on real. Sennrich et al. 2016 — which they cite, in the augmentation related-work, as [36]. The connection is right there.

Two subtleties they’re proud of, and you should understand both:

• Why impute the whole matrix, not just the LI/KI sub-matrix? Because a heart profile for treatment 3 helps you reconstruct the liver and kidney profiles for treatment 3. Restricting to LI/KI would lose 43% of the recoverable pairs in their toy example. NLP analogy: multilingual back-translation — pivot through a third language.
• Why hold out only the target and not the source? If you remove both, the test column is empty, and matrix completion has nothing to anchor on.

Results with augmentation: MAE 0.081→0.059, rare MAE 0.27→0.159, PCC 0.53→0.793. Improvements of 38.3% / 69.1% / 50.1%. Nearly every tissue pair improves. LI→SP hits 0.87.

The multi-task failure — and why you specifically should care. They tried one model translating LI→everything. PCC collapsed to 0.23 (vs 0.53 single-task). Their diagnosis, verbatim:

“given the same source LI profile the model cannot decide which is the correct target”

Anyone from NMT will recognise this immediately. Same liver input, three valid outputs (kidney, heart, spleen) — the model averages them and produces mush. This is precisely the problem multilingual NMT hit and solved in 2016: you have to tell the model which language to decode into. Johnson et al.’s multilingual NMT prepends a <2es> target-language token; mBART and M2M-100 do the same. Add a target tissue embedding — one token, prepended or added to the bottleneck — and the ambiguity is gone by construction.

Formally (16_MATH_NOTES.md §4.2): under squared loss the optimal single-valued $h({\mathbf{s}}_c)$ is the conditional mean $\mathbb{E}[{\mathbf{t}}_c\mid {\mathbf{s}}_c]$ — the average over whichever organs happen to be present for that treatment. The model is behaving optimally for a mis-specified problem. Mush is the correct answer to an ill-posed question.

Two caveats worth keeping attached. This has not been verified to fix the number, and there are reasons it might not: 425 pairs is very little, and the conditioning signal could be swamped entirely. But the stated diagnosis is a textbook conditioning bug with a textbook fix, and the paper reports it as a negative result rather than as a bug. That distinction is the whole point — it reclassifies a finding as an artifact, and the fix costs one embedding.

Biological validation (§6) — the best part of the paper. They took 44 predicted kidney profiles and asked a toxicologist to check them blind:

• Gemfibrozil (a PPARα activator): the top enriched gene set in the predicted kidney profile was PPARα activation, with fatty-acid-metabolism pathways dominating. Correct.
• Cisplatin (nephrotoxic chemotherapy): predicted profile showed TP53 enrichment (genotoxicity) and matched a real published cisplatin-kidney-damage signature. Correct.
• Lead acetate: showed cell cycle and p53, and enriched against Lead (IV) acetate in curated DrugMatrix gene sets. Initially looked like a miss; on inspection, both lead species do the same thing. Correct.

This is worth more than any PCC. It’s the difference between “the numbers went up” and “a domain expert looked at the output and said yes, that’s what lead poisoning looks like.” When you evaluate the rest of this program, this is the bar. It’s also, notably, only 3 compounds reported out of 44 — a fair question is what the other 41 looked like.

3.4 GenTox — “Predictive Transcriptomics with Attention-Augmented Inductive Matrix Factorization…”

Question: predict the profile for a compound never tested. This is the hardest axis — a genuinely new column.

This paper opens with theory, and the theory is the most valuable thing in the entire four-paper set. Read §2 even if you skip everything else.

Theorem 1 — Pearson correlation is a liar (on absolute expression)

Let u = s + δ₁, v = s + δ₂, where s is a shared base vector and δ₁, δ₂ are sparse perturbations with k = o(n) nonzero entries. Then ρ(u,v) → 1 as n → ∞.

In English: take two completely unrelated drug profiles. They share the base “this is a liver cell” signal s. Each has a small, sparse drug-specific perturbation. Their correlation goes to 1 — asymptotically perfect — no matter what the drugs are. Because the shared baseline dominates and the sparse differences vanish as n grows.

Their empirical demonstration is brutal, and it’s their own baseline model:

A model reporting PCC 0.992 that has learned essentially nothing (row-wise correlation is 0.008 — indistinguishable from noise). This is a real, trained MLP, and it looks spectacular by the standard metric.

NLP intuition: BLEU on a task where every reference shares 92% of its tokens. You’d score 0.95 by copying the input. This theorem is the toxicogenomics version of “your metric has a trivial-baseline problem,” proved rather than asserted. Your group has proven that the field’s default metric is broken. That is a strong, citable, defensible position and you should know it cold.

Theorem 2 — but fold-change is different

Let u = t (sparse, k nonzero), v = t + η with η ~ N(0, σ²Iₙ). Then ρ(u,v) → 0 in probability.

Because fold-change already subtracted the baseline, there’s no shared s to inflate things. Now correlation is governed by SNR = ‖t‖²/(σ²n). With large n and constant noise, correlation is crushed.

So the two theorems cut opposite ways, and this is the subtle bit:

• On absolute data (TG-GATEs intensity): PCC is inflated → high PCC is meaningless.
• On fold-change data (DrugMatrix): PCC is suppressed → high PCC is hard-won and meaningful.

This partially defends TransTissueFormer’s PCC 0.793 — DrugMatrix is fold-change, so Theorem 2’s regime applies, and 0.793 against a headwind is a real number. It does not defend it against a mean-predictor baseline, which is a different failure mode neither theorem covers (Theorem 2 assumes i.i.d. Gaussian noise; a mean predictor’s error is structured, not Gaussian). Hold that distinction — it’s the sharpest question in Part 6.

The rank-degeneracy argument

Consider a model that outputs x = αⱼ · y for ground truth y, with a different scalar αⱼ per column. Every column correlates perfectly, ρ = 1. But now look at rows — gene i₁ across all treatments vs gene i₂ across all treatments. Each row got multiplied by a different, arbitrary αⱼ per column. Row-wise structure is destroyed.

Why that’s fatal: gene network analysis — the thing biologists actually want — needs rows. It builds a graph where nodes are genes and edges are co-expression across treatments, then finds hub genes and modules. A model that’s perfect column-wise and garbage row-wise produces profiles that look great and are useless for the downstream science.

So: always report row-wise AND column-wise metrics. Their proposed metric suite: MAE, rare MAE, row-wise PCC, column-wise PCC, with MAD as a data characteristic.

The GenTox method

Three components:

(a) Deep inductive matrix factorization. Vanilla MF can’t handle a new column — Q[:,j] doesn’t exist for an untested drug. So replace the lookup tables with networks over features:

      row features (gene, tissue, platform) ──► Row NN ──┐
                                                          ├──► Mixer NN ──► Ĝ[i,j]
      col features (compound, dose, duration) ─► Col NN ──┘

Sample random (rᵢ, cⱼ) cells across the whole matrix each minibatch — that’s what forces row and column structure to be preserved (directly addressing §2.3). Degenerates to vanilla MF if the Mixer is a dot product and dims match. Now a new drug is just a new feature vector → the Col NN produces its factor. Inductive, not transductive.

(b) Contrastive learning for the induction basis. How do you featurize a molecule? Classical answer: Mordred descriptors (1,826 hand-crafted physicochemical numbers) or Morgan/ECFP fingerprints (substructure bit vectors). Their answer: pretrain a GNN on ~1M compounds (in-house) and use its learned embedding.

Two variants:

• Positive/negative contrastive: augment each molecular graph twice (atom masking + bond deletion, carefully avoiding disconnection), same molecule = positive pair, different = negative, InfoNCE loss.
• Graph InfoMax: maximize mutual information between node embeddings and the graph readout, max_θ Σ_G Σ_v I_θ(z_v, z_G), via a discriminator (MI is intractable directly).

Finding: InfoMax > contrastive pairs > Mordred ≈ Morgan. Learned representations beat hand-crafted ones. Encoder is a 2-layer GCN, hidden 64, output 64×300.

NLP intuition: this is word2vec vs. one-hot, or BERT vs. bag-of-words, for molecules. And note — this is already a foundation model. Pretrained on 1M compounds, self-supervised, frozen, used as a feature extractor downstream. the ORNL–NIEHS program has already done the thing the TransTissue future-work paragraph proposes to explore. They did it on the chemistry side. Nobody’s done it on the gene side. That asymmetry is your opening — see Part 6.

(c) Attention-based aggregation. The same (P ∘ Q) · σ(P' ∘ Q') from ToxCompl+. Ablation: attention > plain.

Data: Open TG-GATEs, in vivo rat liver, 6,766 samples / 2,238 distinct treatments / 139 compounds, 3 doses × 4 durations (3/7/14/28 days), 8:2 split, Affymetrix 31,099 probes.

Note the caveat: the paper is a draft. Broken Figure ?? refs, [?] citations, and §5 (Gene Network Analysis) and §6 (Out-of-distribution validation) are empty section headers. The gene network analysis is the thing §2.3 spends three pages arguing is essential — and it isn’t done yet. That’s not a criticism, that’s an opportunity. If you’re looking for a place to contribute immediately, §5 is an unfilled hole in a paper that already argues for why it must be filled. Ask about it.

Part 4: The NLP ↔ toxicogenomics dictionary

Keep this open while you read. It’s the highest-leverage thing in this document for you specifically.

Part 5: Single-cell foundation models — background and current state

Everything here is the “other half” — the literature the original authors’ TransTissue future-work paragraph gestures at. My reliable knowledge runs to ~mid-2025; it is now July 2026. Treat the “foreground” section as a starting point to verify, not gospel. I can search for what’s landed since.

5.1 The other measurement revolution: single-cell

Everything above is bulk transcriptomics: grind up a whole liver, measure average expression. You get one vector per sample — an average over millions of cells of dozens of types.

Single-cell RNA-seq (scRNA-seq) measures each cell individually. Instead of one vector per liver, you get 10,000 vectors, one per cell. You can see that hepatocytes did X while immune cells did Y — information that bulk averages away.

The cost: data per cell is terrible. You detect maybe 1,000–5,000 genes out of 20,000 in any given cell. The rest are zeros — and you can’t tell “not expressed” from “we missed it” (dropout). So scRNA-seq is many observations, each very noisy and sparse.

NLP intuition: bulk = document-level embeddings. Single-cell = token-level, but every token is 70% masked and you don’t know which. You trade precision for granularity and count.

5.2 Why “foundation model” happened here

Millions of public scRNA-seq cells + self-supervised objectives + transformers = the obvious play. The pitch is exactly BERT’s: pretrain on unlabeled cells at scale, learn “the language of biology,” fine-tune on your small labeled task.

The cell-as-sentence metaphor drives all of it:

The load-bearing weirdness: there is no word order. A cell is a set of (gene, value) pairs. So every one of these models is really a set transformer with no positional encoding, and the hard design problem is how to encode the value — because “gene X is present” is much less informative than “gene X is present at level 7.”

Each model answers that differently, and that’s the main axis of variation:

• Geneformer (Theodoris et al., Nature 2023, ~30M cells): rank-value encoding. Sort genes by expression, feed the ranked list as tokens. The value becomes position in the ranking. Clever — it’s normalization-free and robust across batches. But it throws away magnitude: rank 5 vs rank 6 could be a 2× or a 1.01× difference.
• scGPT (Cui et al., Nature Methods 2024, 33M cells): gene token + binned expression value embedding, added together. Generative masked pretraining. The most widely used; fine-tuned for cell-type annotation, perturbation prediction, GRN inference, batch integration. This is [8] in TransTissue.
• scBERT: BERT-ish, gene2vec embeddings, Performer attention for the long gene sequence.
• scFoundation: ~50M cells, an asymmetric encoder-decoder (xTrimoGene) that only encodes non-zero genes — a direct attack on the sparsity problem.
• UCE (Universal Cell Embeddings, Rosen et al.): 36M cells across 8 species. The trick: genes are tokenized by feeding their protein sequence through ESM2 (a protein language model). So a gene’s token is derived from what its protein looks like, not from a fixed vocabulary lookup. Consequence: it can embed any protein-coding gene from any species, including species never seen in training, zero-shot. Hold that thought hard.
• CellFM (Zeng et al., Nature Communications 2025, 100M human cells, ~800M params, ERetNet). This is [49] in TransTissue.
• GeneCompass, scMulan, SATURN and others in the cross-species / multi-omics direction.

NLP intuition: this is 2018–2020 all over again. scBERT is BERT, scGPT is GPT, UCE is XLM-R-with-a-clever-tokenizer, CellFM is “same thing but 3× the data.” The field is speed-running your field’s history. Which means you know what happens next, and roughly in what order. That is not a small advantage.

5.3 Perturbation prediction — the other cited thread

Different question: given an untreated cell and a perturbation, predict the perturbed cell. Directly relevant to the original authors’ program, since that’s what a toxicogenomics profile is.

• CPA (Compositional Perturbation Autoencoder; Lotfollahi et al., Mol Sys Bio 2023) — [27] in TransTissue. The key idea, and it’s beautiful: decompose the latent into additive, disentangled parts:

  z_cell = z_basal + z_drug(compound, dose) + z_covariate(cell type, ...)
  

  Train an adversarial classifier to force z_basal to contain no drug information. Then you can recombine at test time: take a basal state you’ve seen, add a drug embedding you’ve seen, get a combination you’ve never seen. Compositional generalization.

• chemCPA: CPA + a molecular structure encoder → generalize to unseen chemicals.

• GEARS: GNN over a gene-ontology graph → predict unseen genetic perturbations by leaning on the knowledge graph.

• PerturbNet (Yu et al., Mol Sys Bio 2025) — [47]. Perturbation representation → latent → cell state, via normalizing flows. Handles unseen chemical and genetic perturbations.

• PerturbAtlas (NAR 2025) — [51]. ⚠️ This is a database of bulk RNA-seq perturbation datasets, not a generative method. TransTissue miscites it as one (“Perturbation predictions using generative methods (e.g., see [51, 47, 27])”). Small thing, easy fix, and noticing it is a good way to show the original authors you actually read the references.

Look at CPA and then look at ToxCompl. z_basal + z_drug + z_covariate → decoder versus b_i + b_j + P[i,:]·Q[:,j]. Funk-SVD is the linear special case of CPA. this line of work has been building a linear, transductive CPA and calling it matrix completion. That’s not a criticism — it’s the observation that these two literatures are the same idea at different points on the nonlinearity axis, and nobody has said so in print. That connection is worth a paper by itself.

5.4 The critiques — learn these before you get excited

This is the most important subsection in Part 5. The single-cell FM field has a replication problem, and two papers land hard:

1. Kedzierska et al., Genome Biology 2025 — “Zero-shot evaluation reveals limitations of single-cell foundation models.” Evaluated scGPT and Geneformer zero-shot. They are outperformed by simply selecting highly variable genes — a baseline from 2010 that involves no learning whatsoever. Also beaten by scVI and Harmony. The authors’ hypothesis: masked-language-modeling on cells may simply not produce useful cell embeddings.

2. Ahlmann-Eltze, Huber & Anders, Nature Methods 2025 — “Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines.” Benchmarked five foundation models plus two other deep models. For unseen combinatorial perturbations: they don’t beat an additive model. For unseen genes: they don’t beat predicting the mean of the training perturbations. Their hypothesis for why: the pretraining data is observational, not interventional — you can’t learn what happens when you push the system by only watching it sit still.

3. Related: Boiarsky et al. found logistic regression competitive with scGPT/Geneformer for cell-type annotation.

This is the “BERT beats LSTM… wait, does it?” moment, or the whole ELMo-era-reproducibility discourse. You’ve lived through this genre. The lesson transfers exactly: scale is not automatically transfer. 33M cells of observational data may teach you what cells look like without teaching you what drugs do to them.

Strategically, this is very good news for you. It means the honest answer to “why don’t you just use scGPT?” is not a defensive shrug — it’s “because the current best evidence says it wouldn’t help, here are two Nature-family papers, and we tested it anyway and here’s what we found.” Skepticism, properly cited and properly tested, is a stronger position than enthusiasm.

5.5 The foreground (as of ~mid-2025 — verify this)

• Tahoe-100M (Vevo Therapeutics + Arc Institute, Feb 2025): 100M cells, ~1,100 drugs × 50 cancer cell lines, ~60,000 drug-cell combinations. Open source. This is drug perturbation at scale, which is much closer to the original authors’ problem than any cell atlas. This matters and should be on your radar.
• Arc Institute Virtual Cell Atlas, CZI Virtual Cells Platform: infrastructure plays, aggregating datasets and hosting models.
• STATE (Arc Institute): a perturbation-response model trained on the above.
• The field’s stated goal has consolidated around “the virtual cell” — simulate a cell in silico well enough to replace experiments. Enormous funding, real skepticism, unresolved.
• The evaluation crisis is the live debate. After Kedzierska and Ahlmann-Eltze, the field is (rightly) arguing about benchmarks and baselines rather than about parameter counts. Good time to arrive with a rigorous-evaluation mindset.

Things I’d want to check before relying on any of it (I can search):

• What landed at NeurIPS/ICML 2025–2026 on single-cell FMs?
• Did anyone answer Ahlmann-Eltze? Are there FMs that now beat linear baselines?
• Has anyone built a bulk or toxicogenomics foundation model? (As of my cutoff: not really. That’s conspicuous.)
• Is there a rat-specific or cross-species perturbation FM?
• What happened with STATE / the virtual cell push?

Part 6: Where this program is incomplete

The four papers leave three kinds of gap. They’re developed properly elsewhere; this is the map.

Gap 1 — the baselines are missing. None of the four report what predicting zero scores, what predicting the mean target profile scores, or what a plain linear map scores. This is not pedantry: it is the single most active methodological problem in the surrounding field. The Virtual Cell Challenge 2025 ran 1,200+ teams and reported that perturbation models “are not yet consistently outperforming naive baselines across all metrics.” Ahlmann-Eltze et al. (2025) found five foundation models plus two deep models failing to beat additive and mean baselines.

→ 14_RESEARCH_AGENDA.md §A0, with runnable checks in code/.

Gap 2 — the metrics argument didn’t propagate. GenTox §2 proves from first principles that Pearson correlation is unreliable for transcriptomic profiles, and that row-wise and column-wise metrics must both be reported. TransTissue reports column-wise PCC only. The metric that would settle its central claim is one the same program already argued for, in another paper.

This is now urgent rather than merely tidy: “The Metric Picks the Winner” (June 2026) shows model rankings inverting end-to-end with metric choice on drug-response prediction. GenTox was early and has been vindicated by the field’s own crisis — while remaining an unpublished draft with two empty sections.

→ 16_MATH_NOTES.md §3, and 15_FRONTIER.md F6 for what should replace PCC.

Gap 3 — the foundation-model paragraph is a placeholder. TransTissue §7 proposes adapting scGPT and CellFM “in future work.” The obstacle is more concrete than the paragraph admits: scGPT’s value encoder consumes binned absolute expression; DrugMatrix is log fold-change. That’s a type error, not a domain gap, and no amount of fine-tuning fixes it. Naming it precisely turns a weak hedge into a real argument — and points at the actual bridges.

→ 10_SOTA_LANDSCAPE.md for the 2026 state of that literature, 15_FRONTIER.md for the program.

Part 7: Open questions

Questions the four papers raise and don’t answer. Roughly ordered from “needs the original study metadata” to “needs new work.”

On the data

1. How much of BioSpyderWT is GeniE extrapolation rather than direct measurement? Some “measured” data is itself model output, so ToxCompl+ on DSMatrix is partly imputing from imputations. Nobody has quantified the error this injects.
2. What were the actual study panels? Table 3 decomposes exactly into 24 tissue panels (16_MATH_NOTES.md §6), but the decomposition is underdetermined. The original study designs would settle it — and settle whether the missingness is MNAR.
3. What do the other 41 of the 44 biologically-validated kidney profiles look like? Three are reported and all three are correct. The hit rate is unknown.

On the models 4. Why rank exactly $r=300$? Tuned, or inherited? Does it interact with the bottleneck $s=512$? 5. TransTissueFormer uses $r=16$ per slot — very small by NLP standards. Compute-bound, or does something break when it grows? 6. Is the ToxCompl+ attention win from the nonlinearity or from doubling the parameters? The clean ablation is $P^{\prime },Q^{\prime }$ with $\sigma =\mathrm{id}$. 7. Is the $n\times s$ bottleneck better understood as Perceiver IO than as Linformer/Performer? The papers cite the latter; the former is the closer relative.

On the claims 8. What is TransTissueFormer’s row-wise PCC? Not reported anywhere, and it is the metric GenTox §2.3 argues is mandatory — and the one that exposes a mean predictor (16_MATH_NOTES.md §3.5). 9. Does Theorem 2 defend the reported $\rho =0.793$? It applies to fold-change data, so partly. But it assumes i.i.d. Gaussian noise, and a mean predictor’s error is structured. Neither theorem covers that case. 10. Algorithm 1 line 3 already imputes the withheld $G[t,b]$ cells. What does Funk-SVD alone score on them, without the transformer? 11. Writing out what the imputed matrix implies about the LI→KI map gives something affine of rank $\le r+1$ — verified in code/experiments/exp02_rank_test.py, where ridge solves it exactly ($\rho =1.0000$). Does that mean Figure 7’s pretrain row is measuring linear-map approximation? Or does the implementation depart from the pseudocode in a way that breaks the algebra? 12. Was target-tissue conditioning tried for the multi-task model? The reported diagnosis — “cannot decide which is the correct target” — is the exact failure multilingual NMT has with a missing target-language token, and it has a one-embedding fix.

Loose ends 13. GenTox §5 (Gene Network Analysis) and §6 (Out-of-distribution validation) are empty section headers. §2 spends three pages proving §5 is essential. 14. PerturbAtlas [51] is cited as a generative method in TransTissue §7. It’s a database. 15. The closest prior work to TransTissueFormer — rat→human hepatocyte translation with a bottleneck DNN (PLOS One 2020), explicitly framed as “circumventing the current reliance on orthologs” — is not cited in any of the four papers.

Part 8: Reading order

The program, in dependency order

1. ToxCompl+ (completionplus_CSCI.pdf) — the foundation. Funk-SVD and the MaxAE/sign-flip failure.
2. TransPlatformer (TransPlatformer____BMC_final.pdf) — the architecture and the anti-Seq2Seq argument.
3. TransTissue (TransTissue (1).pdf — the newer one) — scarcity + back-translation-by-matrix-completion.
4. GenTox (GenerativeTox.pdf) — §2 is the most valuable text in the set. Read it twice.

The outside literature 5. Kedzierska et al. (Genome Biology 2025) — the zero-shot critique 6. Ahlmann-Eltze et al. (Nature Methods 2025) — the linear-baselines critique 7. Virtual Cell Challenge 2025 wrap-up — 1,200 teams, baselines still not beaten 8. scGPT (Nature Methods 2024) — [8] 9. CPA (Lotfollahi, Mol Sys Bio 2023) — [27]. Read it against ToxCompl. Funk-SVD is its linear special case.

Read 5–7 before 8–9. Knowing what doesn’t work is worth more than knowing what claims to.

These docs

• 01_BACKGROUND.md — the biology, from zero, for an ML audience
• 10_SOTA_LANDSCAPE.md — the 2026 field, sourced
• 16_MATH_NOTES.md — every derivation, in LaTeX
• 14_RESEARCH_AGENDA.md — defensible vs speculative tracks
• 15_FRONTIER.md — the research program
• code/ — runnable checks; the baselines and the linearity test

Appendix: Glossary

Generated from: completionplus_CSCI.pdf, TransPlatformer____BMC_final.pdf, TransTissue (1).pdf, GenerativeTox.pdf. External claims sourced in 17_SOURCES.md. Single-cell “foreground” reflects knowledge to ~mid-2025 — flag anything you want re-checked against 2026.