Chapter 7 — TransTissueFormer, In Depth

What it is, why it’s fast, why it’s the right shape, where it breaks, and what to do about it.

7.1 The problem it solves

Given the transcriptomic profile of a source tissue under a treatment, predict the target tissue profile for the same treatment.

$${\mathbf{s}}_c\in {\mathbb{R}}^n⟼{\mathbf{t}}_c\in {\mathbb{R}}^n,n=8,565\text{ (CodeLink)}$$

Both are ${\log }_{10}$ fold-change vs. control. A treatment $c$ is a (chemical, dose, duration) triple.

Concretely: Nitrofurantoin, 100 mg/kg, 24h. We measured the liver. Predict the heart.

Why it’s hard: 8,565 → 8,565, from 425 training examples on the best-resourced pair. And 92% of both vectors are ≈ 0.

7.2 The architecture

      (B, n)                          n = 8,565 source genes
        │
        │   [1]  bottleneck FC:  n × s          s = 512
        ▼
      (B, s)                          the profile, compressed to 512 numbers
        │
        │   [2]  projection:  1 × r             r = 16
        ▼
      (B, s, r)                       512 "slots", each a 16-dim token
        │
        │   [3]  L transformer layers ("TP attention")    L = 32
        ▼
      (B, s, r)
        │
        │   [4]  ⌈n/s⌉ = 17 PARALLEL decoders
        ▼
      (B, n, r)
        │
        │   [5]  projection:  r × 1
        ▼
      (B, n)                          predicted target profile

Hyperparameters, from the paper: $n=8565$, $s=512$, $r=16$, $L=32$, $\lceil n/s\rceil =17$ concurrent decoders. Adam, $\eta =10^{-3}$, weight decay $5\times 10^{-4}$, batch 16, 40 epochs.

The three claimed departures from Vaswani

1. The $n\times s$ bottleneck compresses 8,565 genes into 512 slots before attention. Attention then costs $O(s^2)$, not $O(n^2)$. Same spirit as Linformer/Performer — project the sequence dimension down.
2. Every output gene attends to the entire input profile. No context window. Any liver gene can in principle influence any kidney gene.
3. 17 parallel decoders. Not autoregressive — all segments emit at once.

7.3 Worked example: what the bottleneck actually does

Take $n=6$ genes, $s=3$ slots. The bottleneck is $W\in {\mathbb{R}}^{6\times 3}$:

$$W=\left[\begin{array}{ccc}0.9& 0.1& 0.0\\ 0.8& 0.2& 0.1\\ 0.1& 0.9& 0.0\\ 0.0& 0.8& 0.2\\ 0.1& 0.0& 0.9\\ 0.0& 0.1& 0.8\end{array}\right]\mathbf{x}=\left[\begin{array}{c}1.2\\ 0.9\\ 0.0\\ 0.1\\ -0.8\\ -0.6\end{array}\right]$$

The compressed representation is $\mathbf{h}=W^{\top }\mathbf{x}$:

$$h_1=0.9(1.2)+0.8(0.9)+0.1(0)+0(0.1)+0.1(-0.8)+0(-0.6)=1.72$$ $$h_2=0.1(1.2)+0.2(0.9)+0.9(0)+0.8(0.1)+0(-0.8)+0.1(-0.6)=0.32$$ $$h_3=0(1.2)+0.1(0.9)+0(0)+0.2(0.1)+0.9(-0.8)+0.8(-0.6)=-1.09$$

Read the columns of $W$. Column 1 loads on genes 1–2; column 2 on genes 3–4; column 3 on genes 5–6. Those are gene modules — co-regulated programs. Slot 1 says “module A is up (+1.72),” slot 3 says “module C is down (−1.09).”

Now read the rows. Row $g$ of $W$ is a vector in ${\mathbb{R}}^s$ describing how gene $g$ participates in each module.

$$\boxed{\text{Row }g\text{ of }W\text{ is a }512\text{-dimensional embedding of gene }g.}$$

And the operation $\mathbf{h}=W^{\top }\mathbf{x}={\sum }_g{x}_g\cdot W[g,:]$ is:

$$\boxed{\text{an expression-weighted sum of gene embeddings.}}$$

This is word2vec-style document embedding. A document is the weighted sum of its word vectors, with weights = term frequency. Here a profile is the weighted sum of its gene vectors, with weights = fold-change.

So the bottleneck isn’t a compression trick bolted on for compute reasons. It’s a gene embedding table, and the model is doing bag-of-embeddings pooling. That reading is what makes §7.7.1 possible.

7.4 Why it’s fast — the numbers

Attention memory

Standard self-attention over $n$ tokens materializes an $n\times n$ score matrix per head per layer. At fp32, batch 16, $L=32$ layers:

Standard attention on Affymetrix needs about 2 terabytes for score matrices alone. This isn’t “slow.” It doesn’t run, on any hardware that exists.

The bottleneck makes attention memory independent of $n$. It’s always $s^2=262,144$ entries — 0.54 GB across the whole stack. That’s the entire architectural argument in one row of a table.

Decode latency

Autoregressive decoding of an 8,565-token output is 8,565 forward passes, each attending to a growing KV cache. Parallel decoding is one pass. This is the larger practical win, and it’s the one the complexity table doesn’t show.

Complexity table

The paper’s Table 6:

⚠️ The “Standard / $O(s^2)$” entry is a typo. Standard attention’s score matrix is $O(n^2)$; $s$ isn’t even defined for it. Minor, but it’s the row that carries the paper’s central claim, so it should be fixed.

The real difference from Linformer: Linformer projects keys and values to $s$ but keeps $n$ queries, so its score matrix is $O(ns)$. TransTissueFormer projects the input itself to $s$ before attention, so its score matrix is $O(s^2)$ — strictly smaller. The cost is that it can never do gene-level attention again (see §7.6.1).

7.5 Why the shape is right

Three things the design gets genuinely right:

1. No positional encoding. Genes have no linear order — they form pathways and networks. Refusing to impose sequence order is correct, and the reasoning in TransPlatformer §2.2 is sound.

2. Full-profile context. Every output gene sees the whole input. In biology, any liver gene can in principle influence any kidney gene through systemic signalling. A context window would be a lie about the domain.

3. Parallel decoding. No autoregressive error accumulation, no arbitrary generation order. For a set-valued output, generating left-to-right would impose structure that doesn’t exist.

And the compute argument is unanswerable. At $n=31,042$, standard attention needs about 2 TB. There is no version of this project that uses vanilla self-attention over genes.

The honest framing: this is Perceiver-shaped. Cross-attend a huge input into a small latent array, compute in latent space, broadcast back out. The papers cite Linformer and Performer; Perceiver IO is the closer relative and isn’t mentioned. Worth knowing, because Perceiver’s literature — especially on latent-array sizing — applies directly.

7.6 Where it breaks

7.6.1 96.6% of the model is a linear projection ⭐

Parameter budget, CodeLink, from the paper’s stated hyperparameters:

(Standard transformer layer accounting: QKV + output + LayerNorm + a 4× MLP. Decoder depth is ambiguous in the paper — assumed ~1 layer each. Even at 32 layers each, the transformer stack reaches only ~1.8M, and the bottleneck still dominates.)

The transformer stack is 3.4% of the model. The “transformer” in TransTissueFormer is a rounding error on a 4.4M-parameter linear projection.

Three consequences follow.

(a) It explains the results. TransTissueFormer’s MAE is 0.081; the MLP’s is 0.086. A 6% gap. That’s not what you’d expect from a 32-layer transformer beating a 4-layer MLP. But it’s exactly what you’d expect if both models are mostly a big linear map with a small nonlinearity on top.

(b) It explains exp02. The Funk-SVD-augmented task is exactly affine of rank ≤ 301, and empirically rank ~2 (16_MATH_NOTES.md §2). A model that is 96.6% linear projection is a matched filter for a linear task. That’s why it wins Figure 7’s pretrain row (0.90 vs MLP 0.64 vs RF 0.51) — not because it captures “the intricate relationships between transcriptomic profiles from different tissues,” but because its inductive bias is the process that generated the data it’s being scored on.

(c) It’s 10,000 parameters per training sample. 4.4M parameters, 425 examples. The bottleneck is trained from scratch, from random init, on 425 samples.

7.6.2 There is no gene-gene attention

Follow the tensor. After step [1], the profile is 512 numbers. Every gene-level distinction that isn’t preserved by $W^{\top }\mathbf{x}$ is gone before attention runs.

So the 32 transformer layers attend over 512 pooled latent dimensions, not over genes. The claim that “each gene in the target tissue profile can be influenced by any gene in the source” is true — but it’s delivered by the linear pooling, not by the attention. All gene-gene interaction in this model is linear.

Attention here plays the role it plays in ToxCompl+ — re-weighting latent factors — not the role it plays in a transformer. That’s consistent with the program’s style, and it’s a defensible design. But it means the model can’t represent “gene A’s effect on gene B depends on gene C,” which is what pathway biology is made of.

✅ Confirmed against Figure 1. The figure shows (B,n) → [n×s] → (B,s) → [1×r] → (B,s,r) → L stacked layers. The transformer stack operates on (B, s, r) = 512 slots × 16 dims. Genes are gone before layer 1. This is no longer an inference — it’s what the figure draws.

7.6.3 The first attention layer is provably inert

Step [2] maps $(B,s)\to (B,s,r)$ via a $1\times r$ projection. Read literally — and Figure 1 labels the operation once — slot $j$’s token is

$${\text{token}}_j=h_j\cdot \mathbf{v},\mathbf{v}\in {\mathbb{R}}^{16}\text{ shared across all }s\text{ slots.}$$

Every token is then a scalar multiple of the same vector. Verified numerically in code/demo_transtissueformer.py:

Why: $Q_i=W_Q(h_i\mathbf{v})$, $K_j=W_K(h_j\mathbf{v})$, so $S_{ij}\propto h_i{h}_j({\mathbf{v}}^{\top }{W}_Q^{\top }{W}_K\mathbf{v})$ — an outer product, rank 1. A rank-1 score matrix means every query attends with the same pattern up to a scalar. Block 1’s attention contributes nothing.

But this is not fatal, and the distinction matters. The MLP’s nonlinearity breaks the symmetry:

So the model recovers. The cost is waste, not incorrectness:

• block 1’s attention is provably doing nothing
• the model spends its first block manufacturing diversity that a slot embedding would supply for free at step 0
• with $r=16$ there’s very little room to manufacture it in

Say it precisely. “The architecture is broken” is wrong. “The first attention layer is provably inert, and the fix costs 0.18% of the parameters” is right, and it’s a better thing to say.

⚠️ Check the implementation before acting. Figure 1 draws one trapezoid per slot but labels the operation once. If the $1\times r$ is actually per-slot, this is already handled and there’s nothing to fix. The paper alone can’t settle it.

Also: $r=16$ per token, against BERT’s 768. That’s a very small representation to run 32 layers over.

7.6.4 The gaps that follow from everything above

7.7 What to do — actionable, ordered by effort

7.7.1 Initialize the bottleneck from scGPT ⭐⭐ — the best idea in this document

The observation:

• TransTissueFormer’s bottleneck is $W\in {\mathbb{R}}^{n\times 512}$. Row $g$ is a 512-dim embedding of gene $g$ (§7.3).
• scGPT’s gene token embeddings are 512-dimensional (confirmed: scGPT’s $d_{\text{model}}=512$, and gene embeddings are 512-dim vectors from its encoder layer).

$$\boxed{\text{Same shape. }W\in {\mathbb{R}}^{n\times 512}\text{ and scGPT’s gene embedding table }E\in {\mathbb{R}}^{|V|\times 512}.}$$

So you can drop scGPT’s gene embeddings straight in as the initialization of $W$. No adaptation layer. No dimension mismatch. No architecture change. Map rat genes → human orthologs → look up → initialize.

Why this is the right way in, and not a hack:

1. It initializes 96.6% of the model. Not a side input, not an auxiliary loss — the overwhelming majority of the parameters, currently random, on 425 samples.
2. It completely dodges the type error. The fold-change/absolute-count incompatibility (01_BACKGROUND.md §4.3) lives in scGPT’s value encoder. This uses only the gene embeddings. Gene embeddings never touch expression values — they encode which genes are functionally related, which is modality-independent and largely species-conserved.
3. It’s the answer to the future-work paragraph. TransTissue §7 says these models “may nevertheless be finetuned… We plan to explore the adaptation of these models in future work.” The adaptation is one tensor load.
4. Both outcomes are publishable. If it helps, that’s the FM bridge. If it doesn’t — consistent with Kedzierska, Souza & Mehta, and “one PCA still rules them all” — it converts a hedge into “we tested it; here’s the evidence.”

The ablation, designed so it can’t fail to inform:

That last row is the interesting one. Rat P450s — precisely the genes toxicology cares about — have no clean human orthologs. If ESM2’s advantage concentrates in the non-ortholog genes, that’s a mechanistic result, not a leaderboard bump, and it’s exactly what the theory predicts.

Effort: days. This should be experiment #1.

7.7.2 Add a target-tissue embedding

One vector ${\mathbf{e}}_{\tau }\in {\mathbb{R}}^{512}$, added to $\mathbf{h}$ after the bottleneck:

$$\mathbf{h}=W^{\top }\mathbf{x}+{\mathbf{e}}_{\tau }$$

This fixes the ρ=0.23 multi-task collapse by construction — the conditional-mean argument (16_MATH_NOTES.md §4.2) no longer applies once the target is specified. And it’s the prerequisite for zero-shot (Chapter 12 §12.4).

Effort: a day. Risk: at 425 pairs the conditioning may be swamped. Untested.

7.7.3 Tag the synthetic data

We know — provably — that the augmented pretraining data is exactly affine and nearly rank-2. Tagged back-translation (Caswell et al. 2019) is the way to use synthetic data without absorbing its artifacts:

$$\mathbf{h}=W^{\top }\mathbf{x}+{\mathbf{e}}_{\tau }+{\mathbf{e}}_{\text{synthetic}}$$

Set it to the “real” tag at fine-tuning. Effort: hours.

7.7.4 Noise the augmentation

Funk-SVD output is noiselessly linear — the extreme of the pathology Edunov et al. (2018) identified. Sample from a probabilistic factorization’s posterior instead of taking the point estimate, so each epoch sees a different draw.

Prediction: noised augmentation beats clean augmentation on real held-out data while scoring worse on augmented data. Effort: days.

7.7.5 Add a slot embedding

$${\text{token}}_j=h_j\cdot \mathbf{v}+{\mathbf{p}}_j,P\in {\mathbb{R}}^{s\times r}$$

Verified effect (code/demo_transtissueformer.py):

Cost: $512\times 16=8,192$ parameters — 0.18% of the model. Benefit: block 1’s attention starts working.

This is a positional encoding — but over latent slots, not genes, so it doesn’t violate the (correct) no-gene-order principle. Check the implementation first (§7.6.3). Effort: hours.

7.7.6 Raise $r$, lower $s$

$r=16$ is tiny; the transformer has 3.4% of the parameters. Rebalance:

The third row is a real transformer. And attention gets cheaper — $s^2$ shrinks faster than $r$ grows. This is a free axis nobody has swept.

Hypothesis: if performance is flat across this sweep, the transformer isn’t contributing, and the honest model is a low-rank linear map plus a small MLP — consistent with everything in 10_SOTA_LANDSCAPE.md §3.

Effort: a sweep. Value: high either way.

7.7.7 Report the metrics that discriminate

• row-wise PCC — undefined for a mean predictor, so it’s the metric that exposes one (16_MATH_NOTES.md §3.5). GenTox §2.3 argues for it from first principles; TransTissue doesn’t report it.
• macro-F1 over the 5 categories — the 92%/8% split is a classification problem dressed as regression (Chapter 12 §12.8b). The all-zeros predictor can’t game macro-F1.
• enrichment-consistency — do the predicted and true profiles support the same biological conclusions? (15_FRONTIER.md F6.) The mean predictor scores zero here by construction: identical enrichment for every treatment means no discriminative power.

Effort: a week. This is the highest-value non-modelling contribution available.

7.8 The upgrade path

Each step composes with the last. Nothing here requires abandoning the architecture.

v0  TransTissueFormer as published
      32 separate models, random init, column-wise PCC

v1  + scGPT/UCE bottleneck init          [7.7.1]  ← 96.6% of params, days
    + row-wise PCC, macro-F1             [7.7.7]  ← the metrics that discriminate
    + slot embedding if needed           [7.7.5]

v2  + target-tissue embedding            [7.7.2]  ← one model, all pairs
    + tagged synthetic data              [7.7.3]  ← quarantine the linearity artifact
    + noised augmentation                [7.7.4]

v3  + tissue adapters                    [Ch 12 §12.7]  ← shared trunk, per-organ capacity
    + compound embedding (GenTox's GNN)  ← unifies GenTox and TransTissueFormer
    + unpaired profiles via denoising    ← the 1,249 liver profiles on the floor

v4  + zero-shot to the 24 empty pairs    [Ch 12 §12.4]
    + off-target detection               [Ch 12 §12.4]  ← "is this actually a kidney?"
    + unsupervised alignment             [Ch 12 §12.5]  ← the bet

v1 is days of work and touches 96.6% of the parameters. Start there.

7.9 The honest summary

What’s right. The compute argument is unanswerable — vanilla attention on 31,042 genes needs about 2 TB. Refusing positional encoding is correct. Parallel decoding is correct. Full-profile context is correct. The biological validation (§6 of the paper: gemfibrozil→PPARα, cisplatin→TP53, lead→p53) is worth more than every PCC in the four papers, and it’s the part that deserves to be automated and scaled.

What’s overstated. The transformer is 3.4% of the model. There’s no gene-gene attention — gene interactions are entirely linear. The 6% MAE gap over an MLP is what you’d expect from two mostly-linear models. And Figure 7’s pretrain row scores a nearly-linear model on a provably-linear task, which is a matched filter, not a finding.

What’s missing. Tissue conditioning. Compound representation. The unpaired profiles. Parameter sharing across pairs. Row-wise metrics. And a pretrained gene embedding table of exactly the right shape, sitting in a public checkpoint.

The one-sentence version:

TransTissueFormer is a 4.4M-parameter gene embedding table with a small transformer attached, trained from random initialization on 425 examples — and scGPT ships a pretrained gene embedding table with precisely matching dimensions, whose fold-change incompatibility lives entirely in a value encoder this architecture doesn’t have.

Parameter counts computed from the paper’s stated hyperparameters; decoder depth is ambiguous and assumed shallow. Architecture read from the paper’s prose and figure — §7.6.2 and §7.6.3 in particular should be checked against the implementation. The linearity result is verified in code/experiments/exp02_rank_test.py. scGPT’s 512-dim gene embeddings verified against its documentation.