Chapter 8 — The Augmentation: What Is Done, Why, and What Needs Doing On Top

Walked through on the paper’s own Figure 4 example, with real numbers. Run code/demo_augmentation.py alongside — it prints every number here.

8.1 The problem, in one table

We want a liver → kidney translation model. We need treatments where both organs were measured.

Here is the paper’s own toy: 4 tissues, 7 treatments.

        1    2    3    4    5    6    7
  LI   ##   ##   ..   ##   ..   ##   ..
  BR   ##   ##   ..   ..   ##   ##   ##
  KI   ##   ##   ..   ..   ..   ##   ..
  HE   ..   ##   ##   ##   ..   ##   ..

       ## = measured      .. = missing

LI measured on:  1, 2, 4, 6
KI measured on:  1, 2, 6
─────────────────────────────
LI–KI PAIRS:     1, 2, 6        ← THREE. That's the whole training set.

At real scale that number is 425, against 8,565-dimensional inputs and outputs. 88% of DrugMatrix is empty.

Section 5 exists to fix this.

(The sparsity pattern above is reconstructed from the paper’s own arithmetic — see §8.5. It reproduces their stated 43% figure exactly, which is how we know it’s right.)

8.2 The idea: manufacture the missing data

The paper’s reasoning, and it’s sound:

“The scarcity of our data samples for translation is due to many of the missing endpoints in DrugMatrix. Had all entries contained observed measurement values, for all cross-tissue translation models there would be orders of magnitude more training samples.”

So: fill in the holes, then train on the filled-in matrix.

Why not the usual augmentation tricks?

They’re right to rule them out, and the reasoning is worth understanding:

“Unfortunately none of the existing augmentation techniques are suitable in training cross-tissue translation models.”

Correct — with one large asterisk. They then invent back-translation anyway. See §8.9.

8.3 Funk-SVD in one paragraph

Stack the data: rows are (tissue, gene), columns are treatments.

Assume the matrix is low-rank, and factor it:

$$G_{ij}\approx b_i+b_j+⟨P_{i,:},Q_{:,j}⟩$$

• $P$ is $(\text{rows}\times r)$ — “maps genes into an $r$-dimensional latent space”
• $Q$ is $(r\times \text{cols})$ — “maps treatments into the same space”
• $b_i,b_j$ — per-gene and per-treatment biases
• $r=300$ in the paper

Fit by minimising squared error on observed entries only:

$$\underset{P,Q,b}{\min }\sum _{(i,j)\in \Omega }(G_{ij}-(b_i+b_j+{\sum }_{f=1}^r{P}_{i,f}{Q}_{j,f}){)}^2$$

Adam, $\eta =10^{-3}$, weight decay $5\times 10^{-4}$.

Then read the missing entries off the product. That’s the entire augmentation.

This is the Netflix Prize. Genes are users, treatments are movies, fold-change is the rating. Most users haven’t rated most movies; predict the ratings they’d give.

Why low-rank is defensible

Their argument, and it’s a good one:

“there are similarities between the drugs, for example, Doxorubicin and Epirubicin, and the genes form co-expression networks.”

3,000 columns but only 636 distinct drugs, each at ≤5 dose/duration regimes. Doxorubicin and Epirubicin are nearly the same molecule doing nearly the same thing. Genes move in co-regulated modules. The same gene is measured in 8 organs. Rank 300 out of 3,000 is plausible.

8.4 Algorithm 1, line by line

Algorithm 1  AugmentTrain&Transfer(G, m, s, t)
──────────────────────────────────────────────────────────
 1: split treatments into train set a, test set b
 2: F ← G − G[t, b]            # withhold TARGET rows at TEST cols
 3: G' ← Funk_SVD(F)           # impute EVERYTHING
 4: G' ← G'[:, s ∪ t]          # keep source+target tissue rows
 5: split G' column-wise → train / validation
 6: train model M on augmented data
 7: fine-tune M on real measured data G[s ∪ t, a]
 8: test M on real measured data G[s ∪ t, b]

In our toy: train = {1, 2}, test = {6}.

Line 2 — withhold the test target

  (a')  F = G with KI[6] removed
        1    2    3    4    5    6    7
  LI   ##   ##   ..   ##   ..   ##   ..
  BR   ##   ##   ..   ..   ##   ##   ##
  KI   ##   ##   ..   ..   ..   ..   ..     ← KI[6] gone
  HE   ..   ##   ##   ##   ..   ##   ..

Look at column 6 carefully:

The paper is explicit that this is deliberate:

“the correlation of LI treatments 6 and 2 makes it possible to infer BR treatment 6, which can further help to infer the withheld KI profile for treatment 6.”

And explains why they don’t remove more:

“If both the LI profile and the KI profile for treatment 6 are removed from G, then an extreme situation occurs, that is, the entire column 6 will be empty.”

Both statements are true. Hold that thought until §8.7.

8.5 The payoff — and the genuinely clever part

After imputation, every cell is filled:

BEFORE:  LI–KI pairs = {1, 2, 6}                 → 3 training samples
AFTER:   LI–KI pairs = {1,2,3,4,5,6,7}           → 7 training samples

3 → 7. At real scale: 7 → 2,711 for HE–TM.

Where did treatment 3 come from? ⭐

Treatment 3 has no liver and no kidney measurement. Only HE[3] exists. Yet $G^{\prime }$ now has an LI–KI pair for it. How?

HE[3] is measured
   → it constrains Q[:,3], the latent vector for treatment 3
   → Q[:,3] + P[LI rows]  gives you an imputed LI[3]
   → Q[:,3] + P[KI rows]  gives you an imputed KI[3]
   → an LI–KI pair for a treatment where NEITHER was measured

Heart data created a liver–kidney training pair.

That’s a real, clever contribution, and it’s why they impute the whole matrix rather than the LI/KI sub-matrix. Their own arithmetic:

The paper says 43%. The toy reproduces 43% exactly — which is how we know the reconstructed sparsity pattern is right.

NLP analogy: this is pivot translation. You can’t get Portuguese→Spanish directly, so you route through English. Here you can’t get liver–kidney for treatment 3, so you route through heart.

8.6 What the augmented data actually contains ⭐⭐

This is the important section. Everything above is setup.

Funk-SVD produced:

$$G_{ij}^{\prime }=b_i+b_j+⟨P_{i,:},Q_{:,j}⟩$$

So for treatment $c$, the liver and kidney profiles are:

$${\mathbf{s}}_c={\mathbf{b}}_{\mathrm{LI}}+b_{c}1+P_{\mathrm{LI}}{\mathbf{q}}_c{\mathbf{t}}_c={\mathbf{b}}_{\mathrm{KI}}+b_{c}1+P_{\mathrm{KI}}{\mathbf{q}}_c$$

Both are built from the same ${\mathbf{q}}_c$. Count the unknowns:

$$\underset{\text{equations from }{\mathbf{s}}_c}{\underbrace{8,565}}\gg \underset{{\mathbf{q}}_c\text{ (300 dims)}+b_c}{\underbrace{301}}$$

Wildly over-determined. So ${\mathbf{q}}_c$ is exactly recoverable from the liver profile by least squares. And the kidney profile is then just $P_{\mathrm{KI}}{\mathbf{q}}_c$:

$$\boxed{{\mathbf{t}}_c=\left[P_{\mathrm{KI}}1\right]A^{+}({\mathbf{s}}_c-{\mathbf{b}}_{\mathrm{LI}})+{\mathbf{b}}_{\mathrm{KI}},A=\left[P_{\mathrm{LI}}1\right]}$$

Liver → kidney translation, on augmented data, is an exact linear map.

Verified in the toy (demo_augmentation.py STEP 6):

No model was trained. That’s a closed-form matrix inverse.

What this means

Pretraining a 32-layer transformer on augmented data teaches it to invert a low-rank linear map. It cannot teach biology, because there is no biology in the augmented data beyond that map.

This reframes Figure 7 (RF 0.51 / MLP 0.64 / TransTissueFormer 0.90 on augmented data). It is not measuring who captures “the intricate relationships between transcriptomic profiles from different tissues” — the paper’s phrase. It’s measuring who best approximates a linear map.

And TransTissueFormer’s $n\times s$ bottleneck is 96.6% of its parameters and structurally a linear map (07_TRANSTISSUEFORMER.md §7.6.1). It wins because its inductive bias is the generative process of the data it’s being scored on. Matched filter, not insight.

A ridge regression belongs in that figure. It would top it.

What this does NOT say

⚠️ The fine-tuned 0.72 on real measured data is untouched by this. That number is real held-out measurement. This is about one figure and one missing baseline — not about the model being bad.

Also: at real scale the effective rank is ~2, not 301 (16_MATH_NOTES.md §2.5). $r$ is a ceiling, not the rank. The task is even more degenerate than the algebra suggests.

8.7 Is the test set leaking?

Line 2 withheld KI[6]. Good. But trace what informed the imputed $G^{\prime }[\mathrm{KI},6]$:

LI[6] measured, in the matrix   ─┐
BR[6] measured, in the matrix   ─┼→ all constrain Q[:,6]
HE[6] measured, in the matrix   ─┘
                                    ↓
G'[KI,6] = b_KI + b_6 + P_KI · Q[:,6]

Is that leakage? The measured KI[6] was never seen, so in the narrow sense: no.

But now ask what happens at deployment. A new compound arrives. You ran the liver panel. You want the kidney prediction:

${\mathbf{q}}_{\text{new}}$ can only be informed by the liver profile.

But in the evaluation, ${\mathbf{q}}_6$ was informed by BR[6] and HE[6] too.

The test condition is easier than the deployment condition. The number doesn’t measure what the paper claims it measures.

8.8 The fix — and why their objection doesn’t apply

The paper rejects withholding more:

“If both the LI profile and the KI profile for treatment 6 are removed from G, then an extreme situation occurs, that is, the entire column 6 will be empty. As such, no correlation can be established with other parts of the matrix.”

Correct — but that’s not the protocol deployment needs. Deployment keeps the source:

$$F\leftarrow G-G[\text{all tissues EXCEPT source},b]$$

  F under the STRICT protocol
        1    2    3    4    5    6    7
  LI   ##   ##   ..   ##   ..   ##   ..     ← liver KEPT
  BR   ##   ##   ..   ..   ##   ..   ##
  KI   ##   ##   ..   ..   ..   ..   ..
  HE   ..   ##   ##   ##   ..   ..   ..

The column is not empty. It has the liver profile — exactly what you’ll have in production. Their objection is against removing the source too. It doesn’t apply.

What it costs

How well is the withheld, measured KI[6] recovered?

The gap is how much of the reported performance comes from information you won’t have at deployment.

(In this toy the gap is modest because the latent dim is 3 and liver alone nearly determines $\mathbf{q}$. At $r=300$ on real data the gap should be larger — which is why it’s worth measuring rather than arguing about.)

8.9 What to do on top ⭐

The realisation

The augmentation IS back-translation.

Same algorithm. And the paper cites Sennrich et al. 2016 — as reference [36], in the related-work list, apparently without noticing the connection is exact rather than decorative.

MT then spent seven years improving it. Each of the following addresses a problem this paper reports.

FIX 1 — Tag the synthetic data (Caswell et al. 2019)

The MT result: mark synthetic pairs with a tag. The model uses their statistics without absorbing their artifacts. Consistent gains, essentially free.

Why it’s not a minor tweak here: §8.6 proved the artifact. The synthetic data says “translation is a rank-2 linear map.” Fine-tuning has to un-teach that. Tagging is the mechanism that lets the model keep the useful statistics and quarantine the artifact — and this may be the cleanest application of tagged BT anywhere, because the artifact isn’t suspected, it’s provable.

Do: add a learned e_synthetic vector to the bottleneck during pretraining; switch to e_real at fine-tuning. Cost: one 512-dim vector. Hours. Highest value-to-effort item in this chapter.

FIX 2 — Noise the synthetic data (Edunov et al. 2018)

The MT result, and it’s counterintuitive: noisy synthetic data substantially beats clean synthetic data. Clean output is too easy — no uncertainty, so the model learns a degenerate mapping.

Why it matters here: Funk-SVD output isn’t merely clean. §8.6 showed it is noiselessly, exactly linear — the strongest possible form of the pathology Edunov identified. The model is being pretrained on data with a closed-form solution.

Do: use a probabilistic matrix factorization and sample from the posterior over $G^{\prime }$ each epoch, rather than taking the point estimate. Prediction: noised augmentation beats clean augmentation on real held-out data while scoring worse on augmented data. Cost: days.

FIX 3 — Iterate (Hoang et al. 2018)

round 0:  G'  ← Funk-SVD(G)             [rank-300 LINEAR]
          M₀  ← train(G'), finetune(G)
round 1:  G'' ← M₀ fills the gaps       [NONLINEAR, and better than Funk-SVD
                                         — which is the paper's own claim]
          M₁  ← train(G''), finetune(G)

This directly attacks §8.6. Round 0’s synthetic data is exactly linear. Round 1’s is whatever the transformer learned. The linearity artifact dilutes with each round — and the mechanism is the paper’s own result that the transformer beats Funk-SVD.

Cost: one retrain.

FIX 4 — Report the strict protocol

Report both numbers (§8.8). The paper’s protocol answers “how well can we fill a hole in this matrix?” The strict protocol answers “how well can we translate a new compound?” The paper claims the second.

FIX 5 — The baselines (this one gates everything)

(a) Funk-SVD alone. Line 3 already imputes $G^{\prime }[\mathrm{KI},6]$. Just read it out and score it. No transformer. That number already exists inside the pipeline. If it’s ~0.75, the transformer adds little over the matrix completion, and the contribution is the augmentation — still a contribution, differently stated. Ten minutes.

(b) Ridge on the augmented pairs, in Figure 7. §8.6 says it scores 1.0.

(c) The mean predictor. Ignore liver, always predict the average kidney response. Ahlmann-Eltze et al. (Nature Methods 2025) found five foundation models plus two deep models failing to beat exactly this.

(d) Row-wise PCC. Undefined for the mean predictor — constant rows, no denominator — which is precisely what makes it the metric that exposes it (16_MATH_NOTES.md §3.5). GenTox §2.3 argues for row-wise metrics from first principles. TransTissue reports column-wise only.

8.10 Summary

WHAT IS DONE 3 real LI–KI pairs → impute the whole matrix with Funk-SVD → 7 pairs → pretrain on imputed → fine-tune on the 3 real ones. Imputing the whole matrix (not just LI+KI) is the clever part: heart data creates liver–kidney pairs, worth 43% more training data. That is a real contribution.

WHY IT WORKS Real pairs are the bottleneck — 425 at full scale. The matrix is 88% empty. Filling it multiplies the training set by ~400×.

WHAT’S WRONG

1. The synthetic data is exactly a linear map (verified). Pretraining on it teaches linear-map inversion, not biology. Figure 7 measures that.
2. The eval protocol lets the test treatment’s other tissues inform the imputation. Deployment has only the source.
3. No baselines. Funk-SVD-alone, ridge, and mean are all missing — and the first already exists inside the pipeline.

WHAT TO DO tag it (hours) → noise it (days) → iterate it (one retrain) → report the strict protocol → and run the baselines first.

This is back-translation. They cited Sennrich and stopped. Seven years of follow-up work is sitting there, and it addresses the exact problems the paper reports.

Every number here is produced by code/demo_augmentation.py. The sparsity pattern is reconstructed from the paper’s stated 43% figure and reproduces it exactly. The linearity result is a property of Funk-SVD’s algebra, so it should transfer to real data — but run it there before saying so.