Chapter 12 — What Machine Translation Offers Cross-Tissue Translation

Thirty years of low-resource MT research, mapped onto a problem that has never heard of it.

12.1 The claim

Cross-tissue transcriptomic translation and machine translation are the same problem with different nouns. Not “similar,” not “analogous in spirit” — structurally the same, with the same failure modes, the same data pathologies, and — this is the point — the same solutions, most of which have not been tried.

The correspondence:

This chapter walks the MT toolkit and asks, for each tool: does it transfer, and what would it buy?

The headline: the existing work independently reinvented back-translation (via matrix completion) and cited Sennrich for it — and then stopped. The seven years of back-translation research after Sennrich 2016 — tagging, noising, iterating — are unexploited, and at least two of them address problems the papers explicitly report.

12.2 The corpus table is the whole story

Here is the DrugMatrix CodeLink pair table. Rows are source tissue, columns target, entries = treatments measured in both.

        BM   BR   HE   IN   KI    LI   SP   TM
   BM  325    0   24    0  158   223  161    0
   BR    0   65   19    0    3     0    0    0
   HE   24   19  629    4  159   176   14    7
   IN    0    0    4   20    0     0    0    0
   KI  158    3  159    0  906   425   84   24
   LI  223    0  176    0  425 1674  164   28
   SP  161    0   14    0   84   164  180    0
   TM    0    0    7    0   24    28    0   29

Anyone who has worked on low-resource MT has seen this table a hundred times. Read it that way:

• Liver is English. 1,674 profiles. It appears in more studies than everything else combined, because that’s where the field’s attention has been — not because it’s more interesting.
• 425 sentence pairs for the best-resourced pair. Against 8,565-dimensional inputs and outputs. Sit with that ratio.
• 24 of 64 directions have zero parallel data. Not “few.” Zero.
• The zeros are structural, not accidental. BR–LI = 0 despite BR=65 and LI=1,674 — independent sampling predicts ~40. Brain was only ever measured in studies that did not do liver. The table decomposes exactly into 24 study panels (16_MATH_NOTES.md §6; NNLS residual 0.0, all 24 zeros reproduced).

That last point has a hard consequence, and it’s the one that should drive the research:

No pairwise model can ever reach those 24 directions. Not with a better architecture, not with more compute. The parallel data does not exist and never will — no one is going to re-run 600 rat studies with a brain panel.

MT has spent a decade on exactly this. Zero-shot NMT, pivot translation, and unsupervised NMT all exist because the same thing is true of Uyghur→Basque.

12.3 Back-translation: they started and stopped

What they did

TransTissue’s AugmentTrain&Transfer:

1. hold out test targets
2. impute the whole matrix with Funk-SVD
3. train on the synthetic pairs
4. fine-tune on the real ones

That is back-translation. Manufacture synthetic parallel data, pretrain on it, fine-tune on real. HE–TM goes from 7 real pairs to 2,711 synthetic. They cite Sennrich et al. 2016 — as reference [36], in the augmentation related-work section, apparently without noticing the connection is exact rather than decorative.

Reported gains: MAE 0.081→0.059, rare MAE 0.27→0.159, PCC 0.53→0.793.

What MT learned next, and they didn’t

Sennrich 2016 was the beginning. Three major results followed, and each one addresses a problem the papers explicitly report.

(a) Tagged back-translation — Caswell et al., 2019 ⭐

The finding: synthetic parallel data should be tagged. Prepend a token marking “this pair is synthetic.” The model then learns to use synthetic data for its statistical structure without mistaking its artifacts for the target distribution. Consistent gains, essentially free.

Why this is not a minor tweak here: we know exactly what artifact the synthetic data carries.

From code/experiments/exp02_rank_test.py: the Funk-SVD-augmented translation task is exactly affine. Ridge regression scores $\rho =1.0000$, MAE $0.0000$ on augmented pairs. The analytic construction $W=[P_{\mathrm{KI}}1]A^{+}$, with no fitting at all, also scores $1.0000$. And the induced map is nearly rank-2.

So the pretraining phase teaches the model: “translation is a rank-2 linear map.” Then fine-tuning has to un-teach that. Tagging is the mechanism that lets the model keep the useful statistics and discard the artifact — and this is the single cleanest application of tagged BT I can think of, because the artifact is not merely suspected, it’s provable.

Actionable: add a learned is_synthetic embedding to the bottleneck during pretraining. Set it to the “real” value at fine-tuning. Cost: one embedding vector. This is the highest value-to-effort ratio idea in this chapter.

(b) Noised / sampled back-translation — Edunov et al., 2018 ⭐

The finding, and it’s counterintuitive: synthetic data generated by sampling or with added noise substantially outperforms clean beam-search output. Clean synthetic data is too easy — it contains no uncertainty, so the model learns a degenerate mapping.

Why it matters here: Funk-SVD output isn’t merely clean, it’s noiselessly, exactly linear — the strongest possible version of the pathology Edunov identified. The model is being pretrained on data with a closed-form solution.

Actionable, and testable this week:

• Add calibrated Gaussian noise to the imputed matrix before pretraining, at the residual scale of the factorization
• Better: use a probabilistic matrix factorization and sample from the posterior over $G^{\prime }$ rather than taking the point estimate. Each epoch sees a different draw. That’s Edunov’s sampling, exactly.
• Ablate: clean augmentation vs noised vs sampled

Prediction: noised augmentation beats clean augmentation on real held-out data, even though it scores worse on augmented data. If that holds, it’s a clean, citable, mechanistically-explained result — and it reframes the exp02 finding from “a problem with Figure 7” into “a fix for the augmentation.”

(c) Iterative back-translation — Hoang et al., 2018

The finding: don’t do it once. Use the improved model to regenerate better synthetic data, retrain, repeat. Gains compound over 2–3 rounds.

The mapping: Funk-SVD imputes $G^{\prime }$ once, at rank 300, linearly. But after training, TransTissueFormer is itself a better cross-tissue predictor than Funk-SVD. So:

round 0:  G' ← Funk-SVD(G)              [rank-300 linear]
          M₀ ← train(G'), fine-tune(G)
round 1:  G'' ← M₀ fills the missing entries    [nonlinear, better]
          M₁ ← train(G''), fine-tune(G)
round 2:  ...

This directly attacks the exp02 problem. Round 0’s synthetic data is exactly linear. Round 1’s is not — it’s whatever the transformer learned. The linearity artifact dilutes with each round.

Actionable: one round of iteration. If MAE improves, the mechanism is confirmed and it’s a paper section.

12.4 Multilingual NMT: the fix for a reported negative result

The bug

TransTissue trained one model LI→everything. PCC collapsed from 0.53 to 0.23. The stated diagnosis:

“given the same source LI profile the model cannot decide which is the correct target”

This is the missing target-language token. Same source, multiple valid targets, no conditioning ⟹ the model averages.

Formally (16_MATH_NOTES.md §4.2), under squared loss the optimal single-valued $h$ is the conditional mean:

$$h^{\star }({\mathbf{s}}_c)=\mathbb{E}[{\mathbf{t}}_c\mid {\mathbf{s}}_c]=\frac{1}{|{\mathcal{T}}_c|}\sum _{\tau \in {\mathcal{T}}_c}{\mathbf{t}}_c^{\tau }$$

The model is behaving optimally for a mis-specified problem. Mush is the correct answer to an ill-posed question. This is not a finding about biology; it is a specification error.

The fix, from 2016

Johnson et al. prepend <2es> to tell the model which language to decode into. mBART, M2M-100, NLLB — all of them, universally.

Here: add a target-tissue embedding ${\mathbf{e}}_{\tau }$ to the bottleneck. One vector, 512 numbers.

Why the fix is worth more than the fix

Johnson et al.’s target token didn’t just enable multi-task. It enabled zero-shot translation — the model translated Portuguese→Spanish having never seen a single PT–ES pair, because it had learned PT→EN and EN→ES and the target token let it compose them.

That is precisely the 24-empty-pairs problem.

If ${\mathbf{e}}_{\mathrm{BR}}$ trains from any brain data (65 profiles exist) and the shared trunk learns the systemic response from the well-resourced pairs, then decoding brain from liver becomes possible with zero LI–BR pairs.

That’s the paper. Not “we improved PCC by 5%,” but “we translated to organs for which no paired data exists, and here is the enrichment analysis showing the predictions are biologically coherent.”

The pathology to expect, and how to detect it ⭐

Zero-shot NMT has a famous failure mode: off-target translation. Ask for Portuguese, get English. The model ignores the target token and defaults to the highest-resource language it knows.

The prediction, stated so it can fail:

A zero-shot cross-tissue model asked for brain will produce something that looks like liver — the highest-resource tissue — rather than brain.

And it’s detectable. Train a tissue classifier on real profiles. Run it on the zero-shot outputs. If brain predictions classify as liver, that’s off-target translation, diagnosed by name, with a literature of fixes attached:

• residual/language-specific layers (Liu et al. 2021)
• target-token position — encoder-side conditioning beats decoder-side prepending in some setups
• denoising auxiliary objectives to strengthen the conditioning signal

Actionable: the tissue classifier is a two-hour job and it is the right instrument regardless of whether zero-shot works. It answers “did the model produce a kidney?” — a question no PCC can answer.

12.5 Unsupervised MT: the idea nobody has tried here ⭐⭐

This is the most under-exploited connection in this chapter.

The MT result

Between 2017 and 2019, MT solved a problem that had been considered impossible: translation with zero parallel sentences.

• Mikolov et al. (2013): monolingual word embedding spaces for different languages are approximately isomorphic. A linear map $W$ aligns them. With a seed dictionary of ~5,000 pairs, learn $W$ by least squares.
• Procrustes refinement: constrain $W$ to be orthogonal. Closed form via SVD: $$W^{\star }=\arg \underset{W\in O(d)}{\min }\Vert WX-Y{\Vert }_F=U{V}^{\top },\text{where }U\Sigma V^{\top }=\mathrm{SVD}(Y{X}^{\top })$$
• Conneau et al. (2018), MUSE: you don’t even need the seed dictionary. Learn $W$ adversarially — a discriminator tries to tell $WX$ from $Y$; $W$ tries to fool it. Then refine with Procrustes on the induced dictionary. Fully unsupervised.
• Lample et al. / Artetxe et al. (2018): full unsupervised NMT — shared encoder, denoising autoencoder on monolingual data, iterative back-translation. BLEU in the high teens with zero parallel sentences.

The mapping

All the ingredients are present:

Note the second column. DrugMatrix has 1,674 liver profiles but only 425 paired with kidney. There are ~1,249 “monolingual” liver profiles that no cross-tissue model currently uses. Matrix completion touches them; the translation models do not. In MT terms: they are training on the parallel corpus and throwing away the monolingual data, in a low-resource setting, in 2026.

Why isomorphism is plausible here — and this is the interesting part

The isomorphism assumption in MT is a hypothesis about language: that “dog” sits in the same relative position in English space as “chien” does in French space, because both languages carve the world similarly.

In cross-tissue, there is a mechanistic reason to expect it. Every organ is responding to the same systemic event — the same compound, the same dose, in the same animal. Liver’s response space and kidney’s response space are two readouts of one underlying perturbation space. If both are (approximately) linear images of a shared latent:

$${\mathbf{x}}_c^{\mathrm{LI}}\approx W_{\mathrm{LI}}{\mathbf{z}}_c,{\mathbf{x}}_c^{\mathrm{KI}}\approx W_{\mathrm{KI}}{\mathbf{z}}_c$$

then the two spaces are related by $W_{\mathrm{KI}}{W}_{\mathrm{LI}}^{+}$ — isomorphic by construction. The physiology supplies the assumption that MT has to hope for.

That is a better a priori case than cross-lingual embedding alignment ever had.

The concrete method

1. Compute a profile embedding space per tissue independently, from ALL
   profiles of that tissue — paired or not. (PCA, or an autoencoder,
   or the co-expression structure.)

2. For a pair with SOME data (LI–KI, 425 pairs): learn the orthogonal
   map by Procrustes on the paired subset. Establishes the ceiling and
   validates the isomorphism assumption.

3. For a pair with NO data (LI–BR): learn the map ADVERSARIALLY, MUSE-style,
   using only unpaired LI and unpaired BR profiles. Refine with Procrustes
   on the induced pseudo-dictionary.

4. Validate on LI–KI by pretending the 425 pairs don't exist. If the
   unsupervised map recovers the supervised one, the method works and
   you can trust it on LI–BR.

Step 4 is the whole experiment. It’s a clean, self-validating design: you have a pair with data (LI–KI) to check the method, and pairs without data (the 24 zeros) to apply it to.

The known limitation — and it’s a gift

Søgaard et al. (2018), “On the Limitations of Unsupervised Bilingual Dictionary Induction”: unsupervised alignment works for typologically close languages and fails for distant ones. The isomorphism assumption breaks down with linguistic distance.

Map that onto the reported observation: HE–TM (both striated muscle, mitochondria-dense) achieves ρ=0.7 on 7 pairs. LI–KI (different mechanisms — hepatic P450 metabolism vs renal tubular transport) achieves ρ=0.4 on 425.

That is the isomorphism gradient, in toxicogenomics.

Which yields a falsifiable prediction:

Unsupervised cross-tissue alignment will succeed for same-family pairs (HE–TM, BM–SP) and fail for distant ones (LI–BR). The failure boundary should track biological relatedness, not sample size.

And a negative result here is informative, because it would measure where the shared-systemic-latent model stops holding — which is a statement about physiology, not about the method.

12.6 Choosing what to transfer from: LangRank → TissueRank

The MT result

Lin et al. (2019), “Choosing Transfer Languages for Cross-Lingual Learning.” Given a low-resource target, which high-resource language should you transfer from? They learn a ranker over features: corpus size, typological distance (WALS), lexical overlap, phylogenetic distance, geographic distance. LangRank beats human intuition.

The mapping

Given brain (65 profiles), which tissue should you transfer from? Currently: nobody asks. Models are trained per pair, and pairs are chosen by data availability.

TissueRank features — all computable today:

That last one is the good one: you can estimate whether two tissues are alignable using only unpaired data, via subspace distance (principal angles between the two PCA bases). No parallel data required. So you can rank transfer candidates for brain without ever having a brain pair.

Why it’s worth doing

With 24 of 64 directions empty and most of the rest tiny, “which transfers are worth attempting?” is not academic. It’s the difference between 40 models of unknown quality and a ranked shortlist.

Actionable, and cheap: compute principal angles between per-tissue PCA subspaces for all 28 pairs. Correlate against the reported PCCs. If subspace distance predicts translation quality, you have a pre-training-time predictor of translatability — and an explanation for HE–TM vs LI–KI that isn’t post-hoc storytelling.

12.7 Parameter sharing: adapters

The MT result

Bapna & Firat (2019), “Simple, Scalable Adaptation for Neural Machine Translation.” Freeze a massively multilingual trunk; inject small per-language adapter modules. Gets language-specific capacity without a separate model per language, and without the curse of multilinguality (Conneau et al.: adding languages to a fixed-capacity model degrades all of them past a point).

The mapping

Currently: 32 separate cross-tissue models, one per direction with data. Each trained from scratch. Nothing shared.

That is the pre-2016 MT world — a bilingual model per pair — and MT abandoned it because it wastes the statistical strength of related pairs. With 425 samples per pair, wasting statistical strength is not affordable.

Proposed:

shared trunk        learns the systemic response — trained on ALL tissues
tissue adapter τ    small, per-tissue          — ~1% of parameters

This composes with everything else in this chapter. Adapters + target-tissue conditioning = a single multilingual model with per-tissue capacity. It’s M2M-100’s architecture, and it’s the natural home for the zero-shot claim.

And there’s an architectural reason it fits — see Chapter 7. 96.6% of TransTissueFormer’s parameters live in the input bottleneck, which is a gene→slot projection. The transformer stack is 3.4%. So “shared trunk + tissue adapters” maps onto “shared bottleneck (gene semantics are universal) + tissue-specific adapters (organ programs differ)” almost too neatly.

12.8 What does NOT transfer

The false friends. Getting these wrong is worse than not making the analogy.

(a) No word order — and they’re right about this

TransPlatformer §2.2 rejects Seq2Seq partly because “the first s genes as tokens may not necessarily hint at the (s+1)th token.” Correct. Genes have no linear order; they form pathways and networks.

What dies: positional encodings, autoregressive decoding, causal masks, beam search, teacher forcing, and most of the decoding literature.

What survives: attention itself. “Not a sequence” implies no positional encoding — i.e. a set transformer — not “no attention.” The papers’ argument is sound on order but overshoots on architecture. What they built is closer to a set transformer than their own framing admits.

(b) The output is continuous, not discrete ⭐

The deepest disanalogy. MT decodes a discrete token from a softmax over a vocabulary. Cross-tissue regresses 8,565 continuous values.

What dies: softmax, cross-entropy, perplexity, likelihood, sampling, beam search, label smoothing, BLEU.

But — and this is a real idea: the categories already exist. DrugMatrix Table 1 defines five: extremely-under, under, normal, over, extremely-over.

Discretize the output and the entire MT machinery returns.

Predict the 5-way category per gene instead of a real number. You immediately regain:

• cross-entropy with class weights — which directly attacks the 92%-normal imbalance that every one of these papers struggles with
• calibration — is the model confident this gene is over-expressed? Nothing in the current setup has a notion of confidence
• label smoothing, focal loss — mature tools for exactly this imbalance
• a likelihood, hence sampling, hence honest uncertainty in the predicted profile

The 92%/8% imbalance is a classification problem that has been dressed as regression, and MAE is the wrong loss for it. Rare MAE is a patch over that mistake, not a fix.

Actionable: a hybrid head — categorical for direction/magnitude class, regression for the value within class. Report macro-F1 over the five categories alongside MAE. A macro-F1 the all-zeros predictor cannot game, unlike MAE.

(c) Length and alignment don’t exist

Fixed 8,565 → 8,565, gene $g$ in source corresponds to gene $g$ in target. No length modelling, no alignment, no attention-as-alignment interpretation.

This makes the problem easier than MT and means the identity baseline is meaningful in a way “copy the source sentence” is not.

(d) Monolingual data is scarcer, not abundant

MT’s monolingual advantage is overwhelming — billions of sentences vs millions of pairs. Here it’s ~1,249 unpaired liver vs 425 paired: a ratio of 3:1, not 1000:1.

So: unsupervised MT methods will be weaker here than in MT. But 3:1 is still 3:1, and currently the ratio being exploited is 0:1.

(e) There is no pretrained model in the right modality

MT has mBART, NLLB, XLM-R. Cross-tissue has scGPT and CellFM — trained on binned absolute counts, while DrugMatrix is log fold-change. That is a type error, not a domain gap (01_BACKGROUND.md §4.3). No amount of fine-tuning fixes a type error.

The exception, and it’s the important one: gene embeddings don’t touch values. See Chapter 7 §7.7.1 — that’s where this becomes actionable.

12.9 The scope, honestly

What transfers cleanly (high confidence):

What transfers with real risk (medium confidence):

The bet (low confidence, high value):

What doesn’t transfer: autoregressive decoding, beam search, positional encoding, BLEU, and the assumption that a pretrained model exists in your modality.

12.10 The one-paragraph version

Cross-tissue transcriptomic translation is low-resource machine translation with 425 sentence pairs, 24 zero-shot directions whose emptiness is structural rather than incidental, and a 3:1 monolingual-to-parallel ratio that nobody exploits. The existing work independently reinvented back-translation and cited Sennrich for it, then stopped — leaving tagged BT, noised BT, and iterative BT on the table, each of which addresses a pathology the papers themselves report. It reported a multi-task failure that is textbook missing-target-token. And it has never tried the one method built for exactly its worst case: unsupervised alignment, whose central assumption — that the two spaces are isomorphic — is not a hope here but a consequence of the fact that both organs are reading out the same systemic event.

References

Back-translation and data augmentation

• Sennrich, Haddow & Birch (2016). Improving Neural Machine Translation Models with Monolingual Data. ACL. — cited as [36] in TransTissue
• Edunov, Ott, Auli & Grangier (2018). Understanding Back-Translation at Scale. EMNLP. — noised/sampled BT
• Caswell, Chelba & Grangier (2019). Tagged Back-Translation. WMT.
• Hoang, Koehn, Haffari & Cohn (2018). Iterative Back-Translation for NMT. WNMT.

Multilingual and zero-shot

• Johnson et al. (2017). Google’s Multilingual NMT System: Enabling Zero-Shot Translation. TACL. — the <2es> token
• Liu et al. (2020). Multilingual Denoising Pre-training for NMT (mBART).
• Fan et al. (2021). Beyond English-Centric Multilingual Machine Translation (M2M-100).
• Zhang et al. (2020). Improving Massively Multilingual NMT and Zero-Shot Translation. — off-target
• Conneau et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-R). — curse of multilinguality

Unsupervised alignment

• Mikolov, Le & Sutskever (2013). Exploiting Similarities among Languages for MT.
• Conneau, Lample, Ranzato, Denoyer & Jégou (2018). Word Translation Without Parallel Data (MUSE).
• Artetxe, Labaka, Agirre & Cho (2018). Unsupervised Neural Machine Translation. ICLR.
• Lample, Conneau, Denoyer & Ranzato (2018). Unsupervised Machine Translation Using Monolingual Corpora Only. ICLR.
• Søgaard, Ruder & Vulić (2018). On the Limitations of Unsupervised Bilingual Dictionary Induction. ACL. — the distance limitation

Transfer selection and adapters

• Lin et al. (2019). Choosing Transfer Languages for Cross-Lingual Learning. ACL. — LangRank
• Bapna & Firat (2019). Simple, Scalable Adaptation for NMT. EMNLP. — adapters
• Houlsby et al. (2019). Parameter-Efficient Transfer Learning for NLP. ICML.

Citations are from background knowledge and should be checked against the originals before use. The mapping onto cross-tissue translation is interpretive throughout — an analogy is a way to see a problem, not evidence about it.