Chapter 6 — GenTox: Deep Inductive Matrix Factorization

The three components, from zero, with numbers. Run code/demo_inductive_mf.py alongside. Read 04_TOXCOMPL.md first — this builds on plain MF.

Paper: “Predictive Transcriptomics with Attention-Augmented Inductive Matrix Factorization and Graph Mutual Information for Small Molecule Drugs” (GenTox).

The paper has three components:

This chapter is where the whole document set converges. The punchline is in §6.6.

6.1 The problem plain MF cannot solve

Recall ToxCompl:

$$G_{ij}=⟨P_{i,:},Q_{j,:}⟩$$

$P$ and $Q$ are lookup tables. To get $Q_{j,:}$ you look up row $j$ of a big array of learned numbers.

Now a chemist synthesizes a brand-new compound. Predict its profile.

Which row of $Q$ do you look up?

There isn’t one. The table has 2,238 rows, and this drug is number 2,239.

You can’t even ask the question. The model has no input slot for it. This isn’t “poor accuracy” — the function is undefined.

This is the cold-start problem. Netflix has it too: a movie released today has zero ratings, so no learned vector, so no recommendation.

That’s why GenTox exists. ToxCompl fills holes in a matrix. GenTox predicts a whole new column.

Two different verbs

6.2 §3.1 — the fix: replace the lookup with a function

The whole idea in one line:

$$\text{TRANSDUCTIVE:}{Q}_{j,:}=\text{lookup}[j]⟵\text{dies on new }j$$ $$\text{INDUCTIVE:}{Q}_{j,:}=f(\text{features}(j))⟵\text{works on any }j$$

A new drug has no row in a table. But it has features — it’s a molecule, and you can measure things about it. Feed the features to $f$, get a vector out.

GenTox’s Figure 3:

  row features (gene, tissue)          ──► Row NN ──┐
                                                     ├──► MIXER ──► G'[i,j]
  col features (compound, dose, time)  ──► Col NN ──┘

The paper’s own note on why this is a generalization, not a new thing:

“The model degenerates to a standard matrix factorization approach when the outputs of the row NN and the column NN have the same dimensions and the mixer network is simply a dot product.”

So plain MF is just inductive MF with $f$ = lookup and mixer = dot product. GenTox is strictly more general.

Watch the difference

A toy: 40 genes × 60 drugs. Each drug has 12 molecular descriptors that cause the response. Train on 50 drugs; the last 10 are new compounds — never seen, not one measurement.

---- TRANSDUCTIVE (plain MF, ToxCompl-style) ----
    on SEEN drugs : relative error 0.0000   ← excellent
    on NEW drugs  : *** CANNOT RUN ***
                    Q has 50 rows. The new drugs are rows 50-59.
                    There is nothing to look up. The function is undefined.

---- INDUCTIVE (GenTox-style) ----
    on SEEN drugs : relative error 0.0040
    on NEW drugs  : relative error 0.1656   ← *** IT RAN ***

It predicted profiles for 10 compounds it had never seen, from their molecular descriptors alone.

Nothing clever happened. We replaced one lookup with one small MLP. That’s the entire content of “deep inductive matrix factorization.”

One more thing §3.1 does

“As during training the samples in each minibatch are drawn uniformly at random from the whole G, the model in GenTox with three networks is expected to learn to not only predict the profile for a certain treatment but also preserve the relationships between endpoints in different regions of the matrix.”

Sampling random cells (not whole columns) is what forces the model to preserve row structure as well as column structure. That directly addresses the rank-degeneracy argument in 16_MATH_NOTES.md §3.4. It’s a deliberate, well-motivated design choice.

6.3 §3.2 — everything now depends on the features

The inductive model is only as good as $\text{features}(j)$.

The paper calls $\text{features}(j)$ the induction basis — the thing that makes induction possible at all.

Feature quality decides everything

induction basis                   SEEN drugs     NEW drugs
----------------------------------------------------------
informative descriptors               0.0020        0.1664
descriptors + heavy noise             0.0050        1.2549
random features                       0.0002        1.8598   ← !!
half the descriptors                  0.0049        1.0792

⭐ Read the random features row. It has the best in-distribution fit of all four (0.0002 — the MLP simply memorises) and the worst generalization (1.86).

A model can look perfect in-distribution and be worthless inductively. Only the new-drug column reveals it. That gap is the whole game.

Option A — hand-crafted features (the classical answer)

Mordred descriptors: about 1,826 numbers a chemist decided are meaningful. Molecular weight, logP, ring count, polar surface area. Deterministic — run the molecule through a fixed algorithm.

Morgan / ECFP fingerprints: a bit vector. Bit 4,412 = “does this molecule contain –C(=O)NH– ?” One bit per substructure.

In GenTox: $s=1826+1+1=1828$ — Mordred, plus dose, plus duration.

NLP analogy: this is a bag of hand-designed features. It’s what you did before word embeddings — someone decides in advance what properties matter, and you count them.

Option B — learned features (GenTox’s contribution)

Pretrain a graph neural network on about 1 million compounds, self-supervised.

A molecule is a graph: atoms are nodes, bonds are edges.

B1 — Contrastive (positive/negative pairs)

Take a molecule. Corrupt it twice, differently. Both corruptions came from the same molecule, so they’re a positive pair. A corruption of a different molecule is a negative pair. Train so positives end up close and negatives far apart.

Their corruptions: atom (node) masking plus bond (edge) deletion, “taking care not to select a node whose removal will result in a disconnected graph.”

$$\mathcal{L}=-\sum _i\log \frac{\exp (\mathrm{sim}(z_i,z_i^{+})/\tau )}{\exp (\mathrm{sim}(z_i,z_i^{+})/\tau )+{\sum }_{j\ne i}\exp (\mathrm{sim}(z_i,z_j^{-})/\tau )}$$

This is SimCLR. Identical objective. Molecules instead of images, atom-masking instead of cropping.

B2 — Graph InfoMax (mutual information)

Different idea: make each atom’s embedding predictive of the whole molecule’s embedding.

$$\underset{\theta }{\max }\sum _G\sum _{v\in V(G)}{I}_{\theta }(z_v;z_G)$$

Mutual information is intractable, so train a discriminator $D_{\theta }$ to tell real (atom, its-own-molecule) pairs from fake ones:

$$\mathcal{L}=-\sum _{v\in V}\log D_{\theta }(z_v,z_G)-\sum _{v^{\prime }\in V^{\prime }}\log (1-D_{\theta }(z_{v^{\prime }},z_{G^{\prime }}))$$

Intuition: if an atom’s neighbourhood tells you which molecule you’re in, the embedding has captured real structure.

Encoder: 2-layer GCN, hidden 64, output FC 64×300.

$$X^{\prime }={\hat{D}}^{-1/2}\hat{A}{\hat{D}}^{-1/2}X\Theta ,\hat{A}=A+I$$

Graph embedding = mean of node embeddings.

Their finding ⭐

graph InfoMax > contrastive pairs ≫ Mordred ≈ Morgan

“Both mutual information based approaches perform slightly better than the positive-negative pair approach, and much better than traditional fingerprinting (e.g., Morgan fingerprint) or descriptor (Mordred) approaches.”

Learned representations beat hand-crafted ones.

That sentence should sound familiar. It’s word2vec beating bag-of-words. BERT beating TF-IDF. Your field’s entire 2013–2018.

And note what this means ⭐⭐

GenTox pretrains a GNN on 1,000,000 compounds, self-supervised, freezes it, and uses it as a feature extractor.

That is a foundation model.

This program already built one — on the chemistry side. It did so before the TransTissue paper wrote “we plan to explore the adaptation of these models in future work.”

6.4 §3.3 — attention-based aggregation (the Mixer)

Now you have $R_i$ (gene factor) and $C_j$ (drug factor). Combine them.

The plain version: $G_{ij}=R_i\cdot C_j={\sum }_f{R}_{if}{C}_{jf}$. Every one of the 300 dimensions counts equally, because that’s what a dot product is. But maybe dimension 7 is decisive for this gene–drug pair and irrelevant for another one. A dot product can’t say that.

So GenTox learns two extra factors, $R_i^{\prime }$ and $C_j^{\prime }$, whose only job is to decide how much each dimension counts:

$$\boxed{G_{ij}=(R_i\circ C_j)\cdot \sigma (R_i^{\prime }\circ C_j^{\prime })}$$

$\circ$ is the Hadamard (elementwise) product — multiply position by position, don’t sum. $\sigma$ is softmax.

Worked out, $r=5$

R_i           = [ 3.0  -2.0   4.0   1.0  -0.5]
C_j           = [ 2.0   1.0  -1.0   3.0   2.0]
R_i ∘ C_j     = [ 6.0  -2.0  -4.0   3.0  -1.0]    ← WHAT each dim says

R'_i          = [ 0.2   0.1   2.8   0.3   0.1]
C'_j          = [ 0.4   0.5   2.6   0.2   0.3]
R'_i ∘ C'_j   = [0.08  0.05  7.28  0.06  0.03]
softmax(...)  = [0.00  0.00  1.00  0.00  0.00]    ← HOW MUCH each dim counts

plain dot product  : sum(R ∘ C)        =  2.000
attention-weighted : (R ∘ C) · softmax = -3.984

The plain version gives every dimension weight 1.0. Attention gives dimension 3 weight ≈ 1.00 and the rest ≈ 0, so the answer is dominated by dimension 3’s value (−4.0). The sign flipped, because the plain sum was averaging away the dimension that actually mattered.

Why this really is attention

values  = R ∘ C               (r of them)
weights = softmax(R' ∘ C')    (sum to 1)
output  = Σ values × weights

Query, keys, values — with $r$ “positions” and a single query. The same three lines as a transformer.

Their claim to novelty:

“To our knowledge no prior study has directly introduced attention to the low-rank factorization approach for matrix completion.”

It’s the same mechanism as ToxCompl+ (04_TOXCOMPL.md §4.9). One idea, two papers — worth knowing they’re the same thing.

6.5 A drafting bug worth reporting

The paper contradicts itself inside one paragraph.

§3.1 says:

“From the columns $c_1,c_2,c_3$ GenTox produces column (treatment) features” → columns = treatments

But the Results section says, in consecutive sentences:

“the fully connected layers in the column NN are $s\times 300$ … if the compound is represented by a Mordred descriptor, $s=1828$” → column NN handles compounds

“The col NN is a look-up table of $31099\times 300$ as there are 31099 genes/probesets in AffyMetrix” → col NN handles genes

The same network is described as both a 31,099-gene lookup table and an $s=1828$ compound MLP. The row/column labels are swapped somewhere.

The sensible reading, and what the model must actually be:

Easy fix. Worth flagging because §3.1 is the paper’s central contribution, and the labels are inverted in the very section that implements it.

(The paper is also a draft — broken Figure ?? refs, [?] citations, and §5 “Gene Network Analysis” / §6 “Out-of-distribution validation” are empty section headers.)

6.6 The asymmetry nobody has acted on ⭐⭐⭐

Read this sentence from the implementation section:

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

So here’s the split:

They built a foundation model for the columns and left the rows random.

And they proved, in their own ablation, that learned representations beat hand-crafted ones — for molecules. They never applied that lesson to genes.

This is the same gap in all three papers

Three papers. One randomly-initialized gene embedding table at the core of each. And a public checkpoint trained on 33 million cells that has exactly that object, already learned.

6.7 The fix: be inductive on genes too

$$\text{TODAY:}\text{gene\_factor}(i)=\text{lookup}[i]\text{(transductive)}$$ $$\text{PROPOSED:}\text{gene\_factor}(i)=\text{RowNN}(\text{gene\_features}(i))\text{(inductive)}$$

Where do gene features come from? The same place drug features come from: a pretrained model.

The symmetry is exact. It’s the argument they already made and won, applied to the other axis of the same matrix.

What inductive-on-genes buys — more than a leaderboard bump

1. New platforms. A CodeLink probe and an Affymetrix probe for the same gene are different lookup rows today — the model has no idea they’re the same gene. With gene features they get the same features. That’s TransPlatformer’s entire problem, solved structurally.

2. New species. Rat → human. A lookup table can’t do this, ever. UCE tokenizes genes by their protein sequence, so a rat gene and its human ortholog get similar features automatically, with no dictionary needed (11_SC_FOUNDATION_MODELS.md §6).

3. The low-data tissues. ⭐ This is the big one. From 04_TOXCOMPL.md §4.6: a brain gene has ~65 observations and 300 free parameters. The lookup row is underdetermined — arbitrary in most directions. An inductive gene factor doesn’t need 300 observations. It’s a function of features shared with every other gene, so every gene’s data helps every other gene.

4. Unmeasured genes. A gene never on any array still has a protein sequence → still has features → still predictable.

Demonstrated — the underdetermined regime

45 genes x 40 drugs, observed 11%
-> 197 observations vs 340 parameters, ratio 0.58   (UNDERDETERMINED)
   (values scaled so mean |G| = 1.00, so errors read as fractions)

gene factor                               held-out MAE     MaxAE
----------------------------------------------------------------
LOOKUP TABLE (GenTox today)                      0.990     9.399
MLP on gene features (proposed)                  0.268     3.050

-> MAE 73% lower,  MaxAE 68% lower

Both improve — and MaxAE is the metric ToxCompl+ was built to fix.

⚠️ The control that matters

The comparison is not “features vs lookup.” That’s rigged when the system is underdetermined — of course a function beats 300 free parameters fit from 65 observations.

The real question: do scGPT/UCE gene features beat the co-expression structure you can compute from DrugMatrix itself? That’s a free basis, no foundation model required.

If scGPT loses to DrugMatrix’s own co-expression, that’s a result too. It’s what Kedzierska (2025) and Souza & Mehta (2026) would predict. Either outcome publishes.

6.8 Summary

The problem. Plain MF uses lookup tables. A new drug has no row, so you can’t even ask the question. Cold start.

§3.1 — deep inductive matrix factorization. Replace the lookup with a function of features: $Q_{j,:}=f(\text{features}(j))$. Now a new drug works — it has features even though it has no row. Degenerates to plain MF when $f$ = lookup and mixer = dot product. Random-cell minibatch sampling forces row and column structure to be preserved.

§3.2 — representation learning for the induction basis. Everything depends on $\text{features}(j)$. Random features give the best in-distribution fit and the worst generalization — only the new-drug column reveals it.

• hand-crafted: Mordred (1,826 numbers), Morgan (substructure bits)
• learned: GNN on 1M compounds — contrastive (= SimCLR for molecules) or graph InfoMax

Finding: InfoMax > contrastive ≫ Mordred ≈ Morgan. Learned beats hand-crafted. That’s word2vec vs bag-of-words. And a GNN pretrained on 1M compounds is a foundation model — they built one.

§3.3 — attention-based aggregation. $G_{ij}=(R\circ C)\cdot \sigma (R^{\prime }\circ C^{\prime })$. Values from one pair of factors, weights from another. Attention with $r$ positions and one query. Same mechanism as ToxCompl+.

The opening.

“As we do not perform inductive learning on the genes or tissues, the [gene] NN we use is a simply embedding layer.”

They went inductive on drugs with a 1M-compound foundation model, and left genes as a random $31,099\times 300$ lookup table. All three papers have the same random gene table at their core.

Being inductive on genes buys new platforms, new species, the low-data tissues, and genes never measured. The argument is the one they already made and won — applied to the other axis of the same matrix.

Every number here is produced by code/demo_inductive_mf.py. Quotes are verbatim from the GenTox draft. The row/column inconsistency in §6.5 is in the paper as written and may be a labelling slip rather than a modelling error — check the implementation.