Chapter 21 — Do Fine-Tuned scFMs Live in a Composable Weight Space? Task Arithmetic, Critiqued and Planned

Your idea: take one single-cell foundation model (scGPT; Geneformer as a replication target), fine-tune it separately into several specialists, and ask whether the resulting checkpoints form a composable weight space — whether task vectors $\tau ={\theta }_{\mathrm{ft}}-{\theta }_{\mathrm{base}}$ can be added (build one model that does many tasks), negated (subtract a batch/donor confound as a weight-space alternative to batch correction), and composed (${\tau }_A-{\tau }_B+{\tau }_C$, transfer to an unseen tissue). Not a new architecture — a transfer of a mature NLP capability, where either outcome is a result. This chapter takes that plan apart the way Chapter 20 took apart the reasoning-dataset idea: what task arithmetic actually is, what works and what fails in its home domains, whether the transcriptomic setting even satisfies its preconditions, the prior art you must cite, a concrete implementation with downstream tasks, the critics’ strongest objections, and the way out of each.

Verdict up front: the plan is well-posed and worth running — scGPT and Geneformer are genuinely full-fine-tuned from a shared checkpoint, so the arithmetic is at least defined, which is more than can be said for most “transfer an NLP trick” ideas. But the honest expectation, from both the toy in §21.4 and the field’s own skepticism (Chapter 18 §18.4), is: addition will recover a useful-but-lossy fraction of the specialists; negation is the sharpest and most publishable operation (selective confound removal); plain analogy is the riskiest and may fail. The single highest-value reframe is to stop selling “a new merged model that beats specialists” (it won’t) and sell “the first characterization of the fine-tuning geometry of scFMs” — where a negative result is as citable as a positive one. And do not use plain summation: use the interference-aware merges (TIES, DARE) as your real method, or you are testing 2022’s algorithm, not 2025’s.

21.1 The idea, stated fairly

Restating your plan precisely so we critique the real thing:

1. One base, several specialists. Take ${\theta }_{\mathrm{base}}$ (a published scGPT checkpoint) and full-fine-tune it separately on $K\ge 3$ downstream tasks or tissues from public atlases — e.g. cell-type annotation in distinct tissues — giving ${\theta }_1,\dots ,{\theta }_K$.
2. Extract task vectors. ${\tau }_i={\theta }_i-{\theta }_{\mathrm{base}}$.
3. Test the three operations.
   • Addition: ${\theta }_{\mathrm{add}}={\theta }_{\mathrm{base}}+\lambda {\sum }_i{\tau }_i$, sweep $\lambda$, measure per-task accuracy against specialists and joint multi-task fine-tuning.
   • Negation: ${\theta }_{\mathrm{neg}}={\theta }_{\mathrm{base}}-\lambda {\tau }_j$ — does subtracting a capability remove it selectively? The application of interest: subtract a task vector fine-tuned on a batch/donor-confounded objective, as a weight-space alternative to explicit batch correction.
   • Composition/analogy: ${\theta }_{\mathrm{comp}}={\theta }_{\mathrm{base}}+\lambda ({\tau }_A-{\tau }_B+{\tau }_C)$ — transfer a capability to an unseen tissue without fine-tuning on it.
4. Baselines and read-out. Compare against each specialist, joint multi-task fine-tuning on pooled data, naive uniform averaging (model soup), and the zero-shot base. Report per-task accuracy vs $\lambda$; the multi-task trade-off surface; negation selectivity; transfer accuracy; and stability across seeds and fine-tuning orders.

The two-outcome framing is the strength of the proposal, and it is exactly the Chapter 18 §18.6 discipline: if task vectors compose, you deliver the first demonstration that the algebra of task arithmetic holds on real scFM checkpoints — extending beyond the additive-only, cross-modal fusion of the Model Synthesizer (§21.6) to same-modality composition and to confound subtraction by negation. If they do not compose, you provide rigorous evidence that fine-tuned scFM checkpoints lack the linear mode connectivity language models exhibit — which would explain the field’s reliance on full fine-tuning and caution against naive checkpoint reuse. Either way you characterize, for the first time, the geometry of the fine-tuning landscape in single-cell foundation models.

Now the machinery, then the knife.

21.2 What task arithmetic is, from zero

The founding paper is Ilharco et al., “Editing Models with Task Arithmetic,” ICLR 2023 (arXiv:2212.04089). One object, three operations, one knob.

The task vector. Fine-tune a pretrained model ${\theta }_{\mathrm{base}}$ on task $i$ to get ${\theta }_i$. The task vector is the element-wise difference

$${\tau }_i={\theta }_i-{\theta }_{\mathrm{base}}.$$

It is the direction in weight space that fine-tuning moved the checkpoint — a full-dimensional vector, one number per parameter. Editing is done by walking along sums of these directions: ${\theta }_{\mathrm{new}}={\theta }_{\mathrm{base}}+\lambda {\tau }_{\mathrm{new}}$.

Addition → one multi-task model. Summing task vectors and adding them back builds a single model competent on all the constituent tasks, with no joint training and no extra inference cost. On the field-standard 8-task CLIP ViT benchmark (Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, SVHN), adding two task vectors retains ~98.9% of the specialists’ accuracy (in normalized units — merged accuracy divided by the specialist’s, averaged over tasks); adding all eight reaches ~91.2%. That downward drift from 98.9% → 91.2% is the interference story (§21.7): the more you add, the more the directions collide.

Negation → unlearning. Subtracting a task vector suppresses a capability while sparing the rest: ${\theta }_{\mathrm{base}}-\lambda \tau$. On GPT-2-large, negating a “toxicity” task vector cut toxic generations from 4.8% → 0.8% (~6×) while holding WikiText-103 perplexity within ~0.5 points of the base. This is the operation your batch-confound application rides on.

Analogy → transfer without target data. ${\tau }_{\mathrm{new}}={\tau }_A-{\tau }_B+{\tau }_C$ (“A is to B as C is to D”) improves a fourth task without training on it — but only when the tasks share the relational structure the analogy assumes. This is the fragile operation; it has no free lunch when the tasks are unrelated (see the toy, §21.4).

The knob $\lambda$. A single scalar multiplies the (summed) task vector, chosen by grid search on a held-out validation set, typically $\lambda \in \{0,0.05,\dots ,1.0\}$, maximizing average validation accuracy. One global $\lambda$ is the method’s convenience and its main fragility (§21.7).

The dictionary (for the NLP reader). This is the same move as 09_WALKTHROUGH.md’s NLP↔tox dictionary, one level up: a task vector is a LoRA-like “capability delta,” addition is a model merge / soup, negation is machine unlearning, and $\lambda$ is a merge coefficient. Nothing here is single-cell-specific — which is the whole question.

21.3 Why it works — and why that is the entire risk

Task arithmetic is not a law of neural networks. It is a property that holds when the fine-tuned checkpoints all sit in one loss basin around a shared initialization, and it degrades or vanishes when they do not. Three results pin this down, and all three are load-bearing for your proposal.

Weight disentanglement is the real mechanism. Ortiz-Jiménez, Favero & Frossard, “Task Arithmetic in the Tangent Space,” NeurIPS 2023 (arXiv:2305.12827) argue that arithmetic works because distinct directions in weight space govern distinct, localized regions of function/input space — so you can add and negate them without cross-task interference. They show fine-tuning the linearized model (in the neural-tangent-kernel regime) amplifies this disentanglement and improves arithmetic (reported gains ≈ a few accuracy points on the 8-task suite — treat the exact delta as approximate). The important takeaway for you: disentanglement is a property you can measure, and its presence or absence in scFMs is precisely your paper’s finding.

Linear mode connectivity — the shared-basin prerequisite. Frankle et al., ICML 2020 (arXiv:1912.05671) showed that networks sharing enough early training can be linearly interpolated with no loss barrier — they live in the same basin. Fine-tunes of a common checkpoint inherit this. It is the empirical reason a sum of task vectors lands somewhere useful rather than in a random high-loss region.

Permutation symmetry — why different inits can’t be merged naively. Ainsworth, Hayase & Srinivasa, “Git Re-Basin,” ICLR 2023 (arXiv:2209.04836) showed that independently initialized networks are separated by loss barriers largely because their hidden units are in different permutations; align the neurons and the barrier often collapses. The corollary: weight-space arithmetic presupposes a shared initialization and identical architecture. Break that and ${\theta }_1-{\theta }_{\mathrm{base}}$ is a direction in the wrong coordinate system.

This is exactly the risk your proposal names — “the transcriptomic loss landscape may be insufficiently linear for arithmetic to hold.” It is not a vague worry; it is the specific question of whether scFM fine-tunes exhibit the disentanglement and mode-connectivity that CLIP and GPT-2 fine-tunes do. And it is testable directly (interpolate two same-base specialists; look for a barrier — §21.9).

21.4 A toy that shows the premise is the whole ballgame

Following the book’s habit (20 §20.6, 18 §18.8), here is a small, run demonstration — pure NumPy, a two-layer MLP, in code/demo_task_arithmetic.py. It is not biology; it is the mechanism, isolated. A shared “base” is trained on a generic task; specialists A and B are then full-fine-tuned from that base; task vectors are formed and composed. Then the same is tried with different random initializations to show the premise is not free.

Shared-init regime (seed 0):

Addition recovers a useful but lossy fraction — task A rises 0.28 → 0.66, task B 0.43 → 0.70 — while both stay far below the 0.97 specialists, and it essentially ties naive model soup and joint multi-task training. This is the honest shape of the promise: one model, no joint data, at a real accuracy discount. (The literature’s rosier 98.9%-at-two-tasks is in normalized units on structured vision tasks; this toy uses harsher, unstructured random tasks — the gap is the point, not a contradiction.)

Negation is the clean, selective operation. Start from a model carrying both capabilities, ${\theta }_{AB}={\theta }_{\mathrm{base}}+{\tau }_A+{\tau }_B$ (A=0.69, B=0.66), and subtract ${\tau }_B$:

At $\lambda =1$, task B falls toward chance (0.41; chance ≈ 0.33) while task A is restored to 0.97 — removing B’s interfering direction cleans up A. This is the selectivity your batch-negation application needs, demonstrated in miniature.

The premise is not free. Interpolating the midpoint of two solutions:

• two shared-init task-A solutions → midpoint acc 0.97 (endpoints 0.97 / 0.98) — flat, one basin.
• two different-init task-A solutions → midpoint acc 0.92 (endpoints 0.97 / 0.98) — a barrier appears.

The barrier is mild in this easy, low-dimensional toy (the real effect is far stronger in large nets — that is what Git Re-Basin and Frankle establish), but it is in the right direction: shared init → mergeable; different init → not, without alignment.

What the toy deliberately does not show: clean analogical transfer. My synthetic tasks are independent random read-outs with no shared relational structure, so ${\tau }_A-{\tau }_B+{\tau }_C$ has nothing real to exploit — and inventing a number there would be dishonest. In real scFMs, tissues do share biology, which is exactly why analogy might work on transcriptomes even though it can’t in this toy. That is a hypothesis your experiment tests, not a result the toy can borrow.

The lesson to carry into the scFM setting: addition buys you a lossy generalist; negation is the sharp instrument; analogy needs genuine shared structure; and all of it is contingent on the fine-tunes sharing a basin. Your paper’s real question is whether scFM fine-tunes do.

21.5 Does the scFM setting even satisfy the preconditions?

This is where transferring an NLP trick usually dies (see 20 §20.3, and the type-error in 11_SC_FOUNDATION_MODELS.md §3). Here it mostly survives — with two real caveats.

Good news: the arithmetic is genuinely well-posed. Both anchor models are full-fine-tuned from a shared, published checkpoint:

• scGPT (Cui et al., Nature Methods 2024, doi:10.1038/s41592-024-02201-0) — 12 transformer blocks, dim 512, ~51M params [approx], pretrained on >33M human cells. Its methods state all pretrained weights initialize the fine-tuned model across annotation, integration, and perturbation. Checkpoints on GitHub (bowang-lab/scGPT) / Zenodo, MIT-licensed.
• Geneformer (Theodoris et al., Nature 2023, doi:10.1038/s41586-023-06139-9) — encoder-only, rank-value encoding, variants 10M / 104M / 316M on HuggingFace (ctheodoris/Geneformer). Full fine-tuning is standard for its classification benchmarks.

So ${\theta }_{\mathrm{base}}$ and full-fine-tuned ${\theta }_1,\dots ,{\theta }_K$ are real, shared-init, identical-architecture objects. Unlike the fold-change type-error, there is no category mistake here — the operation is defined.

Caveat 1 — merge within a family, never across. scGPT (value-binning) and Geneformer (rank encoding) have different tokenizations, sizes, and coordinate systems. Treat Geneformer as a replication target (repeat the whole study on it), not as something to merge with scGPT. Cross-family arithmetic is meaningless (§21.3).

Caveat 2 — the skeptic’s trap: is there structure to compose, or is the delta doing all the work? This is the deepest objection and it connects straight to the book’s spine (18_GENOMIC_FM_LANDSCAPE.md §18.4). The critical evaluations are unanimous that scFM bases are weak:

• Kedzierska et al., “Zero-shot evaluation reveals limitations of single-cell foundation models,” Genome Biology 2025 (doi:10.1186/s13059-025-03574-x) — zero-shot scGPT/Geneformer are beaten by plain highly-variable-gene selection on clustering and by Harmony on batch integration.
• Ahlmann-Eltze, Huber & Anders, Nature Methods 2025 (doi:10.1038/s41592-025-02772-6) — for perturbation prediction, no FM beats a simple linear baseline.

The double-edged implication, stated as the analysis it is (not a settled fact): if the base’s representations are weak but full fine-tuning moves the weights a lot, then the task vector may carry almost all the task-specific signal — which cuts both ways. It could mean the deltas are large and information-rich (good for arithmetic), or that each specialist has essentially re-solved its task on top of a near-inert substrate, so the deltas are idiosyncratic and interfere badly (bad for arithmetic). Which of these holds is an empirical question your experiment answers — and it is a genuinely interesting one, because it probes whether scFM “knowledge” lives in a shared, composable substrate or is manufactured fresh per task. Say this explicitly; it is the intellectual core of the paper.

21.6 Prior art — what to cite, and how you go beyond it

The one direct single-cell precedent: the Model Synthesizer. Chen et al., IEEE BIBM 2025 (doi:10.1109/BIBM66473.2025.11356145) decompose weight matrices into Fourier phase and amplitude, and enhance a data-constrained scRNA-seq model by keeping its phase while fusing in the amplitude of a model pretrained on another modality (text). It establishes that weight-space fusion is viable on single-cell models — cite it as the closest prior art. But it leaves your core question open in three specific ways, and naming them is your novelty statement:

1. It performs only additive amplitude fusion; it never constructs task vectors $\tau ={\theta }_{\mathrm{ft}}-{\theta }_{\mathrm{base}}$ and never tests the algebra — negation and analogical composition.
2. Its setting is cross-modal amplitude borrowing (text → scRNA-seq), not same-modality composition of two specialists of one single-cell base.
3. It uses a BERT-Small backbone configured Geneformer-like, not published scGPT or Geneformer checkpoints.

Integrity flag. As of writing, the Model Synthesizer is very recent and not yet indexed on the open web (the DOI resolves to IEEE Xplore but the text is gated; no Semantic Scholar / Scholar cache hit). Cite it from the PDF you hold, state that it is unreplicated, and — per your own plan — run a forward-citation check on the task-arithmetic literature immediately before submission, because this is a fast-moving area and a same-modality scFM task-arithmetic paper could appear at any time.

Adjacent single-cell work to acknowledge:

• scPEFT (Nature Machine Intelligence 2025, doi:10.1038/s42256-025-01170-z) — LoRA/adapter fine-tuning matches or beats full fine-tuning on scGPT for annotation. Relevant because it means LoRA task vectors exist and can be merged — a lighter, cheaper route to your ${\tau }_i$ (see §21.7, “way out”).
• The conceptual “compositional foundation models for cell biology” argument (Cell Systems, 2026) — argues for composition in principle but does no weight-space arithmetic. Cite as motivation, distinguish on method.

The method toolbox you inherit from NLP/vision (none applied to biology — this is the gap): Model Soups (Wortsman et al., ICML 2022, arXiv:2203.05482); Fisher-weighted merging (Matena & Raffel, NeurIPS 2022, arXiv:2111.09832); TIES-Merging (Yadav et al., NeurIPS 2023, arXiv:2306.01708); DARE (Yu et al., ICML 2024, arXiv:2311.03099); AdaMerging (Yang et al., ICLR 2024, arXiv:2310.02575); and compressed task-vector bases (Li et al., TMLR 2025, arXiv:2502.01015).

21.7 Don’t test plain summation — the “way out” for interference

The single most common way this paper gets a weak result is using ${\theta }_{\mathrm{base}}+\lambda {\sum }_i{\tau }_i$ as the method. That is the 2022 algorithm; its interference is exactly what three years of follow-up work exists to fix. Use these as your real merges, and the framing shifts from “does the naive trick work?” to “which merge operator best exposes the composability of scFM weights?”

The clean experimental design is a merge-operator ablation: plain task arithmetic → soup → TIES → DARE+TIES → AdaMerging, on the same specialists, same eval. The shape of that ladder is itself a finding about scFM weight geometry.

21.8 Implementation and downstream tasks — a concrete example

Here is a runnable program, mirrored on the operations in §21.1, grounded in real datasets.

Base and specialists. Start from the published scGPT whole-human checkpoint as ${\theta }_{\mathrm{base}}$. Full-fine-tune one specialist per tissue for cell-type annotation (scGPT’s standard downstream head), on clean, comparable label spaces:

• Use Tabula Sapiens (Quake et al., Science 2022, doi:10.1126/science.abl4896) — ~500k cells across ~24 organs with shared preprocessing — to build, say, specialists for blood, lung, pancreas, heart. Or use scGPT’s own annotation benchmarks (hPancreas, Myeloid, M.S., Zheng68K) as the $K$ tasks.

This yields ${\theta }_{\mathrm{blood}},{\theta }_{\mathrm{lung}},{\theta }_{\mathrm{pancreas}},{\theta }_{\mathrm{heart}}$ and their task vectors ${\tau }_t={\theta }_t-{\theta }_{\mathrm{base}}$.

Worked example — the three operations:

• Addition (a cross-tissue generalist). ${\theta }_{\mathrm{add}}={\theta }_{\mathrm{base}}+\lambda {\sum }_t{\tau }_t$ (and its TIES/DARE variants). Question: does one merged annotator label cells across all four tissues at a useful fraction of the four specialists — without pooling the data or paying 4× storage/inference? Expected: yes, lossily (the toy’s 0.68-vs-0.97 shape), with TIES/DARE narrowing the gap.
• Negation (batch-correction by subtraction — the flagship experiment). Batch/donor effects are the central nuisance in scRNA-seq (01_BACKGROUND.md; benchmarked by scIB — Luecken et al., Nature Methods 2022, doi:10.1038/s41592-021-01336-8). Construct a task vector ${\tau }_{\mathrm{batch}}$ by fine-tuning the base to predict batch/donor identity (i.e., to lean into the confound), then evaluate ${\theta }_j-\lambda {\tau }_{\mathrm{batch}}$: does subtracting the “batch direction” improve integration while preserving cell-type signal? Baselines to beat/complement: Harmony (Korsunsky et al., Nature Methods 2019), scVI, scANVI, scored on the scIB batch-removal vs bio-conservation metrics. A weight-space batch corrector that needs no per-dataset re-embedding would be genuinely novel — and negation is where the toy said your signal is cleanest.
• Composition/analogy (transfer to a held-out tissue). ${\theta }_{\mathrm{comp}}={\theta }_{\mathrm{base}}+\lambda ({\tau }_A-{\tau }_B+{\tau }_C)$ to annotate a tissue you never fine-tuned on. Expected: the riskiest operation; report it honestly, including failure. If it works at all, it works because tissues share biology (the structure the toy lacked).

Compute. This is deliberately cheap: specialists are single-GPU full-fine-tunes of a ≤316M model (or LoRA via scPEFT, cheaper still); merging is free (weight arithmetic); AdaMerging needs only unlabeled cells. The whole study fits the “graduate-student experiment, not a data-center one” budget (18 §18.8).

21.9 Baselines and evaluation

Baselines are the credibility of the paper (18 §18.7; 20 §20.6). Non-negotiable set:

1. Each individual specialist (the ceiling; normalized accuracy is relative to these).
2. Joint multi-task fine-tuning on pooled data (the “if you had all the data and compute” reference).
3. Naive uniform weight averaging (model soup, 2203.05482).
4. Zero-shot base (the floor).
5. For the batch-negation claim: Harmony / scVI / scANVI under scIB metrics — not optional.
6. Stronger merges (TIES, DARE, AdaMerging) as method, not just baselines.

Metrics and curves:

• Normalized accuracy — merged / specialist, averaged over tasks; 1.0 = “as good as the experts.”
• The $\lambda$ trade-off curve — per-task and average accuracy vs $\lambda$, the whole surface as tasks are co-merged (2 → K).
• Negation selectivity — target task degraded while others preserved; report both, not just the drop (the toy’s two-column table is the template).
• Transfer accuracy to held-out tissues for the composition test.
• Stability across random seeds and fine-tuning orders (order-sensitivity would itself be a red flag about basin structure).
• A direct mode-connectivity probe — interpolate two same-base specialists and look for a loss/accuracy barrier (§21.3). A flat interpolation is positive evidence for composability before you even merge; a barrier predicts arithmetic will struggle and is a clean negative result on its own.

The data-hygiene rule your proposal already states, kept: split so that no cells from a fine-tuning condition appear in the evaluation of a different task, and treat contamination/leakage with the same suspicion as Chapter 20 (20 §20.5) — otherwise a “transfer” is just overlap.

21.10 The critics’ strongest objections — and the way out

21.11 Novelty, and what publishes either way

The honest novelty statement: nobody has constructed task vectors from real, same-base scFM checkpoints and tested the full algebra — addition, negation (for confound removal), and analogical composition — with interference-aware merges and a mode-connectivity analysis. The Model Synthesizer did additive, cross-modal amplitude fusion on a BERT-Small backbone; it did not do task vectors, negation, analogy, same-modality composition, or published scGPT/Geneformer weights. That gap is the paper.

Why both outcomes are citable (the Chapter 18 §18.6 discipline):

• If it composes: first demonstration that task-arithmetic algebra transfers to scFM weights → reusable “capability modules” for a shared base, plus a weight-space batch-correction primitive.
• If it does not: rigorous evidence that scFM fine-tunes lack the linear mode connectivity language models have → explains the field’s reliance on full fine-tuning, and warns against naive checkpoint reuse. This is a Nature-Methods-2025-style negative result (18 §18.4), and those get read.

Venue fit: a machine-learning-for-genomics venue (ISMB/RECOMB, Bioinformatics, Genome Biology) if led by the biology; a NeurIPS/ICLR workshop on model merging if led by the method and the geometry analysis. The dataset/benchmark framing (a released suite of same-base scFM specialists + a merge-evaluation protocol) is the safest, because — as in Chapter 20 — the resource and the characterization are bulletproof even if the merged model underperforms.

21.12 Verdict

Bottom line. Your instinct is sound and — unlike the fold-change type-error — the operation is genuinely defined on scFM weights, so this is one of the rare “transfer an NLP capability” ideas that clears the first bar. Run it, but (1) frame the deliverable as the first map of the scFM fine-tuning landscape, where a negative result is a real finding; (2) make negation-as-batch-correction the flagship, because that is where the signal is cleanest and the novelty sharpest; (3) use interference-aware merges, not plain summation, or you are testing the wrong algorithm; and (4) confront the skeptic’s question — is there a composable substrate, or is each delta doing all the work? — head-on, because that question, not the merged model’s accuracy, is what makes the paper interesting.

This chapter is analysis and planning, not a result. The task-arithmetic mechanics and reported numbers trace to the primary papers (Ilharco 2212.04089; Ortiz-Jiménez 2305.12827; Frankle 1912.05671; Ainsworth 2209.04836; TIES 2306.01708; DARE 2311.03099); the scFM facts to scGPT (Cui, Nat. Methods 2024) and Geneformer (Theodoris, Nature 2023); the skepticism to Kedzierska (Genome Biol. 2025) and Ahlmann-Eltze (Nat. Methods 2025) — the same findings Chapter 18 §18.4 leans on. The toy in code/demo_task_arithmetic.py is illustrative NumPy, deliberately non-biological, and its analogy result is intentionally omitted rather than faked. The Model Synthesizer (BIBM 2025) is recent and unreplicated on the open web; verify it, and run a forward-citation check on the task-arithmetic literature, before submission.