Chapter 25 — Reading EVA at Author Depth (Transcriptomic Choices Only)

Fourth and last of the model readings, and read differently from the others by design: EVA is a multimodal model (transcriptomics + histology), but this chapter deliberately ignores the imaging and fusion machinery and looks only at the transcriptomic tower, EVA-RNA, asking one question — what does EVA do to a gene-expression model that earlier ones didn’t, and which of those ideas should the next transcriptomic foundation model steal? Two corrections up front, because they reframe everything. First, despite being benchmarked against scGPT/Geneformer/scFoundation/UCE, EVA-RNA is not a single-cell model — it operates on sample-/patient-level transcriptomes (bulk RNA-seq, microarray, and pseudobulked single-cell), so it is closer to BulkFormer (Chapter 24) than to scGPT, and its “wins over single-cell FMs” are partly a home-field comparison (§25.6). Second — and this is why EVA belongs in this book — its genuinely portable innovations are exactly the fixes Chapters 22–24 kept proposing: magnitude-preserving continuous values, a count-aware (ZINB) objective, knowledge-initialized gene embeddings, and an ortholog-unified cross-species vocabulary. EVA is, in effect, a proof-of-concept that several of the book’s recommendations work when combined.

One-sentence version: EVA-RNA is a ~305M-parameter encoder over patient-level transcriptomes that (a) seeds every gene’s identity vector from five external knowledge sources instead of random noise, (b) unifies human and mouse genes through orthology so knowledge transfers across species, (c) embeds expression as a continuous value and reconstructs masked genes with a zero-inflated negative-binomial likelihood — a bundle of representational choices that matter more than its (in-house, immunology-scoped) benchmark numbers suggest.

25.1 What problem EVA-RNA solves

The data type. Not single cells. EVA-RNA consumes a whole-sample transcriptome — one expression vector per patient/sample — drawn from three technologies at once: bulk RNA-seq, microarray (an older hybridization-based expression platform), and pseudobulk (a single-cell dataset collapsed back to one per-sample vector by summing its cells). This “many platforms, one sample-level representation” is the defining scope choice.

The gap it targets. The authors argue prior transcriptomic FMs have two problems for drug development: they live at single-cell resolution and so cannot natively emit a patient-level phenotype (the unit a clinical decision is actually about), and they are evaluated on technical metrics (clustering, label transfer) “disconnected from actual drug development.” EVA targets patient-level, cross-species, cross-platform representations, judged on drug-pipeline tasks (target efficacy, patient stratification, treatment response).

What one training example is. One sample = a set of (gene ID, expression value) pairs — log-CPM for RNA-seq, log-transformed for microarray — prepended with a CLS token, with a random subset of gene values masked for the pretraining task. No metadata tokens.

Input → output. In: genes + values (some masked). Out: per-gene contextual embeddings and a CLS sample embedding (the patient-level vector everything downstream uses); the pretraining head predicts the masked genes’ expression distribution.

Downstream tasks. A 39-task drug-development suite — zero-shot for the transcriptomic tasks — spanning target-efficacy prediction, gene-function prediction, molecular-perturbation response, patient stratification into endotypes, cross-species treatment effect, and clinical treatment outcome (§25.5).

25.2 How the input is represented (where EVA’s real ideas live)

This section carries most of the chapter’s payload, because EVA’s durable contributions are representational. Start from a sample (log-CPM values):

Each gene becomes identity embedding + value embedding, summed — the familiar §22.2 pattern — but both halves are upgraded.

Idea 1 — knowledge-initialized gene embeddings (EVA’s most reusable trick)

Recall from §22.2 that scGPT’s gene table starts as random noise and must learn everything from expression alone. EVA does the opposite: it warm-starts each gene’s identity vector by concatenating five external descriptions of that gene and compressing them:

1. scGPT’s learned gene embeddings (prior-model knowledge),
2. ESM-2 protein-language-model embeddings (what protein the gene makes),
3. NCBI gene text descriptions (natural-language biology),
4. UniProt protein descriptions (more text/functional annotation),
5. RotatE knowledge-graph embeddings (the gene’s position in a curated interaction graph).

These are PCA-reduced to a shared size (256 for the 305M model) and mapped through a 2-layer MLP to model width. So before EVA sees a single expression value, every gene already “knows” its protein, its literature description, and its graph neighborhood. The paper’s ablation is the important part: using all five sources gives the fastest convergence and lowest final validation loss versus fewer or random init. This is the generalization of the ESM-2 idea BulkFormer used (§24.2) and the scGPT chapter proposed (§22.8-4) — EVA just uses five priors instead of one, and shows it helps.

Idea 2 — an ortholog-unified cross-species vocabulary

The vocabulary is 66,240 NCBI Gene IDs spanning human and mouse together, linked by 16,168 human–mouse ortholog pairs. So mouse Il6 and human IL6 are not unrelated tokens (scGPT’s problem, §22.7) — they are tied through orthology and share knowledge-initialized neighbors. This is a first-class design choice for cross-species transfer rather than an afterthought, and EVA shows (§25.6) that over training the mouse and human representations actively converge.

Idea 3 — continuous value embedding (no binning)

The expression value is passed through an MLP [1 → 16 → 128 → 384 → 768] + LayerNorm — a continuous scalar mapped to a 768-dim vector. No binning, no rank encoding. Magnitude is preserved, exactly the fix §22.8-1 and Chapter 24 argued for.

Special tokens, platforms, metadata

• Special tokens: CLS (sample readout), MASK, PAD.
• Cross-platform handling: microarray, bulk, and pseudobulk are trained jointly, with per-modality normalization (CPM+log for RNA-seq, log-only for microarray). Notably there is no explicit platform/batch token ⚠️ — the model is expected to learn technology-invariance implicitly from the mixed corpus. (Contrast scGPT’s explicit batch condition token; EVA bets that scale + diversity beats an explicit correction knob.)
• Metadata: none conditioned beyond normalization ⚠️.
• Context length / per-sample gene count: not disclosed in the accessible text ⚠️ — do not assume a number.

What is preserved or lost

25.3 How the architecture works (transcriptomic tower)

The architecture is deliberately ordinary — which is itself a finding: EVA’s gains come from representation and data, not a novel attention mechanism.

The encoder. EVA-RNA is a 32-layer transformer encoder, hidden dim 768, 12 heads, FFN 3,072, pre-LayerNorm, with residual scaling $1/\sqrt{2L}$ for training stability — about 305M parameters. Attention is standard self-attention; no FlashAttention/sparse/graph trick is credited as core (contrast BulkFormer’s Performer+GCN, §24.3). The prepended CLS token’s final hidden state is the sample embedding; per-gene contextual vectors are also produced.

Shapes (one sample, $G$ genes, $d=768$):

Multimodal connection (set aside per the chapter’s remit). In the full system, EVA-RNA (300M) + a histology encoder (85M) + a fusion head (55M) ≈ 440M; the RNA tower’s CLS embedding is aligned to imaging via a contrastive loss. But EVA-RNA is a standalone encoder and is released/usable transcriptomics-only — which is the version this chapter reads. I note the fusion exists and move on.

25.4 How the model is trained

The objective — masked expression with a count likelihood

The primary objective is masked gene-expression prediction, as in scGPT — but the decoder is the notable part. Instead of MSE on a value or a bin, EVA reconstructs each masked gene by predicting a Zero-Inflated Negative Binomial (ZINB) distribution, minimizing its negative log-likelihood:

$${\mathcal{L}}_{\text{mask}}=-\sum _{i\in \mathcal{M}}\log \mathrm{ZINB}(x_i\mid {\pi }_i,{\mu }_i,{\theta }_i),$$

where the head outputs, per gene, a zero-inflation probability $\pi$, a mean $\mu$, and a dispersion $\theta$. Why this matters: transcriptomic counts are sparse (many zeros, some technical) and overdispersed (variance ≫ mean), and a ZINB models exactly those two facts — the same principled choice scVI uses and that §22.8-1 and §23.8 argued for, here applied at the pretraining objective itself rather than bolted on. A secondary CLS-reconstruction/compression term and a (multimodal, not-our-focus) multi-positive InfoNCE contrastive loss round it out.

The data — large, but mouse-bulk-heavy and immunology-scoped

EVA-RNA pretrains on 545,343 samples ≈ 330 billion gene tokens (~20.5 effective epochs) from an in-house corpus, ImmunAtlas (>50 tissues/conditions, immunology/inflammation-focused, human + mouse, normal + disease). The composition matters and should be stated plainly:

So EVA is mouse-bulk-dominated and immunology-specialized — a critical caveat for reading both its cross-species claims (the mouse side is where most data is) and its generality (it is not pan-disease).

What it should teach, and the shortcuts

The objective is meant to teach cross-gene co-expression structure that generalizes across platform and species. The same shortcuts as every masked-expression model apply: predict-the-conditional-mean (a ZINB head with good $\mu$ can score well without deep reasoning), and — because pretraining is observational — association, not intervention, so perturbation/treatment tasks remain the hardest (and, per §25.6, the ones where EVA’s margin nearly vanishes). A shortcut specific to the mixed corpus: with 80% mouse bulk, the model can over-fit mouse-immunology structure and lean on it — the interpretability analysis (§25.6) is partly there to argue this didn’t happen.

25.5 How it is used for downstream tasks

EVA’s transcriptomic tasks are mostly zero-shot: freeze the model, take the CLS sample embedding (or gene embeddings), attach a light readout. Concretely, in the book’s in/out format:

Zero-shot target efficacy.

• In: a disease sample’s expression → CLS embedding; a candidate drug target (gene).
• Out: a score for whether modulating that target would help. Result: EVA 0.693 vs scGPT 0.539 vs a statistical baseline 0.569.

Patient stratification into endotypes.

• In: many patients’ sample embeddings.
• Out: clusters/labels grouping patients by molecular subtype (“endotype”). Result: 0.786 vs scGPT 0.706.

Cross-species treatment effect (the transfer test).

• In: a mouse model-system profile + a treatment.
• Out: predicted effect transferable to human. Result: EVA 0.445 vs scGPT 0.439 vs BulkRNABert 0.435 — essentially a tie, and the most honest number in the paper (cross-species transfer is hard even for a model built for it).

Molecular perturbation / clinical outcome. Predict a gene-perturbation response (0.547) or a clinical treatment outcome (0.650), from the sample embedding + condition.

Toxicity relevance

EVA is, with BulkFormer, one of the two most toxicogenomics-adjacent models in this series, and for overlapping reasons: it is bulk/pseudobulk (the tox modality), magnitude-preserving, and — uniquely here — explicitly cross-species (human↔mouse). Toxicogenomics is fundamentally cross-species (rat/mouse → human), so EVA’s ortholog-unified vocabulary + knowledge-initialized embeddings are the closest existing demonstration of the rat↔human bridge the book keeps proposing (20_REASONING_DATASET_IDEA.md §20.6). The gaps for tox are the now-familiar ones: rat specifically (EVA is human+mouse), dose/time conditioning (absent), fold-change vs control (it uses absolute log-CPM), and it is immunology-scoped rather than liver/kidney-tox. So EVA supplies the transfer machinery toxicogenomics needs without being aimed at tox.

25.6 What is technically distinctive — and why “beats single-cell FMs” needs an asterisk

The user question this chapter exists to answer: what makes EVA better, and is it real?

The design choices the authors credit:

1. Multi-technology, multi-resolution training (microarray + bulk + pseudobulk, human + mouse) → cross-platform, patient-level representations.
2. Knowledge-initialized gene embeddings (5 sources; ablated to help most).
3. ZINB distributional decoder (models sparsity + overdispersion).
4. Ortholog-unified cross-species vocabulary (translational mouse→human).
5. Continuous value embedding (no binning).
6. Decision-relevant evaluation (39 drug-development tasks, not clustering metrics).

The cross-species evidence is genuinely interesting. EVA backs its transfer claim with mechanistic interpretability, not just a score: over training, (a) mouse genes’ input embeddings move closer to their human orthologs (nearest-neighbor rank drops), (b) contextual embeddings shift from per-species clusters early → integrated late, and (c) a sparse autoencoder finds 82 of 1,383 learned “concepts” shared across species and technologies, with immune genes aligning most strongly. This is better evidence of real integration than a single benchmark number — worth borrowing as a validation method (§25.8).

Now the asterisk — be skeptical in the book’s way (18 §18.7). Several things confound the “beats scGPT/Geneformer/UCE” headline:

• It is not the same kind of model. EVA-RNA is sample-level; scGPT/Geneformer/UCE are single-cell. In these comparisons the single-cell models are run off-distribution (asked to emit patient-level representations they were never designed for). A single-cell FM losing at a bulk/patient task is unsurprising and not, by itself, evidence of a better architecture.
• Home-field benchmark. The 39-task suite (PRIMO) is also authored by Scienta, and the tasks are immunology/drug-development framed — EVA’s home domain. Both the model and its yardstick are in-house.
• Domain-matched data. EVA trains on immunology data and is tested on immunology tasks; the baselines are not domain-matched. Much of the margin is plausibly data-and-task fit, not method.
• The transfer number is a tie (0.445 vs 0.439/0.435) — the one head-to-head where the comparison is fairest, and EVA barely wins.
• All results are developer-reported, Feb-2026-new, with no independent replication, and the released open weights are a 48.6M model, smaller than the 305M described ⚠️.

So what is genuinely portable? Strip away the data advantage and the multimodal machinery, and what remains are representational choices that were ablated and are architecture-agnostic: knowledge-initialized gene embeddings, the ortholog-unified vocabulary, the ZINB decoder, and continuous value embedding. Those survive the skepticism — they would help a single-cell model, a bulk model, or a toxicogenomics model regardless of EVA’s benchmark. That is the honest distillation, and it is the whole reason to read EVA.

Scaling (a real, clean result). EVA reports a power law $L=2.515C^{-0.032}$ — about 7% validation-loss reduction per 10× compute — with no plateau at 305M (explicitly contrasted with a competitor plateauing at 100M), and pretraining gains that transfer to downstream tasks. This is one of the cleaner scaling demonstrations for a transcriptomic model and is not confounded by the benchmark issues, since it’s measured on held-out loss.

25.7 Limitations and research gaps

Scope. Immunology/inflammation only — not pan-disease; and the corpus is ~80% mouse bulk, so both generality and the human side of cross-species transfer rest on a minority of the data.

Representation. No explicit platform/batch modeling ⚠️ (invariance is implicit — untested against a dedicated batch corrector like Harmony); absolute log-CPM, not fold-change; no dose/time/compound conditioning; context length undisclosed ⚠️.

Modality. It is not single-cell — the pseudobulk step discards within-sample cell structure, so cell-composition effects (central to both immunology and toxicology) are confounded, as in bulk generally (§24.7).

Evaluation. In-house model and in-house benchmark; baselines run off-distribution; no independent replication; released model smaller than reported. The strongest independent-of-benchmark evidence is the interpretability analysis and the scaling curve, not the task scores.

Mechanism vs statistics. As with the others, mostly statistics — a strong cross-platform co-expression model with real (interpretability-supported) cross-species alignment, but the perturbation/treatment tasks (the causal ones) are where its margin collapses.

Where it fails out-of-distribution. Rat (human+mouse only — the tox species is absent); dose–response and time-course (no representation); fold-change inputs (never seen); non-immunology tissues/diseases at the depth it has for immunology; and single-cell tasks (it gave up that resolution by design).

25.8 Ideas to extract for future transcriptomic foundation models

Per the chapter’s remit, this is the deliverable: the transcriptomic ideas worth borrowing, each as the idea → why it helps → how to port → what it buys toxicogenomics. These are the parts of EVA that survive the §25.6 skepticism because they are ablated, representational, and independent of the multimodal system and the home-field benchmark.

1. Warm-start gene embeddings from multiple knowledge sources

• Why. Random gene tables must learn everything from expression; EVA’s ablation shows five priors (protein sequence, text, knowledge graph, prior-model) converge faster to lower loss.
• How to port. Initialize any scFM/bulk-FM’s gene identity table by concatenating ESM-2 + NCBI/UniProt text embeddings + a knowledge-graph embedding (RotatE) (+ optionally an existing model’s gene vectors), PCA-reduce, MLP to width. Fully architecture-agnostic.
• For tox. The text/KG priors carry mechanism knowledge (pathways, targets) that a small tox corpus can’t teach — a cheap, strong prior for DrugMatrix/TG-GATEs-scale data.

2. Ortholog-unified cross-species vocabulary as a first-class design

• Why. Tying mouse↔human genes through orthology (not siloed vocabularies) is what let EVA’s species representations converge over training.
• How to port. Build one vocabulary over the union of species’ genes, linked by ortholog pairs, with knowledge-init embeddings (so orthologs start close via shared protein sequence). Add rat for tox.
• For tox. This is the rat↔human bridge the book keeps proposing (20 §20.6; 22 §22.8-4; 24 §24.8-3) — EVA shows it works for mouse↔human, and the ESM-2 route makes adding rat mechanical.

3. A ZINB (count) decoder for the masked-expression objective

• Why. Models sparsity + overdispersion of real counts; better-calibrated than MSE-on-values or MSE-on-bins.
• How to port. Replace the reconstruction head with a ZINB NLL (predict $\pi ,\mu ,\theta$ per gene). Works per-cell (single-cell) or per-sample (bulk).
• For tox. Combined with a fold-change head (24 §24.8-1), a count-aware objective is the magnitude-faithful target dose–response needs.

4. Continuous value embedding instead of binning/ranking

• Why. Keeps magnitude, which dose/effect-size tasks require; EVA and BulkFormer both chose it over binning.
• How to port. A small MLP [1 → … → d] on the (log) value, summed with gene identity. Trivial to adopt; removes scGPT’s §22.2 magnitude loss.

5. Cross-resolution training (bulk + pseudobulk + single-cell together)

• Why. Bridges single-cell mechanism and patient-level phenotype in one representation — a resolution pure scFMs lack.
• How to port. Mix single-cell, pseudobulk, and bulk in pretraining with per-resolution normalization (no separate models). For tox, this could unify single-cell mechanism data with the bulk DrugMatrix/TG-GATEs profiles that regulators actually use.

6. Decision-relevant evaluation, and interpretability-based validation

• Why. EVA’s most defensible evidence is not its (home-field) task scores but its scaling curve and its mechanistic interpretability (ortholog-rank alignment, shared SAE concepts) — proof that cross-species/platform integration is real, not benchmark-fit.
• How to port. For a tox FM: evaluate on decision-relevant endpoints (does it predict the regulatory hazard call?) and validate cross-species integration with the same interpretability probes (do rat and human orthologs align in the learned space?), rather than trusting a single benchmark.

The synthesis this sets up. Notice that EVA is essentially Chapters 22–24’s wish-list assembled in one model: continuous magnitude (fixing scGPT’s binning), a count likelihood (fixing MSE), knowledge-initialized + ortholog-unified embeddings (fixing the species wall), cross-platform/-resolution training (fixing modality silos). It stops short of toxicogenomics on four specific axes — rat, dose, time, fold-change — and it proves those choices out on immunology data with an in-house benchmark. That is the perfect hand-off to the comparison chapter, which lines up all four models on exactly these axes and asks which combination the next transcriptomic (and toxicogenomic) foundation model should actually be.

This chapter is exposition, not a result, and covers only EVA’s transcriptomic tower. Facts are from Bandasack et al., “EVA: Towards a universal model of the immune system,” arXiv:2602.10168 (Feb 2026; earlier bioRxiv 2025.05.02.651839), and the ScientaLab HuggingFace release: EVA-RNA ≈ 305M params (32 layers / 768 dim / 12 heads), 66,240-gene human+mouse vocabulary with 16,168 ortholog pairs, five-source knowledge-initialized gene embeddings, continuous value MLP, ZINB masked-expression decoder, 545,343 training samples (~80% mouse bulk) from the immunology-focused ImmunAtlas. The crucial framing corrections, made throughout: EVA-RNA is a bulk/pseudobulk patient-level model, not single-cell; its wins over single-cell FMs are partly confounded by off-distribution baselines and an in-house (PRIMO) benchmark; and its durable, portable contributions are representational. Uncertain items flagged ⚠️: context length, absence of explicit platform tokens, and the released 48.6M weights being smaller than the described 305M model. All metrics are developer-reported with no independent replication. Next: Chapter 26 — the comparison, lining up scGPT, Tahoe, BulkFormer, and EVA on the axes toxicogenomics needs, and naming the most realistic research directions.