RLVR — Reinforcement Learning with Verifiable Rewards — Deep Dive

Frontier-lab research-scientist depth on RLVR: the algorithm zoo (DAPO, Dr. GRPO, RLOO, REINFORCE++, GSPO, VinePPO, PRIME), verifier design, reward shaping, failure modes, open-source infra, and a substantial section on low-resource multilingual reasoning — the most under-explored applied RLVR direction in 2025-2026, with concrete research-project blueprints.

Pair with REASONING_MODELS_DEEP_DIVE.md (the broader reasoning-RL story), FRONTIER_REWARD_MODELING.md (RM landscape), and OPEN_SOURCE_POSTTRAIN_PLAYBOOKS.md (R1, Tülu 3 recipes).

RLVR — what it is, why it works, why it’s special
The 2024-2025 RLVR algorithm zoo
Verifier design — the often-neglected half
Reward shaping — correctness, format, length, language, step
KL regularization choices
Curriculum, exploration, and warm-start
Common failure modes and debugging
Open-source RLVR infrastructure
Multi-modal RLVR — math, code, tool use, vision
Low-resource multilingual reasoning with RLVR
Concrete research project blueprints
Datasets and benchmarks
Open frontier questions
References

1. RLVR — what it is, why it works

RLVR = train a policy via RL where the reward signal comes from a deterministic verifier on the model’s output, not from a human-feedback reward model.

Examples of verifiable rewards:

Math final answer matches sympy-canonical ground truth.
Generated code passes unit tests.
Tool-use trace ends in the correct final state.
Multiple-choice extraction matches the labeled letter.
Lean / Coq verifier accepts a generated formal proof.
A regex extracts the right structured answer.

Why it’s special.

Zero labeler cost. Verifiers are programs; they scale to millions of problems for free.
Zero labeler bias. No length bias, sycophancy, format bias, or persona bias from the reward signal.
Hard to reward-hack on outcome. The model can’t write more text, be more polite, or use bullet points to get a higher score; only correctness matters.
Strong gradients on hard problems. When the policy gets a problem right, it knows; when it doesn’t, it knows. Crisp signal.
Composes with non-verifiable rewards. Use RLVR for the verifiable subset, RLHF/RLAIF for the rest.

Why it works. Pretraining gives the model priors over reasoning steps. RL turns those priors into a search policy by rewarding correct outcomes. The policy learns to allocate inference compute (long CoT) on hard problems and short answers on easy ones — discovered, not engineered.

The one-line takeaway: RLVR is the cleanest signal in modern post-training, and reasoning RL is currently the highest-impact application.

Saying it out loud. RLVR is reinforcement learning where a program, not a person, grades the answer. Run the unit tests, check the boxed number against sympy, hand the proof to Lean — if it passes, reward one, otherwise zero. That one substitution fixes most of what’s wrong with preference-based RLHF: no labeller cost, so you can scale to millions of problems; no labeller bias, so no length bias or sycophancy leaking into the reward; and the model can’t get a better score by being more polite or using more bullet points. What the model learns from that is a search policy — it discovers on its own that spending more tokens on hard problems pays off, which nobody engineered. The binding constraint, and the thing to name as the tradeoff, is coverage: only a narrow slice of useful behaviour is verifiable, so everything outside math, code and formal tasks still needs a judge model or human preferences.

2. The 2024-2025 RLVR algorithm zoo

The base policy gradient is REINFORCE; everything else is variance reduction, sample efficiency, or robustness. Memorize this list.

2.1 PPO with value head

The classical RLHF setup. Policy + critic (value head). Advantage = reward + GAE bootstrap − baseline.

Pros. Lower variance via critic. Cons. Critic has to be trained, doubles memory. Critic is often poorly calibrated for sparse / sequence-level rewards.

2.2 GRPO (Group Relative Policy Optimization, DeepSeek)

Plain-language gloss. “Group” here means G different completions sampled for the same prompt — nothing to do with groups of users, demographics, or data categories. The advantage for one completion is simply how much better its reward was than the average of its siblings on that same question, standardised by their spread. That group average is doing the job PPO’s learned value network used to do, which is why GRPO can drop the critic entirely.

For a prompt $x$ , sample $G$ rollouts. Use the group’s mean reward as the baseline; standardize by group std. No critic.

$\hat{A}_{i, t} = \frac{r _{i} - mean ({ r _{1} , \dots , r _{G} })}{std ({ r _{1} , \dots , r _{G} })} .$

Pros. No critic; sample-efficient; particularly good for verifiable-reward settings (the reward is sequence-level, so a token-level critic is wasteful). Cons. Group size adds compute; the std normalization is contentious (Dr. GRPO removes it).

Saying it out loud. GRPO is PPO with the value network deleted. Instead of training a critic to predict how good a state is, you sample a group of completions — call it G of them — for the same prompt, and use the group’s own mean reward as the baseline. So an advantage is just “did this rollout do better or worse than my other attempts at this same question”, optionally divided by the group’s standard deviation. That works because in RLVR the reward is a single number at the end of the sequence, so a per-token critic was mostly wasted machinery anyway — and dropping it halves your memory and removes a component that was poorly calibrated on sparse rewards. The tradeoff is compute: you now pay G rollouts per prompt instead of one, and if all G get the same reward the group contributes no gradient at all, which is exactly the problem DAPO’s dynamic sampling exists to fix.

2.3 Dr. GRPO

GRPO without the std normalization in the advantage. Argued to be more stable when reward variance correlates with problem difficulty (you don’t want easy-problem easy-wins to dominate via small std).

$\hat{A}_{i, t} = r_{i} - mean ({r_{1}, \dots, r_{G}}) .$

2.4 DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization, ByteDance 2025)

Four contributions stacked on GRPO:

Clip-Higher. Asymmetric clip — clip(ρ, 1−ε_low, 1+ε_high) with ε_high > ε_low. Lets the policy take bigger steps on positive-advantage tokens (encouraging exploration) without amplifying negative-advantage updates.
Dynamic Sampling. Filter rollout groups where all rewards are equal (no learning signal); resample.
Token-Level Policy Gradient Loss. Sum over tokens not over sequence length-normalized. Long correct answers should not be down-weighted just because they’re long.
Overlong Reward Shaping. Penalize CoTs that hit the max-length cap (often correlated with non-termination / failure).

DAPO results on AIME match or exceed GRPO with significantly less compute. The current canonical algorithm for reasoning RL outside DeepSeek’s stack.

Saying it out loud. DAPO is four small fixes to GRPO that together matter a lot. Clip-higher makes the PPO clipping range asymmetric, so the policy can take bigger steps toward things that worked without equally amplifying the push away from things that didn’t — that’s an exploration fix, aimed at entropy collapse. Dynamic sampling throws away groups where every rollout got the same reward, because those contribute exactly zero gradient and you paid for them anyway. Token-level loss stops long correct answers from being down-weighted just for being long, which is a length-bias fix hiding in the normalisation. And overlong shaping penalises chains that hit the context cap, since those are usually non-termination failures rather than deep thinking. The reason to know it is that it matches or beats GRPO on AIME at substantially less compute, and it’s the default outside DeepSeek’s own stack.

2.5 RLOO (REINFORCE Leave-One-Out, Kool 2019; revived for RLHF by Ahmadian 2024)

Like GRPO but the baseline for sample $i$ is the mean of the other samples in the group:

$\hat{A}_{i} = r_{i} - \frac{1}{G - 1} j \neq = i \sum r_{j} .$

Unbiased; no critic; simple. Ahmadian et al. (“Back to Basics”) showed RLOO matches or beats PPO on RLHF benchmarks.

Saying it out loud. RLOO is the same group-baseline idea as GRPO with one difference that matters theoretically: the baseline for each sample is the mean of the other samples in the group, excluding itself. Leaving yourself out is what makes the estimator unbiased — if you include your own reward in the baseline you’re subtracting something correlated with what you’re estimating. No critic, no standard-deviation normalisation, about four lines of code. Ahmadian’s “Back to Basics” paper showed it matches or beats PPO on RLHF benchmarks, which was a mildly embarrassing result for the field. If I had academic-scale compute and one algorithm to pick, this is the one — simplest thing that works, and hardest to get subtly wrong.

2.6 REINFORCE++

REINFORCE with: token-level KL penalty, advantage normalization, clipping. Closer to PPO but without value head.

2.7 VinePPO

Variance-aware credit assignment via Monte-Carlo rollout from each step. From step $t$ , sample $K$ continuations to estimate the value at $t$ . Per-step advantage = (reward) − (estimated value). Closes the gap between sequence-level rewards and per-token credit.

Pros. Better credit assignment than GRPO/RLOO. Cons. $K \times$ rollout cost.

2.8 PRIME (Process Reward Implicit through Implicit rewards, Cui 2025)

Joint training: a generative reward model (the “implicit PRM”) and the policy. The implicit PRM gives token-level rewards derived from the difference of log-probs between policy and reference. Plus outcome reward from the verifier. No need to train a separate explicit PRM. Demonstrates strong gains on math.

2.9 GSPO (Group Sequence Policy Optimization, Qwen 2025)

Sequence-level (not token-level) importance ratio. Argued to stabilize MoE / large-model RL where token-level off-policy ratios can blow up.

2.10 Step-RL (PRM-supervised RL)

Use a Process Reward Model to give a per-step reward in the RL update. Conceptually appealing; in practice (per R1) hasn’t beaten ORM-only with a strong base.

2.11 Iterative DPO / Online DPO

Not strictly RL, but adjacent. Repeat: sample policy, build preference pairs from verifier-graded outputs, DPO. Cheap, no value head, no rollout heavy lifting. Used by Llama 3 and Tülu 3 in some stages.

2.12 The “what to actually use” heuristic

Verifiable reward, big reasoning model: GRPO or DAPO (ByteDance Stack: DAPO).
Verifiable reward, smaller / academic compute: RLOO is hard to beat for simplicity.
Mixed verifiable + judge reward: PPO with value head (because reward is more dense / smooth) or DAPO with composite reward.
Speed of iteration matters more than peak quality: Iterative DPO.
You want a per-step signal without training a PRM: PRIME (implicit PRM).

Saying it out loud. The honest framing is that all of these are REINFORCE plus variance reduction, and the differences are about what you use as a baseline and how you keep updates from blowing up. PPO learns a critic to give you the baseline. GRPO throws the critic away and uses the mean reward of a group of rollouts on the same prompt instead. RLOO does the same but leaves the sample itself out of its own baseline, which makes it unbiased. Dr. GRPO drops the standard-deviation normalisation because it quietly over-weights easy problems, and DAPO stacks four practical fixes — asymmetric clipping, discarding zero-signal groups, token-level loss, and penalising overlong chains. GSPO moves the importance ratio to the sequence level, which matters specifically because token-level ratios blow up on mixture-of-experts models. If someone asks what to actually use: DAPO or GRPO at scale, RLOO for simplicity, and the honest caveat that the gaps between them are smaller than the gap made by a better verifier.

3. Verifier design — the often-neglected half

Most RLVR papers focus on the algorithm; the verifier is the actual moat. A bad verifier silently caps your model.

3.1 What a good verifier does

High recall on equivalent answers. “0.5”, “1/2”, “\frac{1}{2}”, “0.50” should all match.
Low false-positive rate. Doesn’t accept wrong answers that look like the right one.
Fast. Will be run millions of times during training.
Robust to format. Final-answer extraction handles \boxed{}, <answer>...</answer>, “Final answer: 42”, etc.
Hard to game. Can’t be hacked by exotic equivalent-but-not-quite formats.

Saying it out loud. A good verifier has to get two things right that pull against each other. High recall on equivalent answers — one-half, zero point five, and the LaTeX fraction all have to match, or you punish the model for being right in the wrong notation. And low false positives — because every wrong answer the verifier accepts is a lie you’re training on, and the policy will find those lies faster than you will. Add fast, because you’ll run it millions of times, and robust to format, because final-answer extraction has to survive boxed notation, answer tags, and “the answer is 42”. The failure mode to name is that a lenient verifier silently caps your model: the reward curve keeps climbing while true accuracy plateaus, and you won’t see it unless you’re reading samples.

3.2 Math verifier patterns

Sympy canonicalization. sympy.simplify(expr1 - expr2) == 0. Handles 1/2 vs 0.5. Handles algebraic equivalents. Has corner cases (transcendental equivalence is undecidable).
Numeric tolerance for decimals. abs(pred - gold) < 1e-6 * max(1, abs(gold)).
Set / list / multiset matching. For “list all roots.”
Formal proof verifier (Lean / Coq). For IMO-level problems. Cleanest but limited domain.
Multi-choice extraction. Match (A|B|C|D) from response.

3.3 Code verifier patterns

Hidden test suite. Run code against held-out tests. Watch for: time limits, memory limits, sandboxing.
Functional vs syntactic. Best practice: only count passed-functional-tests, not “compiles without error.”
Fuzz testing. Random inputs match a reference implementation.
Differential testing. Compare outputs to a known-correct implementation.

3.4 Tool-use / agentic verifier patterns

Final-state checking. Did the calendar slot get booked? Did the email send to the right person? (Sandboxed simulation.)
Sub-goal completion. Multi-step task; verify each sub-goal.
Trace replay. Compare against a reference trace.

3.5 Generative LLM-as-verifier

When deterministic verification is impossible (proof correctness in informal English, summary fidelity), use an LLM judge. Caveats:

Slower (one judge call per rollout).
Hackable (prompt injection in the rollout).
Self-preference bias if judge family matches policy family.
Needs adversarial training and held-out human gold to keep honest.

3.6 Composite verifiers

The 2025 frontier pattern: combine verifiable + LLM-judge.

total_reward = correctness * w_correct
             + format * w_format
             + language_consistency * w_lang
             + judge_score * w_judge   (only on non-verifiable subtasks)

Hierarchical is more stable than additive: only count judge_score if correctness == True, etc.

3.7 Verifier robustness — adversarial training

The policy WILL find verifier exploits. Defense:

Build adversarial test cases into the verifier.
Run negative-example fuzz tests on the verifier itself.
Ensemble verifiers (sympy + numeric + regex agreement).
Periodic manual sample inspection.

Saying it out loud. The thing I’d push hardest in an interview is that the verifier, not the algorithm, is the moat. Everyone benchmarks GRPO against DAPO against RLOO and the differences are a few points; a verifier with a two percent false-accept rate will cap your model in a way no optimiser recovers from. Concretely: sympy canonicalisation with numeric tolerance for math, hidden test suites with time and memory limits for code, final-state checks in a sandbox for tool use, Lean when you can afford formal. And when nothing deterministic exists, an LLM judge — with the caveats stated: it’s slower, it’s injectable by content in the rollout, and it has self-preference bias if it’s the same family as the policy. The discipline that actually protects you is treating the verifier as an adversarial target — fuzz it, ensemble it, and read samples, because the policy will find its exploits whether or not you do.

4. Reward shaping in RLVR

Plain-language gloss. Reward shaping is adding extra small terms alongside “was the answer right”, to nudge behaviour the correctness signal alone doesn’t reach — use the right tags, don’t ramble, don’t switch languages mid-thought. The danger is that every term you add is a new thing to game, so the practical rule is that correctness must dominate everything else by a wide margin.

A practical zoo of components, all of which appear in published recipes.

4.1 Correctness reward

Binary or graded:

Binary. $r \in {0, 1}$ (correct / wrong). Simple, sparse.
Partial credit. Math: $r = 0.3$ for “correct setup, arithmetic error”; $r = 1$ for fully correct.
Per-test-case for code. $r = (tests passed) / (total tests)$ . Smoother.

4.2 Format reward

Penalize / reward output structure: did it use <think>...</think><answer>...</answer>? Did it produce a parseable boxed answer?

Magnitude should be much smaller than correctness — otherwise the model learns to emit format with garbage answers.

4.3 Length reward

Soft penalty above a cap. Clip CoT to N tokens; penalize (actual_len − N) / N if exceeded.
Soft bonus for non-trivial CoT. Prevents the model from collapsing to greedy short answers.
DAPO’s “overlong reward shaping.” Specifically penalize CoTs that hit the max-length context — these are usually non-terminating failures.

4.4 Language consistency reward

Critical for multilingual settings (and for R1-Zero’s mid-CoT English↔Chinese drift). Detect dominant language in CoT; penalize switches.

def language_consistency(cot, target_lang):
    detected = langdetect.detect_langs(cot)
    target_prob = next((d.prob for d in detected if d.lang == target_lang), 0)
    return target_prob   # in [0, 1]

4.5 Step-level (PRM) reward

Per-step from a trained PRM. Empirical evidence: helps in some setups (Math-Shepherd, OmegaPRM) but R1’s headline finding is that ORM alone is sufficient for strong reasoning.

4.6 Diversity / exploration bonus

Reward variance within a group reduces; if your group’s rewards are all equal (all wrong or all right), you have no learning signal. DAPO’s “dynamic sampling” simply discards such groups and resamples. Alternative: entropy bonus.

4.7 Repetition penalty

Penalize n-gram repetition in the CoT. Prevents the policy from learning a degenerate “loop forever” attractor.

4.8 Reward composition pattern

Hierarchical with hard gate on format:

def compute_reward(rollout):
    if not has_valid_format(rollout): return 0.0
    correctness = verifier(rollout)
    if not correctness: return -0.1   # small negative for wrong-but-formatted
    bonus = (
        0.1 * language_consistency(rollout) +
        0.05 * (1.0 if not overlong(rollout) else 0.0)
    )
    return 1.0 + bonus

Saying it out loud. The rule I’d lead with is: correctness dominates, everything else is a whisper. Format reward exists so the answer is parseable, and if you weight it too high the model learns to emit beautiful empty tags. Length shaping exists to stop chains growing until they hit the context cap, which is almost always non-termination rather than deep thought. Language-consistency reward exists because R1-Zero drifted between English and Chinese mid-chain. And the composition pattern that works is hierarchical rather than additive — format is a hard gate, correctness is the dominant term, and the soft bonuses only apply once you’re already correct. The named failure mode is Goodhart, and the practical defence isn’t a better formula, it’s reading rollouts at every stage, because reward hacking is glaringly obvious in the samples and completely invisible in the reward curve.

5. KL regularization choices

Plain-language gloss. The KL term measures how far the model you’re training has drifted from the model you started with, and subtracting it from the reward means “improve, but don’t become a different model”. Without it, a policy chasing a proxy reward will happily walk off into text that scores well and reads like nonsense.

KL keeps the policy near the reference (SFT or DPO model) — without it, the policy drifts into incoherence.

5.1 Forward KL vs reverse KL

Forward KL $KL (p ∥ q)$ : mean-seeking; spreads probability mass; encourages diverse outputs.
Reverse KL $KL (q ∥ p)$ : mode-seeking; concentrates on high-prob regions; encourages confident outputs.

Standard PPO/GRPO uses reverse KL (mode-seeking) which is fine for reasoning where you want confident correct outputs.

Saying it out loud. Forward KL is mean-seeking — it punishes you for putting zero probability anywhere the reference put mass, so it spreads out and keeps diversity. Reverse KL is mode-seeking — it punishes you for putting mass where the reference didn’t, so it collapses onto the reference’s high-probability regions and gets confident. Standard PPO and GRPO use reverse KL, and for reasoning that’s the right default, because you want the model confidently committing to a correct chain rather than hedging across many. The cost you’re accepting is diversity, which matters more than people think: if you’re going to sample many rollouts and take the best, an over-confident policy gives you N copies of the same wrong idea.

5.2 Token-level vs sequence-level KL

Token-level. KL added to per-token reward. Strong regularization.
Sequence-level. Estimated from log-prob ratio of full sequences. Weaker but cheaper.

5.3 Adaptive $β$

R1 uses adaptive $β$ with KL targets. If running KL > target, increase $β$ ; if < target, decrease. Standard PPO trick.

5.4 No-KL ablations

Some recent work (Dr. GRPO, some RLVR variants) argues that with strong correctness rewards and small enough updates, the KL term can be dropped. Empirically: depends on base model strength.

Saying it out loud. The KL term is a leash keeping the policy near the model you started from, and the whole art is how long to make it. Too tight and the policy can’t explore, so nothing improves. Too loose and you get the classic collapse — language drift, degenerate repeated templates, fluent nonsense that scores well on your proxy. Two implementation choices matter: token-level KL added to per-token reward is a stronger regulariser than the cheaper sequence-level estimate, and R1 uses an adaptive beta with a KL target, dialling it up when running KL exceeds the target. There’s a live debate about dropping KL entirely — Dr. GRPO and some other variants argue that with a strong correctness reward and small updates you don’t need it — and the empirical answer seems to be that it depends on how strong your base model is. The failure mode to name is reward over-optimisation: proxy reward keeps climbing while true quality plateaus or falls, and the KL cap is your main defence against it.

6. Curriculum, exploration, warm-start

6.1 Cold-start vs warm-start

Cold-start (R1-Zero style): start RL directly from base model. Surfaces emergent reasoning but produces illegible CoTs.
Warm-start (R1, Tülu 3, Qwen): start RL from an SFT’d model that already has the format. More stable and legible, possibly less exploratory.

The empirical answer: warm-start dominates for production. Cold-start is a research curiosity.

Saying it out loud. Cold-start means running RL directly on the base model, R1-Zero style, and it’s a beautiful research result — it proves the reasoning was latent and RL can surface it with no demonstrations at all. Warm-start means running RL from a model that’s already been SFT’d on a few thousand well-formatted long chains, which is what every production recipe actually does. The reason is legibility and stability: cold-start models produce chains that mix languages and use non-standard formatting, and they’re erratic outside math. The tradeoff to state honestly is exploration — the cold-start model isn’t anchored to a human-chosen reasoning style, so in principle it can find strategies the SFT data never contained, and warm-start gives some of that up for reliability. In practice everyone takes the reliability.

6.2 Curriculum

If the base model gets <1% on the training problems, gradients are pure noise. Solutions:

Filter problems by difficulty (start at problems the base solves >5% of the time).
Stage-wise: easy → medium → hard.
Adaptive: drop problems with all-correct or all-wrong groups; promote problems with intermediate success rates.

6.3 Exploration

Temperature. R1 uses T=0.6. Too low → no exploration; too high → noise.
Top-p sampling. 0.9 is default.
Group size. Larger groups = more diverse rollouts per prompt; cost scales linearly.
Best-of-N during data construction (rejection sampling). Even RLVR can be stuck; rejection-sampled SFT data refresh helps.

6.4 Replay buffers

Store high-reward trajectories. Replay them during training to keep good behavior alive. Useful when reward is sparse.

Saying it out loud. The single biggest practical failure in RLVR isn’t the algorithm, it’s that your problems are too hard. If the base model solves under one percent of them, essentially every rollout returns zero, the group baseline is zero, and your gradient is noise — you can train for a week and learn nothing. So you filter by difficulty and train on problems the model already gets right some of the time, ideally somewhere in the middle, and promote harder ones as it improves. That’s the curriculum. Exploration is the other side: temperature around point six, top-p around point nine, and a group size big enough that rollouts actually differ, since a group where everything is identical carries no signal either. The tradeoff to name is that group size scales cost linearly, so you’re literally buying gradient signal with GPU hours.

7. Common failure modes

The interview signals — be ready for these.

Length explosion. CoT grows to fill context window with no quality gain. Fix: length penalty + DAPO overlong reward shaping.
Mode collapse. Single reasoning template wins; diversity dies. Fix: entropy bonus, KL cap, diversity-aware sampling.
Language mixing. Mid-CoT switch between languages. Fix: language-consistency reward.
Verifier hack. Model finds answer formats the verifier wrongly accepts. Fix: harden verifier with adversarial cases.
Format reward gaming. Model emits perfect <think><answer> with garbage inside. Fix: gate other rewards on correctness; small format weight.
Reward sparsity. All rollouts wrong → no gradient. Fix: curriculum, easier base data, larger group size, longer rollouts.
Off-policy drift. Old samples in the buffer no longer represent current policy. Fix: importance-sampling clipping (PPO ratio clip), refresh buffer.
Catastrophic forgetting of non-RL tasks. Reasoning model regresses on chat. Fix: rejection-sampling SFT after RL stage (R1’s stage 3).
Reward overoptimization. True quality plateaus or drops as proxy reward keeps climbing. Fix: KL cap, ensemble verifiers, periodic human eval.
Instability with MoE / large models. Token-level ratios blow up. Fix: GSPO (sequence-level), gradient clipping, smaller LR.

Saying it out loud. These cluster into three families. Reward hacking: the model games format, gams length, or finds inputs the verifier wrongly accepts — the fix is a hard gate on correctness, small shaping weights, and hardening the verifier with adversarial cases. Optimisation pathology: mode collapse onto one reasoning template, length explosion until the chain fills the context, off-policy drift, and instability on mixture-of-experts models where token-level ratios blow up. And distribution damage: the model gets much better at math and measurably worse at chat, which is why R1’s third stage is a big rejection-sampled SFT to broaden it back out. The one that’s hardest to detect is reward over-optimisation — your proxy reward keeps climbing while real quality plateaus or drops — and the only reliable detector is a held-out human eval you run periodically and actually look at.

8. Open-source RLVR infrastructure

What you’d actually use to run RLVR experiments today.

8.1 Libraries

TRL (HuggingFace). PPO, DPO, RLOO, GRPO. Most accessible. Single-machine and multi-GPU. Best for small-to-mid scale.
veRL (ByteDance, 2024). Production-grade. Supports DAPO, GRPO, PPO. Distributed training with vLLM rollouts. Used to reproduce R1-style training.
OpenRLHF. Another PPO/DPO/RLHF library, distributed. Good docs.
HuggingFace Open-R1. Specifically reproducing R1’s recipe. Curated data + GRPO. Good starting point for replication studies.
NeMo-Aligner (NVIDIA). Production-grade for large models.
TRL-Lite / GRPO trainers. Lightweight community implementations (Will Brown, etc.) for academic-scale experiments.

8.2 Rollout backends

vLLM. De facto standard. Fast inference for RL rollouts. Supports continuous batching and chunked prefill.
TGI (Text Generation Inference). Alternative.
SGLang. Flexible, good for tool-use rollouts.

8.3 Sample architecture

[Trainer (Ray actors, DeepSpeed/FSDP)] ← gradient updates
        ↓ weights sync
[vLLM Rollout Worker(s)] → rollouts → [Verifier(s)] → rewards
        ↑                       ↓
        └─────── replay buffer / batch ──────────┘

8.4 Compute budget reality check

To reproduce R1-style reasoning RL for a 7B-base model:

~1k-10k math problems.
~50k rollouts per RL epoch.
8-64 H100s.
2-7 days.

For a 70B+ model: 100-500 H100s, weeks. Out of academic reach without partnerships.

For a 1.5B-3B model + R1-Distill-style approach: feasible on 8 GPUs.

Saying it out loud. The stack has converged, which makes this an easy answer. TRL from Hugging Face if you’re on one machine or a small cluster, veRL from ByteDance if you’re doing this properly at scale with DAPO and GRPO, OpenRLHF and NeMo-Aligner as the other production options, and Open-R1 if you specifically want to reproduce the R1 recipe. Underneath all of them the rollout backend is vLLM, because RLVR is dominated by generation cost, not by the gradient step — you’re sampling many long chains per prompt, so continuous batching is what makes it tractable. The architecture is a trainer holding the weights, rollout workers generating, verifiers scoring, and a weight-sync step between them, and the engineering difficulty is mostly in that sync. The budget reality worth stating: a 7B R1-style reasoning run is roughly eight to sixty-four H100s for a few days, a 70B run is hundreds of GPUs for weeks — so academic work happens at 1.5B to 3B or on distillates.

RLVR generalizes beyond text math.

Vision-language reasoning. MathVista with image-rendered math problems. Reward = numeric match.
Code with side-effects. Generated code modifies a sandbox; verifier checks final state.
Tool-augmented reasoning. Model thinks → calls calculator/Python/search → integrates result → continues. Verifier grades final answer. Critical: gradient must flow correctly through tool boundary (usually only the LM tokens are differentiated; tool call results are treated as fixed context).
Long-horizon agents. SWE-Gym, OSWorld, AgentDojo, TAU-bench. Verifier = task-completion check. Credit assignment over hundreds of steps is hard; prevailing approach is final-only reward + GRPO with patience.

Saying it out loud. RLVR generalises anywhere you can programmatically check the outcome, which is a lot more than text math. Vision-language reasoning on image-rendered problems, where the reward is still just a numeric match. Code with side effects, where the verifier inspects the final state of a sandbox. Tool-augmented reasoning, where the model thinks, calls Python or a search tool mid-chain, and continues — and the implementation detail that matters there is that you only take gradients through the model’s own tokens; the tool’s returned text is fixed context, not something you differentiate through. Long-horizon agents are the hard frontier: SWE-Gym, OSWorld, TAU-bench, where the verifier is task completion and the credit-assignment problem is brutal because a single reward arrives after hundreds of steps. The current prevailing approach is honestly just final-only reward with a large group and a lot of patience, which is a statement about how unsolved this is.

10. Low-resource multilingual reasoning with RLVR

This is the user’s specific interest and a genuinely under-explored frontier in 2025-2026. Worth thinking through carefully.

10.1 The problem

Almost all reasoning RL has been done in English (and to some extent Chinese, via DeepSeek and Qwen). Yet:

~6 billion people don’t speak English natively.
Most low-resource languages have minimal high-quality reasoning data in pretraining.
Reasoning models trained on English math do not transfer cleanly to math problems posed in Bengali, Swahili, Yoruba, Vietnamese, etc.
Even when the model can produce a correct numeric answer, it often produces it via an English CoT, which is opaque to monolingual users.

There are roughly four communities of users for low-resource multilingual reasoning:

Education — math tutors for students who don’t read English fluently.
STEM access — scientific reasoning in non-English languages.
Government/legal — rule-based reasoning over local documents.
Cultural-context reasoning — problems involving culturally-specific concepts.

Saying it out loud. Nearly all reasoning RL has been done in English, with some Chinese, and the result is that a model which solves a grade-school word problem fine in English can fall apart on the same problem in Bengali or Swahili or Yoruba. Two separate failures are hiding in there. One is capability — the base model saw very few reasoning-shaped tokens in that language, so the prior is weak. The other is legibility — even when it gets the right number, it often reasons in English internally, which is useless to a monolingual student who needs to see the working. So the population that most needs a patient tutor gets the version of the model that can’t show its work. That’s the gap, and the reason it’s tractable is that math verifiers are language-agnostic: the answer is 42 regardless of what language the question was in.

10.2 Why this is hard

Pretraining gap. A 70B base trained on Common Crawl has 5% non-English tokens, often worse for low-resource languages. The reasoning prior is weak.
CoT gap. Even when the model “understands” the question, it tends to think in English.
Verifier-language mismatch. Math verifiers (sympy, regex) are language-agnostic for numeric answers — a small mercy. But word-problem extraction varies by language: “the product of x and y” has language-specific phrasings.
Benchmark gap. GSM8K-multilingual (MGSM), Belebele, AfriBench, BHASA exist but are smaller and noisier than English benchmarks.
Cultural conventions. Lakh / crore (South Asian numerals), date formats, currency, units, conventions in problem framing.

10.3 What’s currently done (and the gaps)

PB-RLSVR — Pivot-Based RL with Semantically Verifiable Rewards (Faisal et al., 2509.25543, Sep 2025). The first framework to systematically extend RLVR from English to multilingual reasoning without target-language human annotation. Uses a high-performing English LLM as a pivot to generate reference responses on reasoning tasks; the multilingual policy is rewarded for semantic equivalence to the English reference, computed via either embedding-based similarity or machine-translation-based equivalence. Reports +16.4% on Llama-3.1-8B and +10.2% on Qwen3-32B average multilingual performance vs PPO baselines. The cleanest “pivot teacher” formulation in the literature for transferring English reasoning capability to other languages via RL — and it’s the canonical reference for any frontier interview discussion of this problem.

MGSM (Multilingual Grade School Math). Translation of GSM8K into 11 languages. Models score much worse on low-resource languages than English. Gap: translation-only doesn’t capture native problem-framing conventions.

Aya / Aya 23 (Cohere). Multilingual instruction-tuning at scale. Not reasoning-RL trained; weak on math beyond English.

Qwen / DeepSeek multilingual variants. Strong on Chinese-English; weak on truly low-resource languages.

MGSM-via-translate-then-reason. “Translate to English, reason, translate back.” Works but loses linguistic faithfulness; brittle to mistranslation.

Cross-lingual fine-tuning experiments (academic). Show that fine-tuning a strong English reasoning model on translated data yields modest cross-lingual transfer. PB-RLSVR (above) is the SOTA RLVR-based approach.

10.4 The technical opportunity

Three things are simultaneously true:

Verifiers for math are language-agnostic (the answer is 42, not “the answer is forty-two”).
The reasoning capability transfers from English-trained models to multilingual via SFT/distillation.
RLVR on multilingual reasoning data has barely been done at scale.

That’s an exploitable gap.

10.5 Approach landscape

A. Translation-augmented training data

Translate English math problems (GSM8K, MATH, AIME) into target language(s).
Translate the CoT solutions too.
SFT on the translated data.
Then RLVR on translated problems with the language-agnostic verifier.

Pros. Cheap, leverages existing high-quality English data. Cons. Translation artifacts; cultural-context loss; doesn’t capture native problem-framing.

B. Distillation from a strong English reasoning teacher

Generate long CoTs from R1 / o1 (in English) on math problems.
Translate the CoT + answer into target language.
SFT a multilingual base.
Optional: light RLVR on top.

Pros. Inherits R1-grade reasoning. Most plausible path to a strong low-resource reasoning model in 2025. Cons. Translation quality of long CoTs is variable; English-style reasoning patterns may not match target-language conventions.

B’. Pivot-based RL with semantically verifiable rewards (PB-RLSVR)

The natural RL extension of approach (B). Instead of distilling once and stopping, keep the English pivot model as an oracle during RL training:

For each training prompt, the pivot model produces a reference response (or reference reasoning trace).
The multilingual policy generates its own response in target language.
Reward = semantic equivalence between policy response and pivot response, computed via:
- Multilingual embedding similarity (e.g., LaBSE, multilingual-E5 cosine), OR
- Round-trip machine translation + token-level overlap, OR
- LLM-as-judge for semantic equivalence.

This is what Faisal et al. 2025 introduces and validates. The key insight: you don’t need a verifier specific to the target language; you only need a way to compare semantic equivalence to a known-good English reference. And cross-lingual semantic similarity is a much easier problem than producing target-language ground truth.

Pros. No target-language ground truth needed. Pivot stays anchored to English-quality. Composable with verifiable rewards (when answer is numeric, both pivot match AND verifier signal can be combined).

Cons. Embedding-based reward can be hacked (model emits text superficially similar to pivot but semantically off). MT-based reward inherits MT errors. Pivot’s reasoning style may dominate target-language outputs (homogenization risk).

Saying it out loud. The trick here is nice because it sidesteps the thing that makes multilingual RLVR hard. You don’t have ground-truth reasoning data in the target language and you can’t afford to annotate it — so instead you keep a strong English model as a pivot, let it produce a reference answer for each problem, and reward the multilingual policy for being semantically equivalent to that reference. Equivalence gets measured by multilingual embedding similarity, round-trip translation overlap, or a judge. The key insight is that cross-lingual semantic comparison is a much easier problem than producing target-language ground truth, and we already have decent tools for it. The reported gains are around sixteen percent on Llama-3.1-8B and ten percent on Qwen3-32B over PPO baselines. And I’d name the two risks unprompted: an embedding-based reward is hackable — you can look similar to the pivot while being wrong — and there’s a homogenisation risk where the pivot’s English reasoning style gets imprinted onto every target language.

C. Cross-lingual code-switched CoT

Allow the model to think in English (its strongest reasoning language) but answer in target language.
Enforce via reward: correctness * (1 + 0.5 * answer_in_target_language).

Pros. Leverages strongest reasoning circuit; easy to verify. Cons. Users may want to see the reasoning in target language.

D. Language-conditioned RLVR

Add reward component: language_consistency(CoT, target_language) * w_lang.
Penalize mid-CoT switches to English.
Pair with curriculum: easier-target-language problems first.

Pros. Forces the model to learn target-language reasoning, not just answer translation. Cons. Needs strong target-language pretraining priors. Compute may be wasted on language alignment.

E. Synthetic data generation

Use a strong English LLM + translator to generate diverse target-language math problems with verifiable answers.
Filter for translation quality.
RLVR on this synthetic set.

Pros. Scales easily. Cons. Quality control is hard; risk of distribution shift from real native problems.

F. Tool-augmented for low-resource

Allow the model to call a calculator / Python / translator mid-CoT.
Reduces the need for the model to do multi-digit arithmetic in target language (a known weakness).
RLVR on full tool-augmented traces.

Pros. Practical; leverages tools. Cons. Needs sandboxing; tool-trace credit assignment is harder.

G. Composite / multi-stage

The realistic recipe combines several:

Cold-start SFT on R1-distilled CoTs translated into target language.
Stage-2 RLVR with (language-consistency + correctness + format) reward.
Stage-3 rejection-sampling SFT to broaden distribution.
Final RLHF for politeness / cultural appropriateness.

This is essentially a “low-resource R1” recipe.

10.6 Key technical questions

How much target-language pretraining is needed for RLVR to work? (Hypothesis: a base with ≥5% target-language tokens in pretraining is the floor.)
How much does R1-distill from English transfer to target languages? (Hypothesis: more than translation alone, less than native English performance.)
Is language-consistency reward strong enough to prevent code-switching, or does it suppress reasoning quality?
Does multilingual training help English (positive transfer) or hurt (interference)?
Are there target-language cultural reasoning conventions (e.g., South Asian “lakh” / “crore” numerals; Arabic right-to-left math layout) that need explicit handling?
What’s the minimum viable problem set size per language for RLVR to converge?

10.7 Failure modes specific to this setting

Code-switching collapse. Model learns to flip to English mid-CoT, gets the answer, switches back. Verifier accepts; user is confused.
Translation cascading errors. If training data is English → translation, then RLVR can amplify translation artifacts.
Numeric format mismatches. Lakh vs million, date formats, decimal commas vs dots.
Verifier limitations. Math verifier is language-agnostic for numbers, but word-answer problems (“what is the capital of X?”) need language-aware verification.
Loss of cultural context. A “fair price” problem assumes Western market conventions; low-resource users may have different price intuitions.

Saying it out loud. The observation that makes this whole area work is that math verifiers don’t care what language the question was in — the answer is still 42. So the reward signal transfers for free, which means the hard part is only getting a policy that reasons in the target language at all. The approach landscape runs from cheapest to most ambitious: translate English problems and fine-tune, distil long chains from an English reasoning teacher and translate them, keep an English pivot model live during RL and reward semantic equivalence to it, or add a language-consistency reward so the model is penalised for slipping into English mid-chain. The realistic recipe is a low-resource R1: cold-start SFT on translated distilled chains, then RLVR with correctness plus format plus language consistency, then rejection-sampled SFT to broaden, then light RLHF. The failure mode I’d name specifically is code-switching collapse — the model flips to English, solves it, flips back, the verifier is perfectly happy, and the user still can’t read the reasoning.

11. Concrete research project blueprints

These are framed as 1-month to 6-month projects. Each is publishable at a major venue if executed well, and any of them is a strong “what would you work on” answer in a research-scientist interview.

11.1 Project A: “BengaliMath-RL — RLVR for low-resource mathematical reasoning”

Goal. Build a strong Bengali mathematical reasoning model and characterize how RLVR transfers from English.

Setup.

Base: Qwen 2.5 7B (multilingual) or Llama 3.1 8B.
Stage 1: SFT on R1-distilled English CoTs translated to Bengali (~100k pairs).
Stage 2: RLVR with GRPO / DAPO on Bengali-translated GSM8K + MATH (~50k problems).
Reward: correctness + 0.1 * language_consistency(CoT, "bn") + 0.05 * format.
Eval: MGSM-bn, AIME-bn (translate manually verified), out-of-distribution Bengali word problems collected from textbooks.

Hypotheses to test.

H1: R1-distill alone gives ~70% of frontier English-on-Bengali performance.
H2: RLVR on top recovers another 15-20% gap.
H3: Language-consistency reward reduces code-switching from 40% to <5%.

Deliverables. Model weights, eval harness, ablation paper.

Why it’s interesting. First public study of low-resource RLVR in Indo-Aryan family. Generalizable recipe.

11.2 Project B: “Cross-Lingual Reasoning Transfer via RLVR — A Systematic Study”

Goal. Rigorously measure how reasoning capability transfers from English-trained RLVR to N target languages.

Setup.

Take an open-source English RLVR-trained reasoning model (Open-R1, R1-Distill).
For each of {Bengali, Swahili, Yoruba, Vietnamese, Tamil, Pashto, Quechua}: light SFT on translated data; light RLVR; measure performance vs English baseline.
Vary: (a) amount of target-lang SFT data, (b) RLVR steps, (c) language-consistency reward weight.

Hypotheses.

Resource level (size of target lang in pretraining) predicts transfer slope.
Language family matters — Indo-European > Bantu > tonal-isolating.
RLVR transfer plateau is reached at ~1k target-lang RL examples per language.

Deliverables. Publishable empirical paper (à la “Scaling Laws for …”).

11.3 Project C: “Code-Switched RLVR — Allowing English-Inner-Voice Reasoning”

Goal. Show that allowing the model to think in English (its strongest language) and answer in target produces better accuracy than forcing target-language CoT.

Setup.

Two reward variants: (i) all-target-lang reward, (ii) target-lang-answer-only reward.
Compare accuracy, user-reported usefulness.

Hypotheses.

(ii) wins on accuracy.
User study (key!) reveals trade-off: users prefer target-language CoT for educational use even if accuracy is lower.

Deliverables. ACL / NAACL paper. Has a clean human-eval angle.

11.4 Project D: “Synthetic Multilingual Reasoning Data with Verifier-Guided Filtering”

Goal. Generate high-quality target-language math problems with verifiable answers via LLM + filter.

Pipeline.

Strong LLM generates problem + reference answer in English.
Translates to target language.
Verifier checks: (a) translation quality (round-trip check); (b) answer correctness (sympy on the original).
Keep ~10% that pass both.

Hypotheses.

1M filtered synthetic problems → 100k high-quality.
Training on these vs human-curated translation has comparable downstream RLVR performance.

Deliverables. Public dataset + recipe.

11.5 Project E: “Multilingual PRMs — Process Supervision Without Per-Step Translation”

Goal. A PRM trained on English process labels, applied to target-language reasoning steps, transfers via shared embedding space.

Setup.

PRM trained on PRM800K (English).
Target-language RLVR with this PRM as a step-reward.
Compare to ORM-only baseline.

Why interesting. Shows that step-level reasoning structure is language-agnostic enough to transfer.

11.6 Project F: “Tool-Augmented Multilingual RLVR”

Goal. Allow target-language models to invoke a calculator/Python tool mid-CoT. RLVR with full trace verification.

Why. Reduces target-language arithmetic burden on the model. Shifts the RLVR signal to when and how to use tools rather than how to do arithmetic in target language.

11.7 The simplest first step

If you have one week and one A100:

Take Qwen 2.5 1.5B-Instruct.
Translate ~5k GSM8K problems into target language with Google Translate / NLLB.
SFT on translated CoTs from R1-Distill-Qwen-1.5B.
Eval on MGSM. Compare to: original Qwen, R1-Distill.
Publish the gap analysis.

This is a tiny project but immediately useful and demonstrates competence.

Saying it out loud. If someone asks what I’d actually build, the smallest honest version is: take a base model with some real presence in the target language, distil a few tens of thousands of long chains from an English reasoning model, translate them, SFT, then run GRPO on translated GSM8K and MATH with correctness plus format plus a language-consistency term, and evaluate on MGSM. That’s a few GPUs and a few weeks, not a frontier budget. The scientific question worth designing around is which of those stages is doing the work — that’s the ablation that makes it a paper rather than a demo. And the honest risk is that the answer turns out to be “the distillation did all of it and the RL added two points”, which is a real possible outcome and worth saying up front rather than discovering at write-up time.

12. Datasets and benchmarks

12.1 English math (transfer-source)

GSM8K — 8.5k grade-school problems.
MATH — 12.5k competition problems.
AIME — American Invitational Math Exam.
MMLU-STEM subset.
GPQA — graduate-level science.
PRM800K — step-level labels.

12.2 Multilingual math benchmarks

MGSM (Multilingual GSM8K) — 11 languages, ~250 problems each.
MATHQA-cn (Chinese math).
GSM-Symbolic — perturbed GSM8K, more robust eval.
Hindu Math — emerging dataset for Hindi math reasoning.
Sinhala Math, Bengali Math — emerging community-curated.

12.3 Multilingual reasoning broadly

Belebele — reading comprehension in 122 languages.
MMLU-translated — knowledge across many languages.
AfriBench — for African languages.
BHASA — Indic-language eval suite.
MEGA / MEGA-Verse — broad multilingual NLP eval.
Aya Eval — multi-domain multilingual.

12.4 Translation infrastructure

NLLB-200 (Meta) — 200 languages; reasonable for low-resource.
MADLAD-400 (Google) — 400 languages.
Aya 23 — 23 languages, instruction-following.

13. Open frontier questions

Can RLVR teach a base model new reasoning capabilities or only surface what’s latent?
What fraction of multilingual transfer is “thinking in English internally” vs genuine target-language reasoning?
Is there a scaling law for cross-lingual reasoning transfer?
What’s the role of formal verifiers (Lean / Coq) for non-English mathematical traditions (Indian classical math, Chinese counting rod methods)?
Can multi-agent debate in multiple languages improve target-language reasoning?
Is there a “low-resource bonus” — does training on low-resource languages improve robustness on English (positive transfer)?
How do you do RLVR for non-numeric reasoning in target languages?
What’s the impact of translating-reasoning-back vs target-language-from-the-start?
Can speech-to-speech reasoning (no transcription) work in low-resource settings?

Saying it out loud. The one I’d lead with is whether RLVR teaches new reasoning or only surfaces what pretraining already put there, because the answer determines whether base quality is still the binding constraint — and the pass-at-k evidence currently leans toward “only surfaces”. The second is the multilingual version of the same question: how much of cross-lingual transfer is genuine target-language reasoning versus the model thinking in English behind a translated surface, which nobody has cleanly measured. Then whether there’s a scaling law for cross-lingual transfer, and how you do RLVR at all for reasoning whose answer isn’t a checkable number. I’d be explicit that these are open — the good version of this answer names what evidence would settle each one, rather than picking a side. For the first, that’s a controlled pass-at-k comparison against the model’s own base; for the second, it’s probing whether the intermediate representations are language-specific or pivoting through English.

14. References

RLVR algorithms

Shao et al. (DeepSeek), DeepSeekMath / GRPO, 2024.
Yu et al. (ByteDance), DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2024.
Liu et al., Dr. GRPO: Understanding R1-Zero-Like Training, 2025.
Ahmadian et al. (Cohere), Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback, 2024.
Kool et al., Buy 4 REINFORCE Samples, Get a Baseline for Free, 2019.
Cui et al., PRIME: Process Reinforcement through IMplicit rEwards, 2025.
Zheng et al. (Qwen), Group Sequence Policy Optimization, 2025.
Kazemnejad et al., VinePPO, 2024.

Reasoning RL

DeepSeek-AI, DeepSeek-R1, 2501.12948, 2025.
Lambert et al., Tülu 3, 2024.
Lightman et al. (OpenAI), Let’s Verify Step by Step (PRM800K), 2023.
Wang et al., Math-Shepherd, 2024.
Luo et al., OmegaPRM, 2024.

Reward modeling and hacking

Gao et al., Scaling Laws for Reward Model Overoptimization, 2023.
Singhal et al., A Long Way to Go: Investigating Length Correlations in RLHF, 2024.
Park et al., Disentangling Length from Quality in DPO, 2024.
Sharma et al., Towards Understanding Sycophancy in LMs, 2023.
Mahan et al., Generative Reward Models, 2024.

Multilingual / low-resource

Faisal, Song, Wang, Ma, Liu, Deng, Indurthi, Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model (PB-RLSVR), arXiv:2509.25543, Sep 2025. The canonical RLVR-for-multilingual paper.
Aya Collective (Cohere), Aya 23, 2024.
Üstün et al., Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model, 2024.
Shi et al., Language Models are Multilingual Chain-of-Thought Reasoners (MGSM), 2023.
NLLB Team (Meta), No Language Left Behind, 2022.
Adelani et al., AfriBench, 2024.
Kudugunta et al., MADLAD-400, 2024.
Ahuja et al., MEGA, 2023.
Bandarkar et al., Belebele, 2024.
Self-Improving Multilingual Long Reasoning via Translation-Reasoning Integrated Training — companion translation-augmented approach.
Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models — characterizes the degradation when multilingual models reason in non-English.
Various IndicNLP / BHASA / SEACrowd consortium papers.

Open-source infrastructure

TRL (HuggingFace) — huggingface/trl.
veRL (ByteDance) — volcengine/verl.
Open-R1 (HuggingFace) — huggingface/open-r1.
vLLM — vllm-project/vllm.
OpenRLHF — OpenRLHF/OpenRLHF.

How to use this chapter

Read §§1-7 once for the core RLVR understanding.
Memorize §2 (algorithm zoo) and §7 (failure modes) — interview gold.
Skim §8-9 to know the infra landscape.
Spend serious time on §10 (low-resource multilingual) — that’s your differentiation.
Pick one of the project blueprints in §11 and write a 1-page proposal as practice.
Read the actual DeepSeek-R1 paper and the DAPO paper — non-negotiable.

The single sentence to remember for the multilingual angle: RLVR’s reward signal is language-agnostic for math; the bottleneck is target-language pretraining priors and CoT-language consistency, both addressable via a 4-stage recipe of distill → SFT → RLVR-with-language-consistency-reward → final-RLHF, and this has barely been done at scale outside English/Chinese — making it a wide-open research direction.

Keyboard shortcuts

ML & LLM Interview Prep — Deep Dives