AI/ML RLVR Dr. GRPO RPO-DPO PyTorch

Beyond Length-Biased RL: Online Dr. GRPO, Iterative RPO-DPO & Rubric Verifiers Under ASR Noise


In LLM Post-Training: A Practitioner's First-Principles Guide and Autonomous AI Scientists, we formulated Group Relative Policy Optimization (GRPO; Shao et al., 2024) and Direct Preference Optimization (DPO; Rafailov et al., 2023). However, when deploying these objectives on multi-turn clinical reasoning and noisy multilingual/code-switched transcripts (such as Hindi–English Bhashini ASR streams governed by deterministic WHO IMNCI triage guidelines), two subtle mathematical pathologies surface:

  1. Offline DPO Likelihood Collapse on Reasoning Chains: Standard DPO maximizes the log-ratio margin $\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$, which admits a degenerate saddle where both $\log \pi_\theta(y_w|x)$ and $\log \pi_\theta(y_l|x)$ decrease simultaneously whenever $y_w$ and $y_l$ share long reasoning prefixes (Pang et al., 2024, arXiv:2404.19733).
  2. Token-Length & Question-Difficulty Bias in Vanilla GRPO: Dividing the sequence policy loss by rollout length $\frac{1}{|o_i|}$ and normalizing advantages by group standard deviation $\mathrm{std}(\{R_1, \dots, R_G\})$ systematically penalizes concise correct answers while encouraging failed rollouts to ramble to the context limit (Liu et al., 2025, Dr. GRPO, arXiv:2503.20783).

0. Why Vanilla GRPO Rewards Rambling on Negative Rollouts – and How Dr. GRPO Fixes It

Inspect the per-token policy gradient weight in standard GRPO for rollout $i \in \{1, \dots, G\}$ of length $|o_i|$ with scalar outcome reward $R_i \in \mathbb{R}$:

Vanilla GRPO vs. Unbiased Dr. GRPO Objective (Liu et al., 2025, arXiv:2503.20783)
$$\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{G}\sum_{i=1}^{G} \color{#f43f5e}{\frac{1}{|o_i|}} \sum_{t=1}^{|o_i|} \min\!\left(\rho_{i,t}\hat{A}_i^{\text{std}},\; \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_i^{\text{std}}\right), \qquad \hat{A}_i^{\text{std}} = \frac{R_i - \bar{R}}{\color{#f43f5e}{\mathrm{std}(R) + \epsilon}}$$ $$\mathcal{L}_{\text{Dr.GRPO}}(\theta) = -\frac{1}{G}\sum_{i=1}^{G} \color{#34d399}{\frac{1}{C_{\max}}} \sum_{t=1}^{|o_i|} \min\!\left(\rho_{i,t}\hat{A}_i^{\text{unbiased}},\; \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\hat{A}_i^{\text{unbiased}}\right), \qquad \hat{A}_i^{\text{unbiased}} = \color{#34d399}{R_i - \bar{R}}$$

In vanilla GRPO, when $\hat{A}_i \lt 0$ (an incorrect diagnosis or failed tool call), multiplying by $1/|o_i|$ dilutes the negative penalty per token on longer responses – actively training the policy to emit verbose filler when uncertain! Replacing $1/|o_i|$ with a constant token budget normalizer $1/C_{\max}$ and dropping $\mathrm{std}(R)$ restores an unbiased Monte Carlo policy gradient.

Anchoring Reasoning Likelihoods with Iterative RPO-DPO

Before running online Dr. GRPO, cold-starting a policy with RPO-DPO (Pang et al., 2024) adds a negative log-likelihood (NLL) anchor directly on the winning reasoning trace $y_w$ normalized by $|y_w|$:

RPO-DPO Hybrid Objective ($\text{DPO} + \alpha\,\text{NLL}$ on Winning Chain-of-Thought)
$$\mathcal{L}_{\text{RPO-DPO}}(\theta) = -\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right) - \alpha \frac{\log \pi_\theta(y_w \mid x)}{|y_w|}$$
1. AtomicFact Simulator Hidden Clinical State z* + Bhashini Indic ASR Noise 2. SFT + RPO-DPO Warmup -100 Prompt Masking L_DPO + α L_NLL Anchor 3. Deterministic Rubric IMNCI Danger-Sign Recall - Shotgun Question Penalty 4. Online Dr. GRPO Â_i = R_i − R̄ (No Std) 1/C_max Length Invariant
Figure 1: End-to-end verifiable post-training pipeline for noisy multi-turn clinical agents: atomic-fact patient simulation with phonetic ASR corruption, RPO-DPO warmup, deterministic anti-shotgun rubric scoring, and unbiased Online Dr. GRPO.
Multilingual RLVR & Multi-Turn Masking Footguns: (1) Tokenization Fertility Amplification: Devanagari and Dravidian scripts often require $2.5\text{–}4\times$ more subword tokens per clinical concept than English; under vanilla $1/|o_i|$ GRPO, code-switched Hindi responses suffer artificially compressed per-token gradients unless normalized by a fixed $C_{\max}$. (2) Shotgun Questioning Exploit: In multi-turn diagnostic interviews, an RL policy rewarded purely for eliciting atomic patient symptoms will pack 8+ unrelated questions into a single turn – always apply an out-of-band regex/AST question-count guardrail ($N_q \le 2$ per turn) directly inside the verifiable reward function.

1. Runnable PyTorch Reference: Dr. GRPO vs. Vanilla GRPO & Anti-Shotgun Rubric Verifier

The self-contained PyTorch script below quantifies the exact 4x token-gradient distortion between short and verbose rollouts in vanilla GRPO and implements the unbiased Dr. GRPO + RPO-DPO loss functions:

💻 Python: Unbiased Dr. GRPO, Iterative RPO-DPO & Anti-Shotgun Rubric VerifierPyTorch
import torch
import torch.nn.functional as F

def score_clinical_turn_rubric(
    response_text: str, required_danger_signs: set[str], elicited_signs: set[str]
) -> float:
    """Deterministic rubric verifier with out-of-band shotgun-questioning penalty."""
    recall = len(required_danger_signs & elicited_signs) / max(1, len(required_danger_signs))
    # Count interrogative clauses to block 'shotgun questioning' reward hacking (>2 questions/turn)
    q_count = response_text.count("?") + response_text.count("क्या")
    shotgun_penalty = 0.4 * max(0, q_count - 2)
    return max(0.0, recall - shotgun_penalty)

def compute_token_policy_weights(
    rewards: torch.Tensor, lengths: torch.Tensor, c_max: int = 256, mode: str = "dr_grpo"
) -> torch.Tensor:
    """Compare per-token gradient scale between Vanilla GRPO and Dr. GRPO (arXiv:2503.20783)."""
    # rewards: [G], lengths: [G]
    if mode == "vanilla_grpo":
        adv = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + 1e-8)
        return adv / lengths  # Biased: penalizes concise positive & rewards verbose negative!
    # Dr. GRPO: zero-mean unscaled advantage divided by constant token budget c_max
    adv_unbiased = rewards - rewards.mean()
    return adv_unbiased / float(c_max)

def rpo_dpo_loss(
    pi_logps_w: torch.Tensor, pi_logps_l: torch.Tensor,
    ref_logps_w: torch.Tensor, ref_logps_l: torch.Tensor,
    len_w: torch.Tensor, beta: float = 0.1, alpha_nll: float = 1.0
) -> torch.Tensor:
    """Iterative RPO objective: DPO loss + length-normalized NLL anchor (Pang et al., arXiv:2404.19733)."""
    logits = (pi_logps_w - ref_logps_w) - (pi_logps_l - ref_logps_l)
    dpo_term = -F.logsigmoid(beta * logits).mean()
    nll_anchor = -(pi_logps_w / len_w).mean()
    return dpo_term + alpha_nll * nll_anchor

if __name__ == "__main__":
    rewards = torch.tensor([1.0, 1.0, 0.0, 0.0])       # 2 passing, 2 failing rollouts
    lengths = torch.tensor([64.0, 256.0, 64.0, 256.0]) # Concise (64) vs Verbose (256)

    w_vanilla = compute_token_policy_weights(rewards, lengths, mode="vanilla_grpo")
    w_dr = compute_token_policy_weights(rewards, lengths, mode="dr_grpo")
    print(f"[PASS] Vanilla GRPO Short/Long Token Weight Ratio: {(w_vanilla[0] / w_vanilla[1]).item():.1f}x (Length-Biased)")
    print(f"[PASS] Dr. GRPO Short/Long Token Weight Ratio    : {(w_dr[0] / w_dr[1]).item():.1f}x (Unbiased!)")
# Output:
# [PASS] Vanilla GRPO Short/Long Token Weight Ratio: 4.0x (Length-Biased)
# [PASS] Dr. GRPO Short/Long Token Weight Ratio    : 1.0x (Unbiased!)

2. Post-Training Alignment & RLVR Objectives: Head-to-Head

Objective Vanilla DPO Iterative RPO-DPO (arXiv:2404.19733) Online Dr. GRPO (arXiv:2503.20783)
Chosen Log-Prob $\log \pi_\theta(y_w|x)$ Can decrease alongside $\log \pi_\theta(y_l|x)$ Strictly anchored via $-\alpha \frac{\log \pi_\theta(y_w|x)}{|y_w|}$ Reinforced online on verified positive rollouts
Length Bias on Failed Rollouts Sensitive to chosen vs. rejected length gap Normalized by $|y_w|$ in NLL anchor Eliminated by replacing $1/|o_i|$ with constant $1/C_{\max}$
Question-Difficulty Scaling Offline static pair margin Iterative online pair construction Unbiased (omits $1/\mathrm{std}(R)$ blowup on easy/hard prompts)