Beyond Length-Biased RL: Online Dr. GRPO, Iterative RPO-DPO & Rubric Verifiers Under ASR Noise
In LLM Post-Training: A Practitioner's First-Principles Guide and Autonomous AI Scientists, we formulated Group Relative Policy Optimization (GRPO; Shao et al., 2024) and Direct Preference Optimization (DPO; Rafailov et al., 2023). However, when deploying these objectives on multi-turn clinical reasoning and noisy multilingual/code-switched transcripts (such as Hindi–English Bhashini ASR streams governed by deterministic WHO IMNCI triage guidelines), two subtle mathematical pathologies surface:
- Offline DPO Likelihood Collapse on Reasoning Chains: Standard DPO maximizes the log-ratio margin $\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$, which admits a degenerate saddle where both $\log \pi_\theta(y_w|x)$ and $\log \pi_\theta(y_l|x)$ decrease simultaneously whenever $y_w$ and $y_l$ share long reasoning prefixes (Pang et al., 2024, arXiv:2404.19733).
- Token-Length & Question-Difficulty Bias in Vanilla GRPO: Dividing the sequence policy loss by rollout length $\frac{1}{|o_i|}$ and normalizing advantages by group standard deviation $\mathrm{std}(\{R_1, \dots, R_G\})$ systematically penalizes concise correct answers while encouraging failed rollouts to ramble to the context limit (Liu et al., 2025, Dr. GRPO, arXiv:2503.20783).
0. Why Vanilla GRPO Rewards Rambling on Negative Rollouts – and How Dr. GRPO Fixes It
Inspect the per-token policy gradient weight in standard GRPO for rollout $i \in \{1, \dots, G\}$ of length $|o_i|$ with scalar outcome reward $R_i \in \mathbb{R}$:
In vanilla GRPO, when $\hat{A}_i \lt 0$ (an incorrect diagnosis or failed tool call), multiplying by $1/|o_i|$ dilutes the negative penalty per token on longer responses – actively training the policy to emit verbose filler when uncertain! Replacing $1/|o_i|$ with a constant token budget normalizer $1/C_{\max}$ and dropping $\mathrm{std}(R)$ restores an unbiased Monte Carlo policy gradient.
Anchoring Reasoning Likelihoods with Iterative RPO-DPO
Before running online Dr. GRPO, cold-starting a policy with RPO-DPO (Pang et al., 2024) adds a negative log-likelihood (NLL) anchor directly on the winning reasoning trace $y_w$ normalized by $|y_w|$:
1. Runnable PyTorch Reference: Dr. GRPO vs. Vanilla GRPO & Anti-Shotgun Rubric Verifier
The self-contained PyTorch script below quantifies the exact 4x token-gradient distortion between short and verbose rollouts in vanilla GRPO and implements the unbiased Dr. GRPO + RPO-DPO loss functions:
💻 Python: Unbiased Dr. GRPO, Iterative RPO-DPO & Anti-Shotgun Rubric VerifierPyTorch
import torch
import torch.nn.functional as F
def score_clinical_turn_rubric(
response_text: str, required_danger_signs: set[str], elicited_signs: set[str]
) -> float:
"""Deterministic rubric verifier with out-of-band shotgun-questioning penalty."""
recall = len(required_danger_signs & elicited_signs) / max(1, len(required_danger_signs))
# Count interrogative clauses to block 'shotgun questioning' reward hacking (>2 questions/turn)
q_count = response_text.count("?") + response_text.count("क्या")
shotgun_penalty = 0.4 * max(0, q_count - 2)
return max(0.0, recall - shotgun_penalty)
def compute_token_policy_weights(
rewards: torch.Tensor, lengths: torch.Tensor, c_max: int = 256, mode: str = "dr_grpo"
) -> torch.Tensor:
"""Compare per-token gradient scale between Vanilla GRPO and Dr. GRPO (arXiv:2503.20783)."""
# rewards: [G], lengths: [G]
if mode == "vanilla_grpo":
adv = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + 1e-8)
return adv / lengths # Biased: penalizes concise positive & rewards verbose negative!
# Dr. GRPO: zero-mean unscaled advantage divided by constant token budget c_max
adv_unbiased = rewards - rewards.mean()
return adv_unbiased / float(c_max)
def rpo_dpo_loss(
pi_logps_w: torch.Tensor, pi_logps_l: torch.Tensor,
ref_logps_w: torch.Tensor, ref_logps_l: torch.Tensor,
len_w: torch.Tensor, beta: float = 0.1, alpha_nll: float = 1.0
) -> torch.Tensor:
"""Iterative RPO objective: DPO loss + length-normalized NLL anchor (Pang et al., arXiv:2404.19733)."""
logits = (pi_logps_w - ref_logps_w) - (pi_logps_l - ref_logps_l)
dpo_term = -F.logsigmoid(beta * logits).mean()
nll_anchor = -(pi_logps_w / len_w).mean()
return dpo_term + alpha_nll * nll_anchor
if __name__ == "__main__":
rewards = torch.tensor([1.0, 1.0, 0.0, 0.0]) # 2 passing, 2 failing rollouts
lengths = torch.tensor([64.0, 256.0, 64.0, 256.0]) # Concise (64) vs Verbose (256)
w_vanilla = compute_token_policy_weights(rewards, lengths, mode="vanilla_grpo")
w_dr = compute_token_policy_weights(rewards, lengths, mode="dr_grpo")
print(f"[PASS] Vanilla GRPO Short/Long Token Weight Ratio: {(w_vanilla[0] / w_vanilla[1]).item():.1f}x (Length-Biased)")
print(f"[PASS] Dr. GRPO Short/Long Token Weight Ratio : {(w_dr[0] / w_dr[1]).item():.1f}x (Unbiased!)")
# Output:
# [PASS] Vanilla GRPO Short/Long Token Weight Ratio: 4.0x (Length-Biased)
# [PASS] Dr. GRPO Short/Long Token Weight Ratio : 1.0x (Unbiased!)
2. Post-Training Alignment & RLVR Objectives: Head-to-Head
| Objective | Vanilla DPO | Iterative RPO-DPO (arXiv:2404.19733) | Online Dr. GRPO (arXiv:2503.20783) |
|---|---|---|---|
| Chosen Log-Prob $\log \pi_\theta(y_w|x)$ | Can decrease alongside $\log \pi_\theta(y_l|x)$ | Strictly anchored via $-\alpha \frac{\log \pi_\theta(y_w|x)}{|y_w|}$ | Reinforced online on verified positive rollouts |
| Length Bias on Failed Rollouts | Sensitive to chosen vs. rejected length gap | Normalized by $|y_w|$ in NLL anchor | Eliminated by replacing $1/|o_i|$ with constant $1/C_{\max}$ |
| Question-Difficulty Scaling | Offline static pair margin | Iterative online pair construction | Unbiased (omits $1/\mathrm{std}(R)$ blowup on easy/hard prompts) |