> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# GRPO algorithm
> Group Relative Policy Optimization — the RL algorithm behind DeepSeek R1. Intuition, hyperparameters, and how to read the metrics.
GRPO (Group Relative Policy Optimization) is the algorithm that powers DeepSeek R1 and is Veri's default training method. It generates **K rollouts per prompt**, scores each with your reward function, and reinforces completions that scored above the group's average.
This page covers the intuition + every hyperparameter Veri exposes.
## The core loop
For each prompt in a batch:
1. **Sample** `rollouts_per_prompt` completions from the current policy
2. **Score** every completion with your reward function
3. **Compute advantage**: `(reward - group_mean) / (group_std + epsilon)`
4. **Update policy** to make above-average completions more likely (bounded by KL distance from the reference model)
The "group relative" part is what makes GRPO simpler than PPO — no value head, no separate critic, no GAE. The rollouts themselves provide the baseline.
## Hyperparameters
Every field of the GRPO hyperparameters, with defaults:
| Parameter | Default | What it does |
| --------------------- | ------- | ----------------------------------------------------------------------------------------------------- |
| `learning_rate` | `1e-6` | AdamW step size. Stable for most LLM RL. |
| `num_epochs` | `1` | Passes over the dataset. Overridden by `max_steps` if both set. |
| `max_steps` | `null` | Total gradient steps. Set this to cap a long run. |
| `rollouts_per_prompt` | `8` | Group size. Higher = better advantage estimates, more compute per step. |
| `kl_coef` | `0.001` | KL penalty weight against the reference (frozen) model. Prevents drift. |
| `max_prompt_length` | `1024` | Tokens. Truncate longer prompts. |
| `max_response_length` | `2048` | Tokens. Truncate longer completions. Big OOM lever. |
| `global_batch_size` | `64` | Total batch across all GPUs. Auto-split per device. |
| `seed` | `42` | RNG seed for reproducibility. |
| `lora_rank` | `null` | Train LoRA adapters of this rank instead of full finetuning. `null` means full finetune. |
| `lora_alpha` | `null` | LoRA scaling factor. Defaults to `lora_rank` when unset. |
| `load_in_4bit` | `false` | QLoRA: load the base in 4-bit (NF4) and train LoRA adapters on top. Requires `lora_rank`. Single-GPU. |
| `use_unsloth` | `false` | Opt-in Unsloth memory optimization. Off by default; vanilla TRL is the base path. Single-GPU only. |
## LoRA and QLoRA
By default Veri full-finetunes the base model. To train low-rank adapters instead, set `lora_rank` (and optionally `lora_alpha`). For QLoRA, also set `load_in_4bit`: the base loads in 4-bit (NF4) and you train adapters on top, which cuts VRAM roughly 4x and lets larger models fit on a single GPU.
**LoRA (16-bit base, multi-GPU capable):**
```python theme={null}
hyperparameters={
"lora_rank": 16,
"lora_alpha": 32,
}
```
**QLoRA (4-bit base, single GPU):**
```python theme={null}
hyperparameters={
"lora_rank": 16,
"lora_alpha": 32,
"load_in_4bit": True,
}
```
`load_in_4bit` requires `lora_rank`: the 4-bit base is frozen, so training happens in the adapters. QLoRA runs on a single GPU today; for multi-GPU, use 16-bit LoRA (`lora_rank` without `load_in_4bit`). Adapters target the attention and MLP projections (q/k/v/o, gate/up/down).
## Picking values
**For first runs, change two things:**
* `max_response_length` — drop to `512` if you're hitting OOM, raise to `4096` for tasks that need long outputs (reasoning, code)
* `max_steps` — set to `50–100` for smoke tests, then drop the cap when the loop looks healthy
**Everything else holds for most tasks:**
* `learning_rate=1e-6` — stable across model sizes
* `rollouts_per_prompt=8` — sweet spot for advantage signal vs. compute. Drop to `4` only if you're cost-constrained
* `kl_coef=0.001` — conservative; raise to `0.01` if the model drifts too fast (incoherent outputs)
## Common failure modes
**Policy collapse** — `policy_entropy → 0`, model produces identical outputs for every prompt. Cause: KL coefficient too low, learning rate too high, or reward function gives the same score to too many completions. Lower learning\_rate, raise kl\_coef.
**Reward hacking** — Reward goes up, but outputs become weird (gaming the reward function rather than solving the task). Tighten the reward — add format checks, length penalties, or a frontier judge as a sanity scorer.
**Length explosion** — `response_length_mean` grows toward `max_response_length` without reward improving. Add a length penalty to the reward, or drop `max_response_length`.
## Reading the metrics
Pull metrics with `GET /v1/training_jobs/{id}/metrics` (or the dashboard if you're integrated with W\&B):
```python theme={null}
metrics = client.training_jobs.get(job_id).metrics # latest sample
# Or for the full series:
# httpx.get(f"{base_url}/v1/training_jobs/{job_id}/metrics?keys=loss,kl_div,reward_mean")
```
| Metric | What healthy looks like |
| ---------------------- | ----------------------------------------------------------------------------------- |
| `reward_mean` | Climbs over training. Plateaus at the ceiling of your reward design. |
| `reward_std` | Stays > 0. If it collapses to \~0, all completions look the same — policy collapse. |
| `kl_div` | Stays bounded. If it grows unboundedly, the policy is drifting; raise `kl_coef`. |
| `loss` | Decreases overall, but expect noise — RL loss is not as clean as supervised. |
| `response_length_mean` | Stable or modestly growing. Steep growth = length explosion. |
| `policy_entropy` | Slow decline as the model becomes more confident. Steep decline = collapse risk. |
## Sample hyperparameter blocks
**Smoke test (small model, quick iteration):**
```python theme={null}
hyperparameters={
"learning_rate": 1e-6,
"rollouts_per_prompt": 4,
"max_response_length": 512,
"kl_coef": 0.001,
"max_steps": 50,
}
```
**Production (Qwen2.5-7B class):**
```python theme={null}
hyperparameters={
"learning_rate": 1e-6,
"rollouts_per_prompt": 8,
"max_response_length": 2048,
"kl_coef": 0.001,
"num_epochs": 1,
}
```
**Reasoning-heavy task (long chain-of-thought):**
```python theme={null}
hyperparameters={
"learning_rate": 5e-7,
"rollouts_per_prompt": 8,
"max_response_length": 4096,
"kl_coef": 0.005,
"num_epochs": 2,
}
```
## When NOT to use GRPO
If your model can't follow the *format* you want yet (e.g. won't wrap answers in `` tags), do an SFT pass first to teach the structure, then GRPO to reinforce correctness. RL works better when the policy already has non-zero probability of producing the right shape.
## Where to go next
Design rewards GRPO can actually optimize against.
Get training data into Veri.
Wire it all into `training_jobs.create`.
Planned math reasoning walkthrough with metrics interpretation.