GRPO algorithm

GRPO (Group Relative Policy Optimization) is the algorithm that powers DeepSeek R1 and is Veri’s default training method. It generates K rollouts per prompt, scores each with your reward function, and reinforces completions that scored above the group’s average. This page covers the intuition + every hyperparameter Veri exposes.

The core loop

For each prompt in a batch:

Sample rollouts_per_prompt completions from the current policy
Score every completion with your reward function
Compute advantage: (reward - group_mean) / (group_std + epsilon)
Update policy to make above-average completions more likely (bounded by KL distance from the reference model)

The “group relative” part is what makes GRPO simpler than PPO — no value head, no separate critic, no GAE. The rollouts themselves provide the baseline.

Hyperparameters

Every field of the GRPO hyperparameters, with defaults:

Parameter	Default	What it does
`learning_rate`	`1e-6`	AdamW step size, sized for full finetuning. With LoRA (`lora_rank`), use a LoRA-scale rate; the L4 example uses `2e-5`.
`num_epochs`	`1`	Not used by GRPO. The run length is `max_steps`.
`max_steps`	`100`	Total training steps (`100` when omitted). Each step trains on one prompt and its `rollouts_per_prompt` completions, so one pass over an N-prompt dataset is N steps.
`rollouts_per_prompt`	`8`	Group size. Higher = better advantage estimates, more compute per step.
`kl_coef`	`0.0`	KL penalty weight against the reference (frozen) model. When omitted it is `0.0`: no KL penalty and no reference model in memory. Set it (for example `0.001`) to limit drift.
`max_prompt_length`	`1024`	Tokens. Only applies with `use_unsloth`, where it sizes the context window together with `max_response_length`. The default trainer does not truncate prompts, so without `use_unsloth` a job that sets it is rejected with a 400.
`max_response_length`	`2048`	Tokens. Truncate longer completions. Big OOM lever.
`global_batch_size`	none	Not supported by GRPO: a job that sets it is rejected with a 400. Each step’s batch is one prompt’s `rollouts_per_prompt` completions.
`seed`	`42`	Random seed for the trainer (sampling and initialization). Set it to make a run repeatable.
`system_prompt`	`null`	When set, each string prompt becomes a system + user chat, so the model’s chat template applies. Use it to tell the model the output format your reward checks.
`lora_rank`	`null`	Train LoRA adapters of this rank instead of full finetuning. `null` means full finetune.
`lora_alpha`	`null`	LoRA scaling factor. Defaults to `lora_rank` when unset.
`load_in_4bit`	`false`	QLoRA: load the base in 4-bit (NF4) and train LoRA adapters on top. Requires `lora_rank` (a 400 at submit otherwise). Single-GPU.
`use_unsloth`	`false`	Opt-in Unsloth memory optimization. Off by default; vanilla TRL is the base path. Single-GPU only.
`use_liger`	`false`	Opt-in liger-kernel fused training kernels. Lowers peak memory but does not train faster. The entropy metric is not reported. Not supported with `gpu_count` > 1, `use_unsloth`, or GPT-NeoX/Pythia models; computes wrong log-probs on Gemma 2, Cohere and Granite models.

LoRA and QLoRA

By default Veri full-finetunes the base model. To train low-rank adapters instead, set lora_rank (and optionally lora_alpha). For QLoRA, also set load_in_4bit: the base loads in 4-bit (NF4) and you train adapters on top, which cuts VRAM roughly 4x and lets larger models fit on a single GPU. LoRA (16-bit base, multi-GPU capable):

hyperparameters={
    "lora_rank": 16,
    "lora_alpha": 32,
}

QLoRA (4-bit base, single GPU):

hyperparameters={
    "lora_rank": 16,
    "lora_alpha": 32,
    "load_in_4bit": True,
}

load_in_4bit requires lora_rank: the 4-bit base is frozen, so training happens in the adapters. QLoRA runs on a single GPU today; for multi-GPU, use 16-bit LoRA (lora_rank without load_in_4bit). Adapters target the attention and MLP projections (q/k/v/o, gate/up/down).

What a LoRA job produces. The final checkpoint is the adapter plus tokenizer (artifact_type: "adapter" on GET /v1/training_jobs/{id}/model), and Veri also keeps the merged weights (adapter merged into the base model, merged_files on the same call). A LoRA job deploys like a full fine-tune (veri deploy <job_id>, or source="training_job" in the API) and is served from the merged weights; saving it to your model library saves the merged weights too, and artifact="adapter" saves the adapter alone for multi-adapter fleets. To publish either artifact to Hugging Face, see Hugging Face.

Picking values

For first runs, change two things:

max_response_length: drop to 512 if you’re hitting OOM, raise to 4096 for tasks that need long outputs (reasoning, code)
max_steps: set to 50 to 100 for smoke tests, then raise it for the real run (omitting it means 100 steps)

Everything else holds for most tasks:

learning_rate=1e-6 for full finetuning; with LoRA, a LoRA-scale rate such as 2e-5
rollouts_per_prompt=8: sweet spot for advantage signal vs. compute. Drop to 4 only if you’re cost-constrained
kl_coef: leave it unset (0.0, no reference model). If outputs drift into incoherence, set 0.001 to 0.01

Common failure modes

Policy collapse: entropy → 0, model produces identical outputs for every prompt. Cause: KL coefficient too low, learning rate too high, or reward function gives the same score to too many completions. Lower learning_rate, raise kl_coef. Reward hacking — Reward goes up, but outputs become weird (gaming the reward function rather than solving the task). Tighten the reward — add format checks, length penalties, or a frontier judge as a sanity scorer. Length explosion: completions/mean_length grows toward max_response_length without reward improving. Add a length penalty to the reward, or drop max_response_length.

Reading the metrics

Pull metrics with GET /v1/training_jobs/{id}/metrics (or the dashboard if you’re integrated with W&B):

series = client.training_jobs.metrics(job_id, keys=["reward", "reward_std", "loss"])
for point in series["metrics"]:
    print(point["step"], point["values"])

Metric	What healthy looks like
`reward`	Mean reward per step. Climbs over training. Plateaus at the ceiling of your reward design.
`reward_std`	Stays > 0. If it collapses to ~0, all completions look the same (policy collapse).
`kl`	Reported only when `kl_coef` is above 0. Stays bounded. If it grows unboundedly, the policy is drifting; raise `kl_coef`.
`loss`	Expect noise: RL loss is not as clean as supervised and does not need to decrease.
`completions/mean_length`	Stable or modestly growing. Steep growth = length explosion.
`entropy`	Slow decline as the model becomes more confident. Steep decline = collapse risk.

Sample hyperparameter blocks

Smoke test (small model on one L4, LoRA):

hyperparameters={
    "max_steps": 50,
    "rollouts_per_prompt": 8,
    "max_response_length": 512,
    "lora_rank": 16,
    "lora_alpha": 32,
    "learning_rate": 2e-5,
}

Production (Qwen2.5-7B class):

hyperparameters={
    "learning_rate": 1e-6,
    "rollouts_per_prompt": 8,
    "max_response_length": 2048,
    "kl_coef": 0.001,
    "max_steps": 1000,  # GRPO runs max_steps steps; num_epochs is not used
}

Reasoning-heavy task (long chain-of-thought):

hyperparameters={
    "learning_rate": 5e-7,
    "rollouts_per_prompt": 8,
    "max_response_length": 4096,
    "kl_coef": 0.005,
    "max_steps": 2000,
}

When NOT to use GRPO

If your model can’t follow the format you want yet (e.g. won’t wrap answers in <answer> tags), do an SFT pass first to teach the structure, then GRPO to reinforce correctness. RL works better when the policy already has non-zero probability of producing the right shape.

Where to go next

Reward functions

Design rewards GRPO can actually optimize against.

Datasets

Get training data into Veri.

Submit a job

Wire it all into training_jobs.create.

Get started

Training

Deployments

Evaluations

Agents

CLI

Volumes

Webhooks

Secrets

Billing

The core loop

Hyperparameters

LoRA and QLoRA

Picking values

Common failure modes

Reading the metrics

Sample hyperparameter blocks

When NOT to use GRPO

Where to go next

Reward functions

Datasets

Submit a job

​The core loop

​Hyperparameters

​LoRA and QLoRA

​Picking values

​Common failure modes

​Reading the metrics

​Sample hyperparameter blocks

​When NOT to use GRPO

​Where to go next

Reward functions

Datasets

Submit a job

The core loop

Hyperparameters

LoRA and QLoRA

Picking values

Common failure modes

Reading the metrics

Sample hyperparameter blocks

When NOT to use GRPO

Where to go next