Direct Preference Optimization (DPO)

Direct Preference Optimization (method="dpo") tunes a model on preference pairs: for the same prompt, one completion is chosen and one is rejected. The model learns to widen the gap between them, relative to a reference copy of the starting model. DPO is a simpler, more stable alternative to reward-model RLHF, and a common pass after SFT to align style and helpfulness. Veri runs DPO on TRL’s DPOTrainer. You provide a preference dataset; Veri provisions the GPU, runs the loop, streams metrics, and uploads the checkpoint. DPO takes no reward function — the preference pairs are the training signal.

Dataset format

Each row is a preference pair: a prompt, a chosen completion and a rejected completion. A row uses one of two shapes for all three fields, and every row of the dataset uses the same one. Standard: all three are strings. The trainer joins prompt + chosen and prompt + rejected as text.

{"prompt": "The sky is", "chosen": " blue.", "rejected": " green."}

Conversational: all three are message lists. Each field is a non-empty list of {"role", "content"} messages, and the trainer applies the model’s chat template.

{"prompt": [{"role": "user", "content": "What color is the sky?"}], "chosen": [{"role": "assistant", "content": "It is blue."}], "rejected": [{"role": "assistant", "content": "It is green."}]}

A row that mixes the shapes (a string prompt with message-list completions, or the reverse) is rejected at upload with a 400 that names the row and the field:

row 17: prompt is a string but chosen is a message list; a preference row must use one shape for all three fields

The same check runs when you submit a dpo job, before a GPU starts, so a dataset uploaded before the check, or a stream whose appends mixed shapes, is rejected at submit with the same message. Tombstone or correct the row and submit again. You can also leave prompt out of every row (the implicit-prompt form, such as {"chosen": "The sky is blue.", "rejected": "The sky is green."}): the trainer takes the shared start of chosen and rejected as the prompt. The one-shape rule still applies to chosen and rejected. Every row needs a chosen and a rejected completion.

Submitting a job

This mirrors TRL’s DPO quickstart (a small Qwen model on the UltraFeedback preference dataset), run on Veri:

from veri_sdk import Client

client = Client()

dataset = client.datasets.connect(
    name="ultrafeedback",
    source_type="hf",
    huggingface_dataset="trl-lib/ultrafeedback_binarized",
)

job = client.training_jobs.create(
    base_model="Qwen/Qwen3-0.6B",
    dataset_id=dataset.id,
    output_name="qwen-0.6b-dpo",
    method="dpo",
    hyperparameters={
        "learning_rate": 5e-6,
        "num_epochs": 1,
        "beta": 0.1,
    },
    gpu_type="L4-24GB",
    gpu_count=1,
)

job.wait(poll_interval=15)
job.download(output_dir="./checkpoints")

Hyperparameters

Parameter	Default	What it does
`learning_rate`	`5e-6`	AdamW step size. Use `≈1e-5` for LoRA adapters; full finetunes go lower (`≈1e-6`).
`num_epochs`	`1`	Passes over the dataset. Overridden by `max_steps` if both set.
`max_steps`	`null`	Total gradient steps. Set this to cap a long run or smoke-test.
`beta`	`0.1`	Strength of the preference signal (KL penalty against the reference policy). Lower stays closer to the base model.
`loss_type`	`"sigmoid"`	DPO loss variant. `"sigmoid"` is the original; `"hinge"`, `"ipo"`, and others are supported.
`max_length`	model default	Total tokens per example (prompt + completion). Big OOM lever.
`max_prompt_length`	model default	Tokens reserved for the prompt before the completion.
`lora_rank`	`null`	Train LoRA adapters of this rank instead of full finetuning. `null` means full finetune.
`lora_alpha`	`null`	LoRA scaling factor. Defaults to `lora_rank` when unset.
`load_in_4bit`	`false`	QLoRA: load the base in 4-bit (NF4) and train LoRA adapters on top. Requires `lora_rank` (a 400 at submit otherwise). Single-GPU.
`use_unsloth`	`false`	Opt-in Unsloth memory optimization. Off by default; single-GPU only.
`use_liger`	`false`	Opt-in liger-kernel fused training kernels. Fits much larger batches, but trains 15-27% slower per token. Full finetunes only (not with `lora_rank` or `load_in_4bit`). Not supported with `gpu_count` > 1, `use_unsloth`, or `loss_type` `"ipo"`, `"aot"`, `"aot_unpaired"`, `"sft"` or `"sigmoid_norm"`. Skews the loss on Gemma 2, Cohere and Granite models.
`micro_batch_size`	`1`	Samples per GPU per forward pass. With `gpu_count` above 1 the optimizer batch is `micro_batch_size x gpu_count` unless `global_batch_size` is set. See Multi-GPU training.
`gradient_accumulation_steps`	`1`	Micro-batches accumulated per optimizer step. Derived from `global_batch_size` when that is set.
`global_batch_size`	derived	Samples per optimizer step (`micro_batch_size x gradient_accumulation_steps x gpu_count`). Set it to keep training dynamics fixed across GPU counts.
`zero_stage`	`"auto"`	Multi-GPU sharding: `0` data parallel, `2` shard gradients and optimizer, `3` also shard parameters (FSDP2). `"auto"` picks `0` when the model states fit one GPU, else `3`. `gpu_count` above 1 only.
`precision`	`"bf16"`	`"bf16"`, `"bf16_mixed"` (fp32 master weights and optimizer, bf16 compute) or `"fp32"`.
`gradient_checkpointing`	`true`	Activation recomputation. Turn off to trade memory for about a third less compute per step.
`cpu_offload`	`false`	Park sharded parameters and optimizer states in host RAM (FSDP2). Needs `zero_stage` `2` or `3`; `gpu_count` above 1 only.

LoRA and QLoRA

DPO uses the same adapter path as GRPO and SFT. Set lora_rank to train low-rank adapters instead of the full model, and add load_in_4bit for QLoRA (a 4-bit NF4 base with adapters on top), which fits larger models on a single GPU.

# QLoRA DPO on a single GPU
hyperparameters={
    "learning_rate": 1e-5,
    "beta": 0.1,
    "lora_rank": 16,
    "lora_alpha": 32,
    "load_in_4bit": True,
}

See LoRA and QLoRA for the full rules (4-bit requires lora_rank; QLoRA is single-GPU; 16-bit LoRA is multi-GPU capable).

DPO compares the policy against a frozen reference copy of the starting model. With LoRA or QLoRA, that reference is the same base with the adapter disabled, so there is no extra copy. A full finetune (no lora_rank) instead holds a second reference model in memory, roughly doubling the VRAM needed versus SFT or GRPO on the same base. On tight VRAM, prefer LoRA or QLoRA for DPO.

Where to go next

Managed Training

Supervised finetune first, then align with DPO.

Datasets

Connect a preference dataset.

Submit a job

The full training_jobs.create schema.

Deploy your checkpoint

Serve the trained model with an OpenAI-compatible API.

Get started

Training

Deployments

Evaluations

Agents

CLI

Volumes

Webhooks

Secrets

Billing

Direct Preference Optimization (DPO)

Dataset format

Submitting a job

Hyperparameters

LoRA and QLoRA

Where to go next

Managed Training

Datasets

Submit a job

Deploy your checkpoint

​Dataset format

​Submitting a job

​Hyperparameters

​LoRA and QLoRA

​Where to go next

Managed Training

Datasets

Submit a job

Deploy your checkpoint

Dataset format

Submitting a job

Hyperparameters

LoRA and QLoRA

Where to go next