Datasets

A dataset is a collection of training examples Veri reads at job submission time. Four ways to get one in:

Upload a JSONL file directly
Connect a Hugging Face dataset — Veri pulls from the Hub
Point at a Veri Volume — train from a JSONL file you already uploaded to a managed volume
Create a dataset stream: an append-only dataset that grows as data arrives; training pins immutable snapshots (name@latest, name@snap-N)

Either way, the result is a dataset record with an id you reference in client.training_jobs.create(dataset_id=...). This page covers the one-shot sources (upload, Hugging Face, volumes); streams have their own page.

JSONL format

Each line is one training example:

{"prompt": [{"role": "user", "content": "What is 12 + 7?"}], "answer": "19"}
{"prompt": "What is 45 - 18?", "answer": "27"}

prompt (required) — string OR list of OpenAI-format message dicts
answer — ground truth used by your reward function (optional but typically present). The column name is passed to your reward as a keyword argument of the same name, so it must match what the function reads. Every example here uses answer; if you map it to another name, read that name instead.

The validation step at upload rejects any row missing prompt.

Source types

`source_type`	Required fields	Notes
`upload`	multipart `file` + `name` (via `POST /v1/datasets`)	JSONL only; one row per line
`hf`	`huggingface_dataset`, `huggingface_config` (split + column_mapping; optional `config_name` for multi-config datasets like GSM8K)	Hugging Face Hub
`volume`	`volume` (name), `volume_path` (absolute path inside the volume, e.g. `/train.jsonl`)	Train from a file in a managed Veri Volume

Connect from Hugging Face

Most users start here — pull openai/gsm8k straight from HF, no local download. The column_mapping tells Veri which source columns become prompt and answer.

client.datasets.connect(
    name="gsm8k-train",
    source_type="hf",
    huggingface_dataset="openai/gsm8k",
    huggingface_config={
        "split": "train",
        "subset": "main",  # GSM8K requires a config: "main" or "socratic"
        "column_mapping": {
            "question": "prompt",
            "answer": "answer",
        },
    },
)

veri datasets connect-hf openai/gsm8k \
  --config main \
  --map question=prompt --map answer=answer \
  --name gsm8k-train

Private and gated Hugging Face datasets are not supported yet; connect public datasets, or upload the data directly.

Supported formats

Veri detects the dataset’s format from its columns and converts it to the shape the trainer expects, so you can connect community datasets as they are:

Format	Columns	What Veri does
Chat (ChatML)	`messages: [{role, content}]`	Used as is.
ShareGPT	`conversations: [{from, value}]`	Converted to `messages`; roles like `human`/`gpt` (also `prompter`, `USER`) are mapped to `user`/`assistant`.
Alpaca	`instruction`, `input`, `output` (optional `system`)	Composed into a `messages` conversation.
Plain text	`text`	Used as is.
Prompt-completion	`prompt`, `completion`	Used as is.
Preference	`prompt`, `chosen`, `rejected`	Used as is (DPO).

Datasets that don’t match any of these formats, or use tool-calling roles (observation, function_call), are rejected with a dataset_format_incompatible error at the start of the job rather than trained on incorrectly. Use column_mapping to rename columns into one of the shapes above. Detection and conversion run in the open-source runtime — see detect_dataset_format and normalize_rows in training_runtime.py to check exactly what your dataset passed through. Each method still needs the right shape: SFT trains on text, messages, or prompt-completion rows; GRPO needs a prompt column; DPO needs chosen and rejected.

Snapshot caching

The first training job on a connected HF dataset downloads it from the Hub, converts it, and caches the result. Later jobs on the same dataset read the cache directly: faster starts, and your data can’t change mid-experiment if the upstream dataset is edited. To pick up upstream changes, connect the dataset again under a new name.

Connect from a Veri Volume

If your data is already in a Veri Volume, point a dataset at a JSONL file inside it:

dataset = client.datasets.from_volume(
    volume="my-corpus",
    path="/train.jsonl",
    name="my-corpus-train",
)

The dataset stays linked to the volume — re-uploading the file inside the volume updates the data for the next training run.

Validate before connecting

Test the connection + preview the first row + see total row count without committing.

result = client.datasets.validate(
    source_type="hf",
    huggingface_dataset="openai/gsm8k",
    huggingface_config={"split": "train", "subset": "main", "column_mapping": {"question": "prompt", "answer": "answer"}},
)

print(f"Valid: {result['valid']}")
print(f"Rows: {result['num_rows']}")
print(f"Columns: {result['columns']}")

If result.valid is false, result.error tells you why.

Upload a JSONL file

For one-off experiments:

dataset = client.datasets.upload(
    "training_data.jsonl",
    name="my-experiment",
)

veri datasets upload training_data.jsonl --name my-experiment

The CLI runs structural validation by default. Pass --skip-validation to skip, --deep-check for tokenizer-aware checks (slower).

Dedup

Datasets are content-addressed. Uploading the same bytes twice (same file, or two files with identical content) returns the same id — one S3 object, one DB row. This means you can re-run a script that uploads the same dataset without worrying about stale duplicates.

Preview rows

Click a dataset on the dashboard’s Datasets page to open a row viewer that pages through the stored JSONL 50 rows at a time, so large datasets never download in full. The row number in the footer is editable — type a row number and press Enter to jump straight there (the first jump into a large dataset builds a small row index, so it can take a moment; later jumps are instant). Uploaded datasets are previewable immediately. Connected datasets (Hugging Face or volume) become previewable after their first training run creates the snapshot. The same paging is available programmatically via GET /v1/datasets/{dataset_id}/rows, with cursor for sequential pages or start_row to jump to an absolute row.

Lifecycle

Datasets persist until you delete them, either from the dashboard or via DELETE /v1/datasets/{dataset_id}. Deletion is blocked while an active training job still references the dataset.

Name conflicts

Connecting two datasets with the same name but different source config returns 409 dataset_name_conflict. Either pick a unique name or reuse the existing dataset (the identity hash on the source config means the same config returns the same id).

Where to go next

Reward functions

Write TRL-format functions and attach them to jobs inline.

Submit a training job

Use your dataset_id in training_jobs.create.

Quickstart

Full GRPO end-to-end with GSM8K.

SFT with Unsloth demo

Run Unsloth’s Qwen3 QLoRA recipe as a managed SFT job.

Get started

Training

Deployments

CLI

Volumes

Webhooks

JSONL format

Source types

Connect from Hugging Face

Supported formats

Snapshot caching

Connect from a Veri Volume

Validate before connecting

Upload a JSONL file

Dedup

Preview rows

Lifecycle

Name conflicts

Where to go next

Reward functions

Submit a training job

Quickstart

SFT with Unsloth demo

​JSONL format

​Source types

​Connect from Hugging Face

​Supported formats

​Snapshot caching

​Connect from a Veri Volume

​Validate before connecting

​Upload a JSONL file

​Dedup

​Preview rows

​Lifecycle

​Name conflicts

​Where to go next

Reward functions

Submit a training job

Quickstart

SFT with Unsloth demo

JSONL format

Source types

Connect from Hugging Face

Supported formats

Snapshot caching

Connect from a Veri Volume

Validate before connecting

Upload a JSONL file

Dedup

Preview rows

Lifecycle

Name conflicts

Where to go next