Datasets

A dataset is a collection of training examples Veri reads at job submission time. Three ways to get one in:

Upload a JSONL file directly
Connect a Hugging Face dataset — Veri pulls from the Hub
Point at a Veri Volume — train from a JSONL file you already uploaded to a managed volume

Either way, the result is a dataset record with an id you reference in client.training_jobs.create(dataset_id=...).

JSONL format

Each line is one training example:

{"prompt": [{"role": "user", "content": "What is 12 + 7?"}], "answer": "19"}
{"prompt": "What is 45 - 18?", "answer": "27"}

prompt (required) — string OR list of OpenAI-format message dicts
answer — ground truth used by your reward function (optional but typically present). The column name is passed to your reward as a keyword argument of the same name, so it must match what the function reads. Every example here uses answer; if you map it to another name, read that name instead.

The validation step at upload rejects any row missing prompt.

Source types

`source_type`	Required fields	Notes
`upload`	multipart `file` + `name` (via `POST /v1/datasets`)	JSONL only; one row per line
`hf`	`huggingface_dataset`, `huggingface_config` (split + column_mapping; optional `config_name` for multi-config datasets like GSM8K)	Hugging Face Hub
`volume`	`volume` (name), `volume_path` (absolute path inside the volume, e.g. `/train.jsonl`)	Train from a file in a managed Veri Volume

Connect from Hugging Face

Most users start here — pull openai/gsm8k straight from HF, no local download. The column_mapping tells Veri which source columns become prompt and answer.

client.datasets.connect(
    name="gsm8k-train",
    source_type="hf",
    huggingface_dataset="openai/gsm8k",
    huggingface_config={
        "split": "train",
        "subset": "main",  # GSM8K requires a config: "main" or "socratic"
        "column_mapping": {
            "question": "prompt",
            "answer": "answer",
        },
    },
)

veri datasets connect-hf openai/gsm8k \
  --config main \
  --map question=prompt --map answer=answer \
  --name gsm8k-train

Private and gated Hugging Face datasets are not supported yet; connect public datasets, or upload the data directly.

Supported formats

Veri detects the dataset’s format from its columns and converts it to the shape the trainer expects, so you can connect community datasets as they are:

Format	Columns	What Veri does
Chat (ChatML)	`messages: [{role, content}]`	Used as is.
ShareGPT	`conversations: [{from, value}]`	Converted to `messages`; roles like `human`/`gpt` (also `prompter`, `USER`) are mapped to `user`/`assistant`.
Alpaca	`instruction`, `input`, `output` (optional `system`)	Composed into a `messages` conversation.
Plain text	`text`	Used as is.
Prompt-completion	`prompt`, `completion`	Used as is.
Preference	`prompt`, `chosen`, `rejected`	Used as is (DPO).

Datasets that don’t match any of these formats, or use tool-calling roles (observation, function_call), are rejected with a dataset_format_incompatible error at the start of the job rather than trained on incorrectly. Use column_mapping to rename columns into one of the shapes above. Each method still needs the right shape: SFT trains on text, messages, or prompt-completion rows; GRPO needs a prompt column; DPO needs chosen and rejected.

Snapshot caching

The first training job on a connected HF dataset downloads it from the Hub, converts it, and caches the result. Later jobs on the same dataset read the cache directly: faster starts, and your data can’t change mid-experiment if the upstream dataset is edited. To pick up upstream changes, connect the dataset again under a new name.

Connect from a Veri Volume

If your data is already in a Veri Volume, point a dataset at a JSONL file inside it:

dataset = client.datasets.from_volume(
    volume="my-corpus",
    path="/train.jsonl",
    name="my-corpus-train",
)

The dataset stays linked to the volume — re-uploading the file inside the volume updates the data for the next training run.

Validate before connecting

Test the connection + preview the first row + see total row count without committing.

result = client.datasets.validate(
    source_type="hf",
    huggingface_dataset="openai/gsm8k",
    huggingface_config={"split": "train", "subset": "main", "column_mapping": {"question": "prompt", "answer": "answer"}},
)

print(f"Valid: {result['valid']}")
print(f"Rows: {result['num_rows']}")
print(f"Columns: {result['columns']}")

If result.valid is false, result.error tells you why.

Upload a JSONL file

For one-off experiments:

dataset = client.datasets.upload(
    "training_data.jsonl",
    name="my-experiment",
)

veri datasets upload training_data.jsonl --name my-experiment

The CLI runs structural validation by default. Pass --skip-validation to skip, --deep-check for tokenizer-aware checks (slower).

Dedup

Datasets are content-addressed. Uploading the same bytes twice (same file, or two files with identical content) returns the same id — one S3 object, one DB row. This means you can re-run a script that uploads the same dataset without worrying about stale duplicates.

Lifecycle

Datasets persist until you delete them from the dashboard. There is no API delete endpoint today — programmatic delete is on the roadmap.

Name conflicts

Connecting two datasets with the same name but different source config returns 409 dataset_name_conflict. Either pick a unique name or reuse the existing dataset (the identity hash on the source config means the same config returns the same id).

Where to go next

Reward functions

Score completions with TRL-format functions.

Submit a training job

Use your dataset_id in training_jobs.create.

Quickstart

Full GRPO end-to-end with GSM8K.

SFT with Unsloth demo

Run Unsloth’s Qwen3 QLoRA recipe as a managed SFT job.

Get started

Training

Deployments

Evaluations

CLI

Volumes

JSONL format

Source types

Connect from Hugging Face

Supported formats

Snapshot caching

Connect from a Veri Volume

Validate before connecting

Upload a JSONL file

Dedup

Lifecycle

Name conflicts

Where to go next

Reward functions

Submit a training job

Quickstart

SFT with Unsloth demo

​JSONL format

​Source types

​Connect from Hugging Face

​Supported formats

​Snapshot caching

​Connect from a Veri Volume

​Validate before connecting

​Upload a JSONL file

​Dedup

​Lifecycle

​Name conflicts

​Where to go next

Reward functions

Submit a training job

Quickstart

SFT with Unsloth demo

JSONL format

Source types

Connect from Hugging Face

Supported formats

Snapshot caching

Connect from a Veri Volume

Validate before connecting

Upload a JSONL file

Dedup

Lifecycle

Name conflicts

Where to go next