> ## Documentation Index
> Fetch the complete documentation index at: https://docs.veri.studio/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets
> JSONL upload + Hugging Face connect + Veri Volumes. Validation, dedup.
A dataset is a collection of training examples Veri reads at job submission time. Three ways to get one in:
1. **Upload** a JSONL file directly
2. **Connect** a Hugging Face dataset — Veri pulls from the Hub
3. **Point at a [Veri Volume](/volumes)** — train from a JSONL file you already uploaded to a managed volume
Either way, the result is a dataset record with an `id` you reference in `client.training_jobs.create(dataset_id=...)`.
## JSONL format
Each line is one training example:
```json theme={null}
{"prompt": [{"role": "user", "content": "What is 12 + 7?"}], "answer": "19"}
{"prompt": "What is 45 - 18?", "answer": "27"}
```
* **`prompt`** (required) — string OR list of OpenAI-format message dicts
* **`answer`** — ground truth used by your reward function (optional but typically present). The column name is passed to your reward as a keyword argument of the same name, so it must match what the function reads. Every example here uses `answer`; if you map it to another name, read that name instead.
The validation step at upload rejects any row missing `prompt`.
## Source types
| `source_type` | Required fields | Notes |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| `upload` | multipart `file` + `name` (via `POST /v1/datasets`) | JSONL only; one row per line |
| `hf` | `huggingface_dataset`, `huggingface_config` (split + column\_mapping; optional `config_name` for multi-config datasets like GSM8K) | Hugging Face Hub |
| `volume` | `volume` (name), `volume_path` (absolute path inside the volume, e.g. `/train.jsonl`) | Train from a file in a managed [Veri Volume](/volumes) |
## Connect from Hugging Face
Most users start here — pull `openai/gsm8k` straight from HF, no local download.
The `column_mapping` tells Veri which source columns become `prompt` and `answer`.
```python SDK theme={null}
client.datasets.connect(
name="gsm8k-train",
source_type="hf",
huggingface_dataset="openai/gsm8k",
huggingface_config={
"split": "train",
"subset": "main", # GSM8K requires a config: "main" or "socratic"
"column_mapping": {
"question": "prompt",
"answer": "answer",
},
},
)
```
```bash CLI theme={null}
veri datasets connect-hf openai/gsm8k \
--config main \
--map question=prompt --map answer=answer \
--name gsm8k-train
```
Private and gated Hugging Face datasets are not supported yet; connect public datasets, or upload the data directly.
## Supported formats
Veri detects the dataset's format from its columns and converts it to the shape the trainer expects, so you can connect community datasets as they are:
| Format | Columns | What Veri does |
| ----------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Chat (ChatML) | `messages: [{role, content}]` | Used as is. |
| ShareGPT | `conversations: [{from, value}]` | Converted to `messages`; roles like `human`/`gpt` (also `prompter`, `USER`) are mapped to `user`/`assistant`. |
| Alpaca | `instruction`, `input`, `output` (optional `system`) | Composed into a `messages` conversation. |
| Plain text | `text` | Used as is. |
| Prompt-completion | `prompt`, `completion` | Used as is. |
| Preference | `prompt`, `chosen`, `rejected` | Used as is (DPO). |
Datasets that don't match any of these formats, or use tool-calling roles (`observation`, `function_call`), are rejected with a `dataset_format_incompatible` error at the start of the job rather than trained on incorrectly. Use `column_mapping` to rename columns into one of the shapes above.
Each method still needs the right shape: SFT trains on `text`, `messages`, or prompt-completion rows; GRPO needs a `prompt` column; DPO needs `chosen` and `rejected`.
### Snapshot caching
The first training job on a connected HF dataset downloads it from the Hub, converts it, and caches the result. Later jobs on the same dataset read the cache directly: faster starts, and your data can't change mid-experiment if the upstream dataset is edited. To pick up upstream changes, connect the dataset again under a new name.
## Connect from a Veri Volume
If your data is already in a [Veri Volume](/volumes), point a dataset at a JSONL file inside it:
```python theme={null}
dataset = client.datasets.from_volume(
volume="my-corpus",
path="/train.jsonl",
name="my-corpus-train",
)
```
The dataset stays linked to the volume — re-uploading the file inside the volume updates the data for the next training run.
## Validate before connecting
Test the connection + preview the first row + see total row count without committing.
```python theme={null}
result = client.datasets.validate(
source_type="hf",
huggingface_dataset="openai/gsm8k",
huggingface_config={"split": "train", "subset": "main", "column_mapping": {"question": "prompt", "answer": "answer"}},
)
print(f"Valid: {result['valid']}")
print(f"Rows: {result['num_rows']}")
print(f"Columns: {result['columns']}")
```
If `result.valid` is `false`, `result.error` tells you why.
## Upload a JSONL file
For one-off experiments:
```python SDK theme={null}
dataset = client.datasets.upload(
"training_data.jsonl",
name="my-experiment",
)
```
```bash CLI theme={null}
veri datasets upload training_data.jsonl --name my-experiment
```
The CLI runs structural validation by default. Pass `--skip-validation` to skip, `--deep-check` for tokenizer-aware checks (slower).
## Dedup
Datasets are content-addressed. Uploading the same bytes twice (same file, or two files with identical content) returns the same `id` — one S3 object, one DB row.
This means you can re-run a script that uploads the same dataset without worrying about stale duplicates.
## Lifecycle
Datasets persist until you delete them from the dashboard. There is no API delete endpoint today — programmatic delete is on the roadmap.
## Name conflicts
Connecting two datasets with the same `name` but different source config returns `409 dataset_name_conflict`. Either pick a unique name or reuse the existing dataset (the identity hash on the source config means the same config returns the same `id`).
## Where to go next
Score completions with TRL-format functions.
Use your dataset\_id in `training_jobs.create`.
Full GRPO end-to-end with GSM8K.
Run Unsloth's Qwen3 QLoRA recipe as a managed SFT job.