veri datasets

The veri datasets group covers every dataset operation: upload local JSONL, connect a HuggingFace dataset, validate structure, inspect registered datasets, and manage append-only dataset streams.

Commands

Command	Purpose
`upload <path>`	Upload a JSONL file. Auto-runs structural validation.
`connect-hf <repo>`	Connect a HuggingFace dataset.
`check <path>`	Standalone structural validator.
`preview <path> [--n N]`	Print the first N rows of a local JSONL.
`list`	List registered datasets.
`get <id>`	Show one dataset (streams: head, unsnapshotted rows, snapshot timeline).
`delete <id>`	Delete a dataset.
`create <name> --format F`	Create an empty append-only stream with a locked row format.
`append <name> <file\|->`	Append JSONL rows from a file or stdin (`--supersedes <row>` to correct a row).
`tombstone <name> <row>`	Remove a row from snapshots cut from now on (`--reason` required).
`snapshot <name> [--note]`	Cut an immutable snapshot at the current head.
`diff <name> <a> [<b>]`	Rows added, superseded, and tombstoned between two snapshots (default `b` is the live head).

Upload a local JSONL

veri datasets upload ./datasets/train.jsonl --name my-train

upload runs structural validation before the network call. If validation fails, nothing is uploaded:

$ veri datasets upload bad.jsonl
Validation failed: 1 issue(s), 0 rows.
  • line 1: invalid JSON (Expecting value)

Use --skip-validation to upload anyway.

The validator checks:

Each non-blank line parses as JSON.
Every row is a JSON object (not a list or scalar).
Row count is > 0.
No duplicate id field values (when an id column exists).
Required columns are present, if specified.

To skip validation:

veri datasets upload bad.jsonl --skip-validation

--deep-check is a reserved flag for tokenizer-dependent checks (length distribution). It is not yet implemented and emits a warning if passed today.

--name defaults to the filename stem. Use --quiet for ID-only output that pipes cleanly:

DS_ID=$(veri datasets upload data.jsonl --quiet)
echo $DS_ID    # ds_abc123

Connect a HuggingFace dataset

No download required; Veri references the HF repo directly.

veri datasets connect-hf openai/gsm8k --config main --split train --name gsm8k-train

GSM8K requires a config name (main or socratic) — datasets with a single config can omit --config. Rename columns at connect time with --map old=new (repeatable):

veri datasets connect-hf openai/gsm8k \
  --config main \
  --split train \
  --map question=prompt \
  --map answer=answer

Flag	Purpose
`--split`	HF split (default `train`).
`--config`	HF dataset config / subset name.
`--map old=new`	Column rename. Repeatable.
`--name`	Display name. Default derived from repo.

Validate without uploading

veri datasets check ./datasets/train.jsonl

Same validator as upload --check, but standalone. Exits 1 on issues:

$ veri datasets check ok.jsonl
OK — 1000 rows.

$ echo $?
0

Use --format json for machine-readable output:

veri datasets check ok.jsonl --format json
# {"ok": true, "row_count": 1000, "errors": []}

Preview rows

veri datasets preview ./datasets/train.jsonl --n 3

Reads the first N JSON rows from a local file. Useful before upload; for registered datasets use veri datasets get <id> followed by the dashboard.

List, get, delete

veri datasets list                              # default 20 rows
veri datasets list --limit 100 --format json
veri datasets get ds_abc123
veri datasets delete ds_abc123

delete is irreversible. The CLI exits 0 and prints the deleted ID; no confirmation prompt.

Streams

Streams are append-only datasets with immutable snapshots; training pins name@latest or name@snap-N. The full flow (create, append, supersede, tombstone, snapshot, diff, pinning a training job) lives on Dataset streams. The short version:

veri datasets create acme-feedback --format preference
veri datasets append acme-feedback ./new-pairs.jsonl
veri datasets snapshot acme-feedback --note "week 2"
veri run configs/train.toml --set dataset.id=acme-feedback@latest
veri datasets diff acme-feedback snap-1 snap-2

Every stream command accepts the dataset’s name or ds_... id. snapshot --quiet prints the pinned ds_...@snap-N id for scripting.

Exit codes

Code	Meaning
`0`	Success.
`1`	User error (validation failed, missing flag, bad URI).
`2`	API error (server returned 4xx/5xx).
`4`	Auth error (no credentials, expired key).

What’s next

Training

Submit a job against this dataset with a reward file.

The run verb

Reference a dataset ID from configs/train.toml.

Get started

Training

Deployments

CLI

Volumes

Webhooks

Commands

Upload a local JSONL

Connect a HuggingFace dataset

Validate without uploading

Preview rows

List, get, delete

Streams

Exit codes

What’s next

Training

The run verb

​Commands

​Upload a local JSONL

​Connect a HuggingFace dataset

​Validate without uploading

​Preview rows

​List, get, delete

​Streams

​Exit codes

​What’s next

Training

The run verb

Commands

Upload a local JSONL

Connect a HuggingFace dataset

Validate without uploading

Preview rows

List, get, delete

Streams

Exit codes

What’s next