Skip to main content
A dataset stream is an append-only dataset that grows as your data arrives. Training never reads the live stream: a job pins an immutable snapshot, so the exact rows a model trained on are reproducible forever. Rows are never edited or deleted in place: a correction is a new row that supersedes an old one, and a removal is a tombstone. Both take effect in snapshots cut from that point on; earlier snapshots keep the original rows, so history stays honest. One-shot datasets (upload or connect) keep working exactly as before; streams are for data that keeps growing, like production feedback.

​
Create a stream

--format locks the row shape and is required: prompt, preference, completion, or chat. Every append is validated against it, so a malformed row is rejected at append time rather than at training time. Every stream command accepts the dataset’s name or its ds_... id.

​
Append rows

Append a JSONL file, or pipe rows on stdin with -:
Each row gets a permanent row id. Blank lines are skipped; a malformed line fails the whole append with its line number, and nothing is written.

​
Correct a row (supersede)

A supersede is a new row that replaces an earlier one in every snapshot cut from now on. Snapshots cut before the correction keep the original, so an old training run still resolves to exactly what it trained on.
--supersedes takes exactly one row.

​
Remove a row (tombstone)

The row disappears from every snapshot cut from now on and stays in every snapshot cut before. Nothing is deleted; --reason is required and shows up in diffs.

​
Cut a snapshot

A snapshot is a pointer at the current head, so cutting one is instant at any size. Reading a snapshot resolves supersessions and skips tombstones as of that snapshot. --quiet prints just the pinned id (ds_9f2c81ab@snap-2) for scripting. To cut a snapshot automatically every N new rows, set a rule at create time (--auto-snapshot 100) or later via the SDK (client.datasets.set_auto_snapshot("acme-feedback", 100)). Auto-cut snapshots show auto in the timeline.

​
Train on a snapshot

Pin a training job to a stream with <name>@latest or <name>@snap-N as the dataset id:
What @latest resolves to at submit time:
  • If the newest snapshot already sits at the stream head, the job pins that snapshot.
  • Otherwise a new snapshot is cut at the head (noted “pinned at job submit”) and the job pins it.
  • A plain stream name with no @ suffix implies @latest.
  • An empty stream is rejected: append rows before pinning @latest.
Either way the job stores the resolved ds_...@snap-N id, so rows appended after submit never change what the job trains on. The pinned id is on the job record as dataset_snapshot_id (SDK: client.training_jobs.get(job_id).dataset_snapshot_id, or GET /v1/training_jobs/{job_id}), and you can record it on the resulting model version with veri models register ... --dataset-snapshot <id> (see Model versions).

​
Diff two snapshots

See exactly what changed between two snapshots (or between a snapshot and the live head, the default for the second argument):
This answers “what data made v6 different from v5” when each version pins a snapshot.

​
Inspect a stream

Shows the locked format, the current head row, how many rows arrived since the last snapshot, and the snapshot timeline (name, head row, note, auto flag, created time). To read rows as of a snapshot programmatically, use the SDK: client.datasets.rows("acme-feedback", snapshot="snap-2") resolves supersessions and tombstones exactly as training does.

​
Where to go next

Model versions

Register the trained job as a version and record the snapshot it trained on.

Datasets

One-shot uploads, Hugging Face connects, and JSONL formats.

Dataset CLI

Every veri datasets command in one place.