Datasets and models: versions you can prove

A model is only as trustworthy as what it was trained on. "The March dataset" stops meaning anything the moment someone adds a file to the folder. On Zaryn, a **dataset version** is a frozen list of files, each named by its hash, under one root. It never changes, it stays downloadable when the bucket does, and it says what it was made from. ```diagram type: flow title: From a bucket to a model you can trace --- Upload | Put the files in a bucket: aws-cli, rclone, your training script | | 📦 ~ Freeze | zev data snapshot corpus: the exact files and their hashes become version 1 | accent | ❄ ~ Train | Your model's weights go in a bucket too | | 🧠 ~ Link | zev data snapshot tiny --trained-on you/corpus@1 | | 🔗 ~ Pull anywhere | zev data pull you/tiny@1: every piece checked, resumes after a cut | good | ⬇ ``` ## Make a dataset A dataset (or a model) is a folder of one of your [S3 buckets](/docs/en/s3-storage). Put the files there with any S3 tool, then: ```bash zev data new corpus my-bucket/train -d "labelled support tickets" zev data snapshot corpus -m "first 40k tickets" ``` ```text Version 1 of you/corpus: 40112 files, 2.1 GB (+40112 added, ~0 changed, -0 removed) root 3f9c1a0e… Anyone who can read it gets exactly these bytes: zev data pull you/corpus@1 ``` Change the folder (add files, fix labels, drop bad rows), and snapshot again. Each version records what was added, changed and removed. If nothing changed, no version is made. A model works the same way, with `--model`: ```bash zev data new tiny my-bucket/models/tiny --model ``` You can do all of this on the [Storage page](/storage#datasets) too: **Datasets and models → Make it**, then **Freeze a version**. ## What a version is ```diagram type: cards title: A version, in one line each --- 📄 | The files | Every file under the folder at that moment: its path, its BLAKE3 hash, its size | accent 🌳 | One root | A Merkle root over those lines. Same root, same files, byte for byte | 🧊 | Kept | Delete or overwrite a file in the bucket: the version still has the old one | good 🔗 | Its sources | Which datasets, models or repositories it was trained, tuned or evaluated on | ``` The root is the [RFC 6962](https://www.rfc-editor.org/rfc/rfc6962) Merkle root of `path:blake3:size` lines, sorted by path. Anyone can recompute it from the files. Write it in a paper or a model card, and a reader can prove they have exactly what you trained on. A file that only old versions still need is kept for them and counts once on your [storage plan](/storage#plans), however many versions hold it. Deleting the dataset frees it. ## Pull: checked, resumable downloads ```bash zev data pull you/corpus@1 zev data pull you/corpus@1 data/ --root 3f9c1a0e… ``` The first goes into `./corpus`. The second goes into `data/` and refuses any version whose root isn't the one you wrote down. ```diagram type: sequence title: How a pull checks every byte lanes: Your computer | Zaryn alt: zev asks for the version's file list and recomputes its root; then it asks for each file in 16 MB pieces, each sent with the BLAKE3 tree nodes that prove it, and checks each piece against the file's hash before writing it --- Your computer -> Zaryn : the version's file list Zaryn --> Your computer : paths, hashes, sizes, root note Your computer : recompute the root: it must match (and match --root) | accent Your computer -> Zaryn : a piece of a file, 16 MB at a time Zaryn --> Your computer : the piece + its BLAKE3 tree nodes (bao) note Your computer : check the piece against the file's hash, then write it | good ``` - **Nothing unchecked is written.** Each piece comes with the part of the file's BLAKE3 tree that proves it (bao). A flipped bit, on the server's disk or on the way, is refused before it reaches your folder. - **Cut off? Run it again.** A pull carries on from the last checked piece. A partial file that was tampered with since is found and fetched again. - **Already have it?** Files already in the folder with the right hash aren't downloaded again. - **Pinned.** With `--root`, nothing is downloaded unless the root matches. ## Lineage: where it came from Say what a version was made from when you freeze it: ```bash zev data snapshot tiny -m "3 epochs" --trained-on you/corpus@2 --evaluated-on you/heldout@1 zev data snapshot corpus-clean --from you/corpus@2 zev data snapshot tiny --fine-tuned-from repo:you/base-model ``` | Flag | Means | |---|---| | `--trained-on` | trained on that dataset version | | `--fine-tuned-from` | started from that model | | `--evaluated-on` | scored on that dataset version | | `--from` | made from it (cleaned, filtered, converted) | | `--uses` | needs it to run | A link is to a dataset version (`owner/name@v`) or a [repository](/docs/en/evals-and-lineage) (`repo:owner/name`). Then follow it all the way up, or see what uses a version: ```bash zev data lineage tiny ``` ```text you/tiny@1 <- trained_on you/corpus@2 (40310 files, 2.1 GB) <- derived_from you/raw-tickets@5 (52000 files, 3.4 GB) <- evaluated_on you/heldout@1 (2000 files, 90 MB) ``` `zev data show you/corpus@2` lists **used by** too. If a source is deleted later, the link stays, marked *deleted since*. ## Share it ```bash zev data public corpus on ``` A public dataset can be pulled by anyone, signed in or not, with the same checks. A private one is yours only. ## Compare versions ```bash zev data versions corpus zev data diff corpus@1 @2 ``` ```text ~ labels.csv (1.2 MB -> 1.3 MB) + tickets/40113.json (2.1 KB) - tickets/00017.json (1.9 KB) you/corpus @1 -> @2: 198 added, 3 changed, 1 removed, +410.2 KB ``` ## Limits | | | |---|---| | Files in a version | 200,000 (snapshot a narrower folder for more) | | Links per version | 20 | | A bucket with datasets | can't be deleted until they are | Related: [S3-compatible storage](/docs/en/s3-storage), [Evals and lineage](/docs/en/evals-and-lineage), [Verified storage](/docs/en/verified-storage).