Evals and lineage

Changing code is easy; knowing nothing got worse is the hard part. A new dependency adds 40 KiB to the bundle, a refactor makes the hot loop 15% slower, a driver change pushes the firmware past the flash, a shader drops the frame rate, a data loader change costs two points of accuracy. Each is usually noticed weeks later. On Zaryn every commit's numbers are kept, and a pull request that makes one of them worse can't be merged. It works for any project: web apps, games, embedded firmware, Rust, Zig, C/C++, Go, Python, and machine learning. You need two commands. ## 1. Let zev look at your project In your repository, once: ```bash zev eval init ``` zev recognizes the project and writes `zev-evals.json`: what to run, what to time and which files to measure. For a Rust project: ```json { "command": "cargo test", "time": { "build_time": "cargo build --release" }, "files": { "binary": "target/release/app" }, "base": "main" } ``` - **`command`** runs and its output is read for test results, coverage and benchmarks. It can be a list. - **`time`** times each command (the median of `"repeat"` runs, if you set it: `{"run": "...", "repeat": 5}`). - **`files`** measures sizes after the commands ran. A path can be a glob (`dist/**/*.js`), and `"gzip": true` also records the gzipped size. Commit it. Then you, your team and CI all run the same checks. What `zev eval init` sets up: | Project | What's tracked | |---|---| | Rust (`Cargo.toml`) | `cargo test` results, the release build's time, the binary's size; Criterion benchmarks | | JavaScript / TypeScript (`package.json`) | `npm test` results, the build's time, the bundle's size and its gzipped size | | Go (`go.mod`) | `go test` results, each benchmark's ns/op, B/op and allocs/op, coverage, the build's time | | Zig (`build.zig`) | `zig build test` results, the ReleaseFast build's time, the binary's size | | C / C++ (`CMakeLists.txt`) | `ctest` results, the release build's time; add your binary under `files` | | Embedded (`platformio.ini`) | RAM and Flash use from `pio run` | | Godot (`project.godot`), Unity | export time and build size | | Python (`pyproject.toml`) | `pytest` results | | Anything with a `Makefile` | `make test` results | Training or evaluating a model? Start from your own command instead: ```bash zev eval init -- python eval.py ``` ## 2. Run it before every pull request ```bash zev eval run ``` ```text $ cargo build --release build_time: 41.25 s $ cargo test ... finished in 3.10 s Found 4 metrics in 1 timing, its output, 1 file size METRIC main fb6a20769d this commit CHANGE build_time 40.80 s 41.25 s +450.00 ms (+1.1%) = held tests.passed 212 214 +2 ✓ better tests.failed 0 0 +0 = held (lower is better) binary 3.84 MiB 4.31 MiB +481.3 KiB (+12.2%) ✗ worse than ±1% ✗ A pull request into main would be blocked: binary ``` That's the whole loop. zev ran your tests and build, compared them with `main` and told you now, not after review, that this change makes the binary 12% bigger. - **Not pushed yet?** It's a check only: nothing is stored. Push, run again (or let CI), and the results are recorded for that commit. - **Pushed?** The results are recorded and show up on the repository's **Evals & lineage** tab. - **On `main` itself?** The commit is compared with the run before it. - `zev eval run` exits with 1 when a metric regressed, so it also works as a CI step. Add `--no-fail` to record without failing. - `--dry-run` checks and stores nothing, even for a pushed commit. ## What zev reads Nothing special is needed: zev reads what your tools already print or write. Run them in `command`, or report a file you already have with `zev eval report <file>...`, or name it under `"results"` in `zev-evals.json`. | Kind | Read from | |---|---| | Test results | cargo, pytest, jest, vitest, `zig build test --summary all`, ctest, go test; JUnit XML from any language | | Coverage | `go test -cover`, lcov, Cobertura, JaCoCo, istanbul (`coverage-summary.json`), llvm-cov | | Benchmarks | Criterion, Go benchmarks, Google Benchmark, hyperfine, pytest-benchmark, vitest bench | | Web | Lighthouse, size-limit, k6 | | Embedded | PlatformIO RAM and Flash, GNU `size` (flash = text + data, RAM = data + bss), ESP-IDF size | | Your own numbers | a JSON object (the last one counts), or lines of `name: value` / `name=value` with units: `frame_ms_p99: 18.2ms`, `load_time: 2.5s`, `heap: 48 MB`, `avg_fps: 58.2`, `accuracy: 91.2%` | | Machine learning | lm-evaluation-harness results (`hellaswag.acc`, stderr left out), Hugging Face Trainer files (`eval_accuracy`, `eval_loss`) | Each metric keeps its unit. Timings are stored in milliseconds and sizes in bytes, and shown the way you'd read them (`2.91 µs`, `356.0 KiB`, `58.2 fps`). zev knows which way each one should go: time, size, memory, errors, failures, loss and latency down; passing tests, coverage, fps and accuracy up. Counters such as `epoch`, `step`, `seed` and `samples_per_second` aren't quality metrics, so they're skipped unless you list them in `"metrics"`. `--only a,b` reports just those, and `--lower a,b` marks metrics where lower is better. ## Tolerance: noise isn't a regression Two runs rarely give exactly the same number. A metric's **tolerance** is the largest change for the worse that still counts as "held", either in the metric's own units or as a percentage of `main`'s value: ```json "metrics": { "build_time": { "tolerance": "15%" }, "binary": { "tolerance": "2%" }, "accuracy": { "tolerance": 0.005 }, "tests.failed": { "tolerance": 0 } } ``` Defaults that fit most projects: timings 10%, sizes 1%, coverage 0.5 points, everything else 0 (any drop counts). `"tolerance"` at the top level applies to every metric; `--tolerance 5%` overrides it for one run. Not sure what to use? Click a metric on the **Evals & lineage** tab: it shows how much the metric typically moves between runs and suggests a tolerance. Timing in CI is noisier than on your machine. Give slow commands `"repeat": 3` or 5 (the median counts), and a wider tolerance if the runners are shared. ## Examples **A web app that must stay small.** `zev eval init` in a Vite, Next or plain npm project tracks the test results, the build time and the size of `dist/**/*.js` and `.css`, gzipped too. A pull request that adds a 50 KiB library to a 200 KiB bundle is blocked at 1%; add Lighthouse's JSON under `"results"` to track performance scores too. **Firmware that must fit.** With PlatformIO, `pio run` already prints RAM and Flash; zev tracks both. With a plain toolchain, put `arm-none-eabi-size build/fw.elf` in `command`. A change that pushes the firmware over a few hundred bytes shows up before it reaches the device. **A game that must stay smooth.** Run a benchmark scene headless and print a few lines (`avg_fps: 58.2`, `frame_ms_p99: 21.4ms`, `load_time: 2.5s`); add Godot's or Unity's export under `"time"` and the build folder under `"files"`. **A library that must stay fast.** Criterion, Go benchmarks, Google Benchmark or hyperfine: each benchmark becomes a metric with its time unit. Add `"repeat"` for timings of whole commands. **A model that must stay accurate.** `zev eval init -- python eval.py`. Print the scores as JSON (`{"accuracy": 0.912, "f1": 0.88}`) or as `name: value` lines, or point `"results"` at lm-evaluation-harness or Hugging Face output. ## On the website The repository's **Evals & lineage** tab shows: - **Every metric's history** on `main`, in its units. Click one for a chart where each point opens its commit, plus the best value, the biggest drop and the commit that caused it, and a suggested tolerance. Runs from other branches are one click away. - **Compare**: any two commits or branches, metric by metric, with the verdict a pull request between them would get. - **Pull requests**: each pull request shows its eval check. A regression blocks the merge until it's fixed, the tolerance changes, or whoever merges overrides the check. ## A badge for your README ```bash zev eval badge binary ``` prints the Markdown to paste into your README; the badge links to the **Evals & lineage** tab. It shows the metric's latest value on `main` in its unit (`4.31 MiB`, `92.4%`, `58.2 fps`): green while it holds, red when the last change regressed it. Private repositories' badges show nothing to people who can't read the repository. ## Compare any two versions ```bash zev eval # this commit next to main zev eval show v2-experiment # a branch or commit next to main zev eval compare v1 v2 # any two zev eval compare main --json # for scripts ``` ## In CI Any CI that can run a shell works. Keep a token from **Settings** as a secret, then: ```bash curl -fsSL https://zaryn.xyz/install.sh | sh export ZEV_SERVER=zaryn.xyz ZEV_TOKEN=$ZARYN_TOKEN zev clone you/app@$BRANCH && cd app zev eval run ``` Every push gets its results recorded, and the job fails when a pull request would be blocked anyway, so you see it in the CI run. Already producing JUnit XML or lcov in an earlier step? `zev eval report junit.xml lcov.info` reports them. Without zev, post the numbers yourself: ```bash curl -X POST https://zaryn.xyz/api/v1/repos/you/app/evals \ -H "Authorization: Bearer $ZEV_TOKEN" -H "Content-Type: application/json" \ -d '{"commit_hash":"main","metrics":[ {"name":"build_time","value":41250,"unit":"ms","higher_is_better":false,"tolerance":"10%"}, {"name":"accuracy","value":0.912,"tolerance":0.005}]}' ``` `unit` is `ms`, `bytes`, `%` or `fps`, or left out for a plain number; `tolerance` is in the metric's units or a percentage like `"10%"`. The answer includes the check against `main`. `POST /v1/repos/<owner>/<repo>/evals/compare` with the same `metrics` checks without storing anything (read access is enough). `GET .../evals/compare?base=main&head=<commit or branch>` compares two versions. ## Lineage: where it came from Record what this repository was built from: ```bash zev lineage record trained_on you/imagenet-mini zev lineage record fine_tuned_from meta/llama-3-8b@<commit> ``` Relations: `trained_on`, `fine_tuned_from`, `evaluated_on`, `derived_from` (a dataset cleaned or labelled from another, or a fork of a project) and `uses` (an app that depends on a model, a library or an asset pack). The source's current commit is pinned, or the one after `@`. Lineage pays off twice: - **When the data changes.** If `you/imagenet-mini` gets new commits after your model was trained on it, `zev lineage upstream`, `zev eval run` and the **Evals & lineage** tab all say so: the results may be stale, so retrain or re-run the eval, then record the link again. ```text ⚠ Trained on you/imagenet-mini has changed since (recorded at c4f51a25cd, its main is now a023c62c92); these results may be stale. ``` - **When something upstream is compromised.** An [advisory](/docs/en/advisories) filed against a poisoned dataset or a backdoored base model reaches every model recorded as built from it, and their owners are told. For a model with weights on Zaryn, the tab can also suggest the base model it was fine-tuned from, by comparing the weights. Record it with one click. ## Good to know - Results are free on every plan. The multi-hop lineage graph (datasets → base models → fine-tunes → agents) is part of Premium; direct links are free. - Metrics are grouped in **suites** (`--suite`, default `default`), so one repository can keep several benchmarks or targets apart (say `esp32` and `rp2040`). - Recording results needs write access to the repository. Reading and comparing needs read access. - `zev metrics`, `zev check` and `zev history` keep numbers in your local repository only; `zev eval` is the one that shares them on Zaryn and gates pull requests.