Content-defined chunking

Large files change unevenly: a small, local edit near the start of a file shouldn't require re-transferring the entire file. zev addresses this with content-defined chunking, built on a Gear-hash rolling window — the same family of algorithm used by tools like `restic` and `borg`, implemented independently in Zig using only the standard library. ## The core property A content-defined chunker splits a file into variable-length pieces based on the file's own content, not fixed byte offsets. When a small edit is made, only the chunks touching that edit change — every chunk boundary elsewhere in the file stays exactly where it was, because the boundary is a function of nearby bytes, not a running byte count. This was proven independently, before any integration work: chunking a real 1.7 MB file and inserting a 93-byte edit near its start left 96.5% of the file's bytes, by chunk, byte-for-byte unchanged. A whole-file storage model would have required re-storing 100% of the file for that same one-line edit. ## How chunks are identified and stored Each chunk is hashed with BLAKE3 (matching the server's own object identity scheme — see [Content-addressed storage](/docs/en/content-addressing)) and stored as an ordinary object under that hash. A manifest — the ordered list of chunk hashes that reconstruct the original file — is stored alongside it, keyed by the whole file's own hash. This design is deliberately additive: a chunked file is still also stored as a complete, ordinary whole-file object. Every existing read path that expects one object, one hash, one complete file continues to work unmodified. Chunking is purely an optimization layered underneath, invisible to anything that doesn't specifically ask for it. ## What actually happens on push 1. The client splits a large file (above a size threshold) into chunks locally. 2. It asks the server which of those chunk hashes it doesn't already have (`POST .../objects/have`). 3. It uploads only the missing chunks. 4. It asks the server to reconstruct the whole file from the chunks (`POST .../objects/reconstruct`) — the server re-derives the file's hash from the chunks itself and refuses to store anything if it doesn't match what was claimed. Measured on a real 92-chunk, ~1.7 MB file after a genuine one-line edit: **1 chunk newly uploaded, 91 already present.** ## What happens on pull The same idea, in reverse, and in batches. The client asks for everything it's missing in a few requests (`POST .../objects/batch`, up to 256 objects each, four at a time). Small objects come back whole. A file stored in chunks comes back as its list of chunks: the client looks each one up in its local chunk cache (filled by earlier pushes and pulls), asks for only the missing ones (also in batches, and a chunk shared by several files only once), puts the file together, and checks that the result has the right hash before it's written. Every object and chunk is checked against its name, which is the hash of its bytes. Measured on the same real file, pulling the same one-line edit into a repository that already had the previous version: **1 chunk fetched over the network, 91 served from the local cache.** zev prints it after the pull: ```text Fetched: 3 (3 in 2 request(s)), already had: 4 Large files: 1 chunk(s) downloaded, 91 reused from earlier pulls ``` ## A genuine operational tradeoff worth knowing Chunking trades bandwidth for request count. A single push of a 92-chunk file can be 90+ individual HTTP requests (pulls avoid this by asking for chunks in batches). This genuinely interacts with request-rate limiting designed for ordinary API traffic — a rate limit sized for typical usage patterns can be exhausted by one large chunked push. zev's server gives chunk-related endpoints (`/have`, `/reconstruct`, individual chunk uploads and downloads) their own, separate, much higher-limit rate bucket specifically because of this — so bulk chunk synchronization never competes with, or starves, the budget protecting the rest of the API.