# Rollouts: make a config change safely > Make a config change on the devices themselves, checked on the live network and taken back by itself when the network doesn't hold. The exact commands both ways, the twin's verdict, approval with four eyes, waves with a canary, MikroTik safe mode and a dead man's switch on the device, every command and answer kept. Source: https://zaryn.xyz/docs/en/network-rollouts Most outages come from changes, and most bad changes are found too late: after someone typed them into a router at 2 a.m. and the phones started ringing. A **rollout** is a config change Zaryn makes for you on the devices of a [site](/docs/en/network-sites), the careful way: 1. **Planned** as the exact commands each device will be given, and the commands that undo them, from the config it runs now. 2. **Checked by the [digital twin](/docs/en/network-twin)**: which traffic stops, starts or takes another way, and the routes each device should have afterwards. 3. **Approved** by a site admin. With four eyes, by someone other than the person who planned it. 4. **Made by the site's collector**, wave after wave, and **checked on the live network**. It's kept when everything holds, and taken back by itself when it doesn't. Everything is kept: who planned and approved it, every command typed, every answer the devices gave, every check. ## Before the first rollout: let the collector make changes Out of the box a collector only reads devices. Only someone on the collector's own machine can let it make changes: ```bash sudo zev site changes on ``` Nothing Zaryn says can turn this on. `zev site changes off` turns it off again, and `zev site changes status` shows the last entries of its log. Every step of every change is also written to `changes.log` next to the collector's credentials. `--max-devices 5` makes the collector refuse waves of more than five devices, whatever was approved. The login the collector uses for a device needs the right to change its config: privilege 15 or an enable secret on Cisco, a write group on MikroTik, `frrvty` or sudo for `vtysh` on FRRouting. ## Plan a rollout Open the site, then **Try a change**. Edit the configs of one or more devices and click **See what it would do**. Below the twin's answer, **Roll it out**: - **What the change is,** and why if you like. - **Let the network settle** (default 60 seconds), **then keep checking** for up to 180 seconds before giving up. Make the settle time longer than your routing protocols' timers: OSPF notices a lost neighbor only after its dead interval (40 seconds by default), BGP after its hold time (90 or 180 seconds). - **This change removes links or neighbors on purpose:** tick it when losing a neighbor is the point of the change, so the check doesn't take it back for that. - **Waves and window (Premium):** a canary, devices per wave, a pause between waves, a maintenance window. You get the rollout's page. Each device shows the commands it will be given and, under **Undo**, the commands that take them back: ```text r1 FRRouting +2 −0 interface eth3 ip ospf cost 100 exit ``` Lines that can't be typed safely are listed as problems, and the rollout can't be approved until the config is changed. Examples: a secret the collector doesn't know, a RouterOS setting that can only be forgotten by hand, a numbered access list changed in the middle. From a pull request: when a pull request on the site's [config repository](/docs/en/network-twin#pull-requests-on-a-config-repository) is merged, its network twin card offers **Roll it out to the devices of …**. The rollout makes every device whose file on the branch differs from what it runs match the file. ## Runbooks: one change, every device Some changes are the same on every device: a new NTP server, a syslog server, a rotated SNMP community, a user who left. A **runbook** describes such a change once, per vendor, and runs it as a rollout on every device that needs it. On the **Rollouts** tab, under **Runbooks**, start **from the library** (NTP server, syslog server, DNS servers, rotate the SNMP community, remove a local user, SSH only on the VTY lines) or write a **new runbook**. For each vendor, one line each: ```text + ntp server {{ntp}} - ntp server * [line vty 0 4] + transport input ssh ? show ntp associations => {{ntp}} ``` - `+` a line the config must have. It's added only where it's missing. - `-` a line it must not have. A trailing `*` matches any end: `ntp server *` removes every NTP server. - `[section]` in front puts the line inside a section (`interface Vlan10`, `line vty 0 4`, Huawei's `aaa`). A section that doesn't exist yet is added. - `?` a read-only command (`show`, `display`, RouterOS `print`) whose output must show the text after `=>` once the change is made. It's checked with the rest, and the change is taken back if it doesn't show. - `{{name}}` is a parameter, filled in when the runbook is run. A runbook can be limited to devices with some tags (`core`, `branch`). **Run** asks for the parameters, then **Preview** shows each device: what it would be given, or why it's left alone (already as the runbook says, no lines for its vendor, not read yet). **Plan the rollout** makes one rollout of all the devices that need the change. It's approved, made in waves, checked and taken back like any other. Parameter values are one line each, so a value can't slip extra commands in. Firmware upgrades, which restart devices, aren't something runbooks do: collectors refuse reloads by design. ## Approve A site admin approves on the rollout's page. Three things can hold approval back: - **Four eyes.** In **Collector and settings**, under **Rollouts and access**, a site can ask for a second person. The author can't approve their own rollout then. - **The twin says it breaks traffic.** Approving needs a tick: *I mean it*. It's kept with your name. - **A device changed since it was planned.** Someone changed its config in between, so the plan no longer fits. Plan it again. ## What the collector does The collector picks up approved waves at its next poll (every 30 seconds). For each wave: ```diagram type: sequence title: One wave, from approval to confirmation lanes: Engineer | Zaryn | Collector | Device alt: An engineer approves a planned wave; the collector reads the device, makes the change with a built-in way back, checks the network, then keeps or undoes the change --- == Plan == Engineer -> Zaryn : plan and approve Zaryn -> Zaryn : the twin predicts the traffic == On the site, every 30 seconds == Collector -> Zaryn : any approved wave? Zaryn --> Collector : yes: the wave Collector -> Device : read the config again note Collector..Device : not what the plan started from? the wave is refused, nothing is typed | warn Collector -> Device : make the change, with safe mode or a dead man's switch Device --> Collector : routes, neighbors, probes note Collector..Device : everything holds twice? keep it. Otherwise take it back, last device first. | good Collector -> Zaryn : the result, before and after Zaryn --> Engineer : done, or undone and why ``` 1. **Reads every device again.** Its config must still be exactly the one the change was planned from: if someone changed it in the meantime, the wave is refused and nothing is typed. Secrets the plan shows as `` tags are put back from the device's own config. They never leave the site. 2. **Measures what must keep working:** each device's neighbors (CDP, LLDP, OSPF and BGP), every other device of the site answering on SSH, and the site's [probes](/docs/en/network-paths) that pass. 3. **Makes the change,** device after device, the device's own safest way: | Device | How the change can be taken back | |---|---| | MikroTik RouterOS | **Safe mode**: the session holds the change while the checks run. If the collector loses the router, or dies, RouterOS undoes everything by itself. | | FRRouting | **A dead man's switch on the device**: the undo commands wait in a file and run by themselves unless the change is confirmed in time. | | VyOS | The same: the config before is saved and loaded back unless confirmed. | | Cisco IOS / IOS-XE, Huawei VRP | The collector types the undo commands. The running config is only saved once the change is confirmed, so a reload brings the old one back too. | 4. **Checks the network** after the settle time, again and again until everything holds or the time runs out, and once more 15 seconds later before keeping it: - every device still **answers**; - its **config** changed; - the **routes** the twin predicted are there, and the ones it said would go are gone; - no **neighbor** was lost; - the **other devices** that answered still do; - the **probes** that passed still pass. Probes the twin says the change breaks on purpose aren't checked. 5. **Keeps it** (safe mode released, the switch disarmed, the config saved with `write memory` or `save`), or **takes it back** on every device of the wave, last first. Then it reads them once more to say whether each config is exactly as before. Commands that aren't config changes are refused by the collector whatever Zaryn sends: reload, erase, file operations, scripts, `do …`, RouterOS `/system` and `/file` commands, VyOS task schedulers. ## Watch it happen While a wave runs, its page follows it live: **Read the devices → Make the change → Check the network → Keep and save**, or **Taking it back**. Each device shows its checks as they come in (✓ Answers, ✓ Routes, ✗ Neighbors…) with the reason for any that fails. The **Transcript** is the whole session, every command typed and every answer, timed: ```text 0.0s wave of "Costly link r1-r3": 1 device(s) 1.9s before: 2 neighbor(s), 3/3 device(s) answering, 2/2 probe(s) passing 2.0s r1: changing 3 line(s) through vtysh; undone by the device itself in 540s unless confirmed 2.0s r1> interface eth3 2.0s r1> ip ospf cost 100 2.0s r1> exit 62.4s checks pass; checking again in 15s before keeping it 77.6s checks pass 78.1s r1: confirmed and saved ``` The site's **Changes** list every rollout planned, approved, done, taken back or failed. ## Undo, cancel - **Undo** plans a new rollout back to the configs the devices ran before. It's reviewed and approved like any other. - **Cancel** stops a rollout before it starts. During one, the wave being made finishes (checked, kept or taken back) and no other starts. - When a device's own guard took a change back, or a wave was taken back, the device's config is back to the approved one, so it isn't shown as drift. ## Waves, canary and maintenance windows (Premium) - **Canary:** the first device alone. The rest only follow when it holds. - **Devices per wave:** the rest, N at a time. Each wave is checked before the next starts, and a wave that's taken back stops the rollout. Waves made before it stay; **Undo** takes them back. - **Pause between waves:** time to watch dashboards and users before going on. - **Not before** and **Window ends:** a maintenance window. Nothing starts before it. If it ends before every wave is made, the rest is stopped. The twin works out the routes expected after each wave for the network as it is at that point, so a canary is checked against what it alone should do. ## If the collector stops in the middle - On MikroTik, the dropped session undoes the change. On FRRouting and VyOS, the device's own switch does. - On Cisco and Huawei, the change stays running but unsaved, so a reload also brings the old config back. Look at the device. - Zaryn marks a wave as lost once the collector has been silent for the settle time, the check time and 15 minutes. The rollout fails, saying so. ## Plans - **Free:** up to 10 devices in one rollout, all in one wave, with every check and the automatic rollback. - **Premium:** more devices, canary, waves, pauses and maintenance windows. ## Good to know - Rollouts change the devices zev can read and change: Cisco IOS and IOS-XE, Huawei VRP, MikroTik RouterOS, FRRouting (through `vtysh`), VyOS. Plain Linux devices aren't changed. - Up to 3000 commands per device in one rollout. Bigger changes are better split anyway. - A secret you type in clear in the new config (a new password) is sent to the collector as typed. Secrets already on the device stay tags and never leave it. - Cisco, Huawei and MikroTik support has been tested against emulated devices, VyOS only in unit tests: try a first rollout on a lab device or in a maintenance window.