Virtualization: hypervisors on a site

## In one minute Zaryn reads your hypervisors and your switches with the same collector and joins the two into one picture: every virtual machine, the physical switch port its traffic leaves by, and what that means when something moves, fills up or fails. This page starts from the basics and goes all the way down to how each check is decided, so read as far as you need. - **Add a hypervisor as a device**, with a login that can only look: a Proxmox VE cluster, a VMware vCenter, a KVM or Hyper-V host, an oVirt / OLVM engine, a Nutanix Prism Central or an OpenStack cloud. Several platforms can sit side by side on one site. - **See every VM where it really is**: its host, its addresses, its VLANs, and its path out: `VM → bridge, VLAN 20 → bond0 → eno1 → sw1 Gi1/0/5`. - **Get told before it hurts**: a VM whose VLAN its switch port doesn't carry, the hosts a VM can't move to, a host hanging off one switch, the VMs that go dark if a switch reboots, a datastore full in 9 days. - **Nothing is changed**. The collector only reads, the passwords stay on the collector, and every call it makes is listed below. ```diagram type: graph title: What Zaryn reads, and where it goes alt: Hypervisors and switches are read by the collector with read-only logins; the collector sends what it read to Zaryn over HTTPS legend: path=Outgoing HTTPS; ghost=Read-only caption: Point at anything for what it is. One collector reads every platform and every switch, and joins them into one picture. --- box Your site | 2 3 66 56 good node pve | Proxmox VE | 12 12 | host | | An API token with the PVEAuditor role: it can only look node vc | VMware vCenter | 12 28 | host | | A read-only vCenter account, logged in and out on every read node kvm | KVM, Hyper-V, … | 12 44 | host | | One fixed read-only script over SSH or PowerShell node sw | switches | 34 28 | switch | | Their configs and LLDP or CDP: which port each host is cabled to, and its VLANs node col | collector | 54 28 | collector | accent | Holds the passwords, joins VMs to switch ports, sends only what it read node zaryn | Zaryn | 88 18 | cloud | accent | Checks VLAN reach, live migrations, redundancy, storage node you | you | 88 44 | user | | The Virtual machines tab: every VM, where it is, what needs attention link pve col | | ghost link vc col | | ghost link kvm col | | ghost link sw col | SSH, read-only | ghost link col zaryn | HTTPS, outgoing | path link zaryn you | VMs, checks, forecasts ``` ## The basics New to virtualization, or to the network side of it? This is the vocabulary the rest of the page uses. ### Hypervisors, VMs and containers A **hypervisor** (Proxmox VE, VMware ESXi, Hyper-V, KVM, oVirt, Nutanix AHV; OpenStack runs KVM on many hosts) runs many isolated machines on one physical server: each **virtual machine** (VM) gets its own virtual CPUs, memory, disks and network cards, carved out of the host's real ones. Containers (LXC) do the same more lightly, sharing the host's kernel. A **cluster** of hosts lets a VM move from one host to another while it runs (**live migration**: vMotion, Live Migration), and come back up on another host when its host fails (**high availability**, HA). ### How a VM reaches the network A VM's network card isn't plugged into anything physical. It's plugged into a **virtual switch** inside its host, and the host's own cables take its traffic to the real network: ```diagram type: graph title: A VM's path to the real network alt: A VM's card sits on a bridge on the host with a VLAN tag; the bridge uses a bond of two uplinks, each cabled to a switch port legend: path=Where the VM's frames go; down=Unplugged start: Both uplinks up caption: Point at each piece for what it does. Frames tagged with VLAN 20 leave through the bond, over both cables. --- box Host pve1 | 2 4 55 52 accent box Physical network | 70 4 28 52 good node web1 | web1 | 10 30 | vm | accent | The VM: its card eth0 is plugged into a virtual switch, not into anything physical node br | vmbr0 | 29 30 | switch | | The bridge: a switch inside the host. It tags web1's frames with VLAN 20 node eno1 | eno1 | 47 16 | dot | | A physical port of the host, one of the two in bond0 node eno2 | eno2 | 47 44 | dot | | The other port of bond0 (LACP): either can carry everything node sw1 | sw1 Gi1/0/5 | 84 16 | switch | | A trunk port that allows VLANs 1 and 20 node sw2 | sw2 Gi1/0/5 | 84 44 | switch | | A trunk port that allows every VLAN link web1 br | VLAN 20 | path link br eno1 | bond0 | path link br eno2 | bond0 | path link eno1 sw1 | trunk 1,20 | path link eno2 sw2 | trunk all | path step A cable is pulled | eno1 loses its link. The bond moves everything to eno2 and web1 doesn't notice: that's what redundant uplinks are for. link br eno1 | down link eno1 sw1 | dim node eno1 | bad ``` 1. The VM's card is on a **bridge** or virtual switch (`vmbr0` on Proxmox, a port group on vSphere, a logical network on oVirt, a subnet on Nutanix, a Neutron network on OpenStack). 2. The VM's traffic is usually **tagged** with a **VLAN** number, which keeps networks apart on the same cables. 3. The bridge sits on the host's physical ports (**uplinks**), often two of them in a **bond** (LACP, NIC team) so that one can fail. 4. Each uplink is cabled to a **switch port**. A **trunk** port carries the VLANs in its *allowed* list, tagged; an **access** port carries one VLAN, untagged. If the port doesn't allow the VM's VLAN, the VM's frames die there. ### Words you'll see | Word | What it means here | |---|---| | Host | A physical server running a hypervisor | | Guest | A VM or a container on a host | | Bridge, vSwitch, port group | The virtual switch a VM's card plugs into | | VLAN | A number that separates networks on shared cables (1–4094) | | Trunk / access port | A switch port carrying many tagged VLANs / one untagged VLAN | | Allowed VLANs | The VLANs a trunk port lets through (`switchport trunk allowed vlan`) | | Uplink | One of a host's physical ports towards the switches | | Bond, LACP, NIC team | Several uplinks working as one, so one can fail | | LLDP, CDP | How a switch learns what is cabled to each of its ports | | Overcommit | Giving guests more vCPUs or memory than the host has | | Datastore, pool, container | Where VM disks live | ## Why it's hard today Virtualization made servers cheap to create and easy to move. The network didn't get the memo, and the gap between the two causes a large share of the incidents that are hardest to explain: - **Two teams, two views.** The network team sees MAC addresses and switch ports; the virtualization team sees VM names and hosts. "Which switch port does `erp-db` leave by?" takes a meeting, and the answer is stale by the next vMotion. - **VMs move; switch configs don't.** Live migration and HA move a VM to a host whose switch port may not carry its VLAN. The VM is up, the hypervisor says green, and nothing can reach it. It's one of the most common causes of "it worked yesterday". - **Redundancy that isn't.** A bond of two cables is only redundant if they land on two switches. Both on one switch, or one cable unplugged months ago, and a single switch reboot takes the whole host's VMs down. - **More platforms at once.** Licensing changes since 2024 (Broadcom's VMware bundles in particular) pushed many organisations to add or move to Proxmox, Nutanix, Hyper-V, OpenStack or oVirt. Few have one platform left; most have two or three, each with its own console and none showing the network. - **Capacity surprises.** Memory handed out beyond what the hosts have, and storage that fills a little every day until VMs pause. - **Migrations without a map.** Moving VMs off a platform without knowing who talks to whom splits applications across the old and new platforms. - **Tools that want too much.** Many visibility tools need write access, agents in every VM, or a SaaS connection into the management network. Security teams rightly say no. ## What Zaryn gives you | The problem | What Zaryn does | Where you see it | |---|---|---| | Two teams, two views | Joins each VM to its host, its bridge and VLAN, the host's physical ports and the switch port on the other end | Virtual machines tab, each VM's detail | | VLAN missing after a move | Reads the switches' configs and checks that every port a VM's traffic crosses carries its VLAN | ✓ / ✗ on each uplink, **What needs attention** | | Unsafe live migrations | For every VM, the hosts of its cluster it can move to with its network and its disks intact, and why not for the others | **Where it can move**, in the VM's detail | | Redundancy that isn't | Rates each host's uplinks: redundant, one switch, one port, or unknown | The badge on each host | | "What breaks if this switch reboots?" | For each switch, the hosts cut off and the running VMs that go dark, and the hosts that only lose redundancy | **If a switch goes down** | | Storage filling up | Fill levels for every platform, and a forecast from an hourly sample | **Storage**: "full in about 9 days" | | Thin disks past their storage | The sizes of the VM disks on each storage added up, next to what it holds | **Storage**: "VM disks: 3 TB (300% of its size)" | | Overcommit | vCPUs per physical CPU and memory given out, per host and in total | The capacity row, each host | | Several platforms | One inventory for seven platforms, side by side | The same tab, whatever the platform | | Migrations | Waves planned from who talks to whom | **Plan a migration** | | Security reviews | Read-only logins, no agent in VMs, passwords kept on your collector | [What the collector sends](#what-the-collector-sends) | ## Add a hypervisor Every platform is added the same way: make a login that can only look, give it to the [collector](/docs/en/network-sites) with `zev site cred add`, try it with `zev site test`, then add the hypervisor to the site as a device. | Platform | Add as one device | Login it needs | Port | How it's read | |---|---|---|---|---| | Proxmox VE | Any node (the whole cluster) | API token, PVEAuditor role | 8006 | REST, GET only | | VMware vSphere | The vCenter | Read-only role | 443 | REST (GET only) and SOAP (reads only), a session each | | KVM (libvirt) | Each host | SSH user | 22 | `virsh -r` (read-only connection) | | Hyper-V | One node (the whole failover cluster) | SSH user in Hyper-V Administrators | 22 | A fixed script of `Get-` cmdlets | | oVirt / OLVM | The engine | ReadOnlyAdmin role | 443 | REST, GET only | | Nutanix AHV | Prism Central | Prism Viewer role | 9440 | v3 list calls | | OpenStack | Keystone (the whole region) | reader role | 5000 | A token, GET only, the token revoked | ## Add a Proxmox VE cluster Zaryn only reads, through Proxmox's API, with a token that can only look (the built-in `PVEAuditor` role). On one Proxmox node: ```bash pveum user add zev@pve pveum acl modify / --users zev@pve --roles PVEAuditor pveum user token add zev@pve zaryn --privsep 0 ``` The last command prints the token's secret once. On the [collector](/docs/en/network-sites): ```bash zev site cred add pve --user 'zev@pve!zaryn' --password-stdin zev site test 10.0.0.11 --cred pve --vendor proxmox ``` Then add the cluster to the site as a device: any node's address, vendor **Proxmox VE**, credential `pve`. The port is 8006 unless you say otherwise. One device is the whole cluster: the API answers for every node. The collector sends only GET requests. The secret stays on the collector, like device passwords; Zaryn only learns the credential's name. ### The TLS certificate Proxmox nodes usually have a self-signed certificate. The first time the collector reaches one, it keeps the certificate's public key (`tls_pins.json`, like SSH's `known_hosts`) and from then on refuses any other: someone in the middle can't read the token. If you replaced the certificate on purpose: ```bash zev site forget-key 10.0.0.11 --port 8006 ``` ## Add a VMware vCenter Zaryn reads vSphere through vCenter's REST API (vCenter 7.0 Update 2 or newer), and its SOAP API for what REST doesn't say, with a user that has the built-in **Read-only** role. In the vSphere Client: Administration → Single Sign On → Users, add `zev@vsphere.local`; then on the vCenter object, Permissions → Add, that user, role **Read-only**, with **Propagate to children**. On the collector: ```bash zev site cred add vc --user zev@vsphere.local --password-stdin zev site test vcenter.lab.local --cred vc --vendor vsphere ``` Add the vCenter as a device: its address, vendor **VMware vSphere**, credential `vc` (HTTPS on 443). One device is everything that vCenter manages: its clusters, ESXi hosts, VMs and datastores. ESXi hosts are read through their vCenter, not one by one. The collector logs in (one request opens a session), then only reads, then logs out. The TLS key is pinned the same way as for Proxmox. vCenter's REST API leaves out the network under the VMs, so the collector also reads the SOAP API (`/sdk`, the one PowerCLI and govc use) in the same visit, with the same user and only reads: - each ESXi host's **vmnics**, its standard **vSwitches** and the vmnics each one uses (active and standby); - each **port group's VLAN**, on standard vSwitches and on **distributed switches** (a trunk port group shows its ranges); - which vmnics each host gives to each distributed switch, and its **VMkernel** addresses (`vmk0`), which become the host's address; - which hosts **mount each datastore**, and whether they can reach it. So a VM's network is its port group with its VLAN, and its way out is port group → vSwitch or distributed switch → vmnic → the switch port that LLDP or CDP reports for that vmnic (turn on LLDP or CDP on the vSwitch or distributed switch so the switches see the hosts). Addresses come from VMware Tools, or the switches' ARP tables. If the SOAP API can't be read (a proxy that only passes `/api`, say), the device says so and the rest still works, as before. ```diagram type: sequence title: One visit to a vCenter lanes: Collector | vCenter alt: The collector opens a REST session and reads with GETs, then opens a SOAP session, reads hosts, switches, port groups and datastores through a container view in pages, and logs out of both --- == REST == Collector -> vCenter : POST /api/session, then GETs: clusters, hosts, VMs, datastores Collector -> vCenter : DELETE /api/session == SOAP, reads only == Collector -> vCenter : Login, CreateContainerView Collector -> vCenter : RetrievePropertiesEx: hosts' NICs, port groups, distributed switches, mounts vCenter --> Collector : a page at a time (ContinueRetrievePropertiesEx) Collector -> vCenter : DestroyView, Logout note Collector..vCenter : nothing that creates, changes or powers anything | good ``` ## Add a KVM host (libvirt) A plain Linux server running KVM VMs through libvirt (`virsh`, virt-manager, Cockpit) is read over SSH, like a router. The collector runs one fixed script: `virsh` with **`-r`**, libvirt's read-only connection, for the VMs, their configs, state, disks, networks and storage pools, plus `ip -j addr show` for the host's bridges and bonds. Nothing else. No root is needed: libvirt's read-only socket is open to every local user unless the distribution closed it. On the host, make a user and check it can list the VMs: ```bash sudo useradd -m zev && sudo passwd zev sudo -u zev virsh -r -c qemu:///system list --all ``` If that says permission denied, the distribution restricted the read-only socket (`unix_sock_ro_perms` in `/etc/libvirt/libvirtd.conf`, or `/run/libvirt/libvirt-sock-ro`'s group): open it to the user. Adding the user to the `libvirt` group works too, though that group may also change VMs; the collector still only runs `virsh -r`. On the collector: ```bash zev site cred add kvm --user zev --password-stdin # or --key ~/.ssh/zev_kvm zev site test 10.0.0.21 --cred kvm --vendor kvm ``` Add the host as a device: its address, vendor **KVM / libvirt**, credential `kvm` (SSH, port 22). libvirt has no cluster view, so each host is its own device. The host key is kept on first contact, as for any SSH device. A VM's network card is on a Linux bridge (`br0`, with its VLAN tag), a libvirt network (`default`, its bridge `virbr0`; a portgroup's VLAN is used), an Open vSwitch bridge, or directly on a host port (macvtap). Disks are matched to their storage pool by path. A VM is known by its UUID, so renaming it keeps its history. ## Add a Hyper-V host A Windows Server running Hyper-V is read over Windows' own **OpenSSH server** (built into Windows Server 2019 and newer), so the collector needs no WinRM and no agent on Windows. On the host, in an administrator PowerShell: ```powershell Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0 Start-Service sshd; Set-Service sshd -StartupType Automatic Add-LocalGroupMember -Group "Hyper-V Administrators" -Member "CORP\zev" ``` Hyper-V has no read-only role: reading VMs takes **Hyper-V Administrators**, a group that could also change them. What keeps it read-only is the collector: it sends one fixed PowerShell script of `Get-` cmdlets (`Get-VM`, `Get-VMNetworkAdapter`, `Get-VMNetworkAdapterVlan`, `Get-VMHardDiskDrive`, `Get-VHD`, `Get-VMSwitch`, `Get-NetAdapter`, `Get-NetIPAddress`, `Get-Volume`, `Get-ClusterNode`, `Get-ClusterSharedVolume`), ending in `ConvertTo-Json`, and nothing the server says can make it send anything else. The script is in zev's source (`src/core/netdev/virt_host.zig`). On the collector: ```bash zev site cred add hv --user 'CORP\zev' --password-stdin zev site test hv01.corp.local --cred hv --vendor hyperv ``` Add the host as a device: vendor **Microsoft Hyper-V**, credential `hv`. In a failover cluster, add one node: the collector asks every node for its VMs (`Get-VM -ComputerName`), and the cluster's nodes and shared volumes are listed. If the account can't reach the other nodes from there, add each node as its own device. Each VM's network adapter shows its virtual switch and VLAN (access mode; `trunk` for trunk mode), its addresses from the Hyper-V integration services, and its virtual disks with the volume or cluster shared volume they're on. A virtual switch's way out is its physical adapters, through a NIC team when there is one. If the account can't read the volumes, the storage list stays empty; the rest still works. ## Add an oVirt or OLVM engine oVirt, and the products built on it (**Oracle Linux Virtualization Manager**, Red Hat Virtualization), are read through the engine's REST API (`/ovirt-engine/api`, oVirt 4.4 or newer), with a user that has the built-in **ReadOnlyAdmin** role. On the engine machine, make the user in the internal domain: ```bash ovirt-aaa-jdbc-tool user add zev ovirt-aaa-jdbc-tool user password-reset zev --password-valid-to="2030-01-01 00:00:00Z" ``` Then in the Administration Portal: Administration → Configure → System Permissions → Add, user `zev`, role **ReadOnlyAdmin**. On an engine that logs in through Keycloak (oVirt 4.5's default), make the user in Keycloak instead; its name is then `zev@ovirt@internalsso`. On the collector: ```bash zev site cred add olvm --user zev@internal --password-stdin zev site test engine.lab.local --cred olvm --vendor ovirt ``` Add the engine as a device: its address, vendor **oVirt / OLVM**, credential `olvm` (HTTPS on 443). One device is everything that engine manages: its clusters, hosts, VMs, logical networks and storage domains. The collector only sends GET requests, a handful per collection (VMs come with their network cards, guest addresses, disks and tags in one request), with the password on each, so no session is left open. The TLS key is pinned the same way as for Proxmox. A VM's network card is on a **vNIC profile**, which belongs to a **logical network**, which may have a VLAN. On each host, a logical network is a bridge of the same name on the host port, bond or VLAN it's attached to (Setup Host Networks), so the way out is the same as on Proxmox: `nic1 → prod VLAN 20 → bond0 → eno1 → sw1 Gi1/0/8`. Addresses come from the guest agent the engine reports (`qemu-guest-agent` in the VM). Disks show the storage domain they're on; only data domains count as storage. A VM is known by its UUID. A host in maintenance or not answering the engine shows offline. ## Add a Nutanix Prism Central Nutanix AHV clusters are read through **Prism Central**'s v3 REST API (Prism Central 2022.6 or newer; a cluster only managed by Prism Element has to be registered with a Prism Central first), with a user that has the built-in **Prism Viewer** role. In Prism Central: Admin Center → IAM → Local User Management, add `zev`; then Authorization Policies → Create, role **Prism Viewer**, that user, on all clusters. A directory user works the same way. On the collector: ```bash zev site cred add prism --user zev --password-stdin zev site test prism.lab.local --cred prism --vendor nutanix ``` Add Prism Central as a device: its address, vendor **Nutanix AHV**, credential `prism` (HTTPS on 9440). One device is every cluster that Prism Central manages: their AHV hosts, VMs and subnets. Prism Central itself and its own VMs are left out. Prism's v3 API asks for lists with a POST whose body says which page to send (`POST /api/nutanix/v3/vms/list`). Those are the only requests the collector makes: clusters, hosts, subnets and VMs, page by page. A Prism Viewer can't change anything whatever is sent, and the password goes with each request, so no session is left open. The TLS key is pinned the same way as for Proxmox. A VM's network card is on a **subnet**, with its VLAN. Its addresses are the ones AHV knows: handed out by AHV's own IP address management, or learned from the VM's traffic. Disks show the storage container they're on; the CD-ROM is left out. Categories (`Environment: Production`) show as tags. A VM is known by its UUID. What Prism's v3 API doesn't say, and Zaryn therefore doesn't show yet: which physical ports a host's virtual switch uses, and how full each storage container is. A host's way out is what the switches' LLDP or CDP neighbors say about it, as for vSphere. ## Add an OpenStack cloud An OpenStack cloud is read through its own APIs: Keystone for a token and the service catalog, Nova for servers and hypervisors, Neutron for networks, ports and floating IPs, Cinder for volumes. Keystone must answer on HTTPS. Make a user with the **reader** role across the whole cloud (a system-scoped role): ```bash openstack user create --domain Default --password-prompt zev openstack role add --user zev --user-domain Default --system all reader ``` On the collector: ```bash zev site cred add cloud --user zev@Default --password-stdin zev site test keystone.cloud.local --cred cloud --vendor openstack ``` Add the cloud as a device: Keystone's address, vendor **OpenStack**, credential `cloud` (port 5000). One device is everything in that region: every project's servers, the hypervisors, networks and volumes. The user name is `name@domain`, and the token is system-scoped. A system reader sees every project when the cloud uses OpenStack's new policy defaults (`[oslo_policy] enforce_new_defaults` and `enforce_scope`, on by default in recent releases). On a cloud still using the old defaults, a reader only sees its own project: name it, `--user 'zev@Default/shop'`, with the reader role on that project (`openstack role add --user zev --project shop reader`). Then only that project's servers are listed, and the hypervisors stay hidden. With the old defaults, seeing every project takes the admin role, an account that could also change things. What the collector sends: one POST to Keystone for a token (the password goes in a private file, never on a command line), then only GETs, then a DELETE that revokes the token. Endpoints come from the token's catalog, using the public ones; HTTPS endpoints are pinned on first contact like Keystone, and HTTP endpoints are read as the cloud publishes them. A server's network cards are its Neutron ports: the network, its VLAN when it's a VLAN provider network (VXLAN and Geneve tenant networks have none), the fixed addresses with their subnet's prefix, and any floating IP. Disks are the flavor's ephemeral root disk, or the Cinder volumes attached, each with its backend (`ceph-ssd`). Each server is tagged with its project. A server is known by its UUID, so an evacuation or live migration shows as a move. What OpenStack's APIs don't say, and Zaryn therefore doesn't show yet: which physical ports a hypervisor uses, and how full each storage backend is. A hypervisor's way out is what the switches' LLDP or CDP neighbors say about it (install `lldpd` on compute nodes). ## What you see A site with a hypervisor gets a **Virtual machines** tab: - **Capacity**: hosts, guests running, vCPUs per physical CPU (overcommit), memory given out to running guests against what the hosts have, storage used (shared storage like Ceph counted once). - **What needs attention**: the risks below, worst first. Empty is the goal. - **If a switch goes down**: each switch the hosts are cabled to, with the hosts it would cut off, the running guests that would go with them, and the hosts that would only lose redundancy. - **Hosts**: CPU and memory in use, vCPUs on physical CPUs, the guests on each, the host's uplinks (which physical port goes to which switch port, and whether that port carries the VLANs) and a redundancy badge. - **Guests**: every VM and container with its host, state, size, addresses and VLANs; filter by name, ID, tag, address, MAC or VLAN. Open one for each network card's way out (`net0 → vmbr0 VLAN 20 → bond0 → eno1 → sw1 Gi1/0/5 ✓`), who it talks to, its disks and **where it can move**. - **Storage**: each datastore, pool, storage domain or container, how full it is, and when it will be full at its current pace. ### Where addresses come from In order: the **QEMU guest agent** (running VMs that have it), **VMware Tools**, Hyper-V's **integration services**, the guest agent as oVirt's engine reports it, what AHV and Neutron know, a container's own config, otherwise the **ARP tables** of the site's routers and switches, matched by MAC. Each address says where it came from. On KVM, the read-only connection can't ask the guest agent, so libvirt's own DHCP leases and the host's ARP table come first. ### Where the uplinks come from The switches' LLDP and CDP neighbors name the host and the host port on the other end. Install `lldpd` on Proxmox, KVM, oVirt and OpenStack compute hosts (`apt install lldpd`) so switches see them; on ESXi, turn on LLDP or CDP on the vSwitch or distributed switch; on Windows, the Data Center Bridging feature has an LLDP agent (`Install-WindowsFeature Data-Center-Bridging`, then `Enable-NetLldpAgent`). A port no switch reports shows as `→ ?`. ## The network checks in depth This section is for whoever has to trust the answers: exactly what each check looks at, and when it says "don't know" instead of guessing. A check never turns an unknown into a red mark. ### VLAN reach The classic VLAN mismatch: a vMotion or live migration lands a VM on a host whose switch port doesn't allow its VLAN. This check finds it before anyone notices the outage, and for every VM, not just the ones that moved. ```diagram type: graph title: The VLAN mismatch that nobody sees alt: A VM moves from host pve1 to host pve3, whose switch port allows only VLANs 1 and 30, so the VM is up but unreachable legend: path=Where the VM's traffic goes; ghost=Where it was; down=VLAN not carried start: web1 on pve1 caption: web1 runs on pve1, whose switch port carries VLAN 20. Everything works. --- box Rack A | 2 4 44 52 dim box Rack B | 54 4 44 52 dim node pve1 | pve1 | 14 20 | host | | Its uplink goes to sw1, a trunk that allows VLANs 1 and 20 node sw1 | sw1 trunk 1,20 | 34 20 | switch | good | Carries VLAN 20: web1's traffic gets through node web1 | web1 VLAN 20 | 50 40 | vm | accent | Tagged with VLAN 20 wherever it runs node pve3 | pve3 | 70 20 | host | | Same cluster, another rack: its uplink goes to sw3 node sw3 | sw3 trunk 1,30 | 88 20 | switch | | Allows VLANs 1 and 30 only link web1 pve1 | runs here | path link pve1 sw1 | VLAN 20 ✓ | path link web1 pve3 | | ghost link pve3 sw3 | | dim step Live migration to pve3 | HA or a vMotion moves web1 to pve3. The hypervisor says green: the VM is running. link web1 pve1 | ghost | was here link pve1 sw1 | dim link web1 pve3 | path | live migration link pve3 sw3 | path step VLAN 20 isn't on sw3 | sw3's trunk doesn't allow VLAN 20, so every frame web1 sends dies at the switch port. Zaryn reads sw3's allowed list and marks this uplink before anyone moves the VM. link pve3 sw3 | down | VLAN 20 ✗ node sw3 | bad node web1 | bad note 76 52 | up, but unreachable | bad ``` For each VM network card with a VLAN, Zaryn follows its uplinks to the switch ports at the other end (from LLDP or CDP), then reads those switch ports' VLANs from the switches' running configs, which the collector already keeps for [config history](/docs/en/network-sites): - **Cisco IOS / IOS-XE**: `switchport mode`, `switchport trunk allowed vlan` with its `add`, `remove`, `except`, `all` and `none` forms, `switchport trunk native vlan`, `switchport access vlan`. A trunk with no allowed list carries every VLAN. - **Huawei VRP**: `port link-type trunk | access | hybrid`, `port trunk allow-pass vlan` (with `to` ranges and `all`), `undo port trunk allow-pass vlan`, `port default vlan`, `port hybrid tagged vlan`. A trunk carries only VLAN 1 until told otherwise. A tagged VLAN on the VM side means tagged frames on the uplink, so the switch port must be a trunk (or hybrid) whose allowed list includes it. An access port fails the check even when its VLAN is the same number, because it doesn't accept the tagged frames. Each uplink then shows ✓ or ✗ with the reason: `trunk port without VLAN 20 (allowed: 10,30)`. No mark is given when the answer isn't known: a VM card without a VLAN (it uses the port's native or access VLAN, or a vSphere port group with VLAN 0 or 4095), a switch whose config isn't parsed yet (MikroTik, FRRouting, VyOS), or a host port no switch reports. ### Where a VM can move For each VM with a VLAN or a disk, every other online host of the same cluster is checked as if the VM had just moved there. The network and the disks get one mark each: 1. **Network**, for VMs with a VLAN. On platforms that report host networking (Proxmox, KVM, Hyper-V, oVirt), the host must have the same bridge, virtual switch or logical network: `no vmbr0 on pve3`. Each of the VM's VLANs must be carried by every switch port that host's uplinks reach: `VLAN 20 not on sw1 Gi1/0/6`. 2. **Disks.** Shared storage (Ceph, NFS, iSCSI, a cluster shared volume, a Nutanix container, a Cinder pool) has to be seen by the host: `ceph isn't on pve3`. A disk on local storage would be copied with the VM, so the host needs the same storage with room for it: `local-lvm on pve2 has 5 GiB free, its disks there need 20 GiB`. The answer is ✓ (safe), ✗ with the reasons, or `?` when something isn't known, such as the target's switch ports. On vSphere the reasons are the ones vMotion would give: the port group isn't on that host (`no prod-vlan20 on esx3`), or the disk is on a datastore only its current host mounts. If the SOAP API couldn't be read, a datastore counts as seen by a host only when another VM on that host has a disk on it. Run it before a maintenance window, before enabling DRS or HA on a cluster, or before draining a host. ### Uplink redundancy Each host's uplinks get one verdict: ```diagram type: graph title: Redundant, or only on the drawing alt: Host A has a cable to each of two switches; host B has two cables to the same switch; host C has one cable; a switch reboot shows which hosts go dark legend: new=Redundant: two switches; warn=One switch, two ports; down=Down start: Three hosts caption: Three hosts, three kinds of cabling. Zaryn gives each host's uplinks one verdict. --- node sw1 | sw1 | 30 10 | switch | | Rebooted for an upgrade in the next step node sw2 | sw2 | 70 10 | switch node a | host A | 15 42 | host | good | One cable to each switch: either can fail node b | host B | 50 42 | host | warn | Two cables, both to sw1: a bond, but one switch node c | host C | 85 42 | host | warn | One cable to sw2 link a sw1 | | new link a sw2 | | new link b sw1 | | warn link b sw1 | | warn link c sw2 | | warn step sw1 reboots | Host A keeps going over sw2. Host B loses both its cables at once: its VMs go dark. "If a switch goes down" lists exactly this, per switch. node sw1 | bad link a sw1 | down link b sw1 | down node b | bad note 50 54 | host B: every VM goes dark | bad ``` | Verdict | Meaning | |---|---| | redundant uplinks | Its uplinks reach two switches or more: one switch can fail | | one switch | Every uplink is seen, and all go to the same switch | | one port | The host has a single uplink | | uplinks unknown | No switch reports this host, or some of its ports aren't seen: Zaryn won't guess | ### If a switch goes down The blast radius of a switch failure, or of a planned switch reboot, before it happens. For each switch that any host's uplinks reach, Zaryn removes it and looks at what's left: - a host whose every uplink went to that switch is **cut off**, and its running guests are listed as going down with it; - a host with another uplink left only **loses redundancy**. A host port no switch reports counts as a possible way out, so this list errs on the side of "loses redundancy" rather than overstate. Use it to answer a change board's "what's the impact?" for a switch upgrade in seconds, or to find the hosts that need a second cable before the next one. ### Storage fill levels Every platform reports how full its storage is: ```diagram type: stats title: Fill levels, forecasts and thin disks caption: Fill comes from every platform. The forecast comes from an hourly sample, and VM disks added up are compared with what a storage holds. --- 87% | ceph-pool | Full in about 9 days | 87 | warn 300% | local-lvm | VM disks: 3 TB, thin, on 1 TB | 100 | bad 42% | nfs-iso | Stable, no trend | 42 | good ``` | Platform | What's measured | |---|---| | Proxmox VE | Each storage (Ceph, ZFS, LVM, NFS, directories) | | VMware vSphere | Each datastore | | KVM (libvirt) | Each storage pool | | Hyper-V | Each volume and cluster shared volume | | oVirt / OLVM | Each data storage domain | | Nutanix AHV | Each storage container (Prism's groups API) | | OpenStack | Each Cinder pool, and each hypervisor's local disk for ephemeral disks | ### Thin disks past what a storage holds Thin disks only take space as they fill, so a 1 TB datastore can carry 3 TB of VM disks while it's a third full. That works until the VMs fill their disks, and then everything on it stops writing at once. Each storage shows the sizes of the VM disks on it added up, next to its size. Above 100% it's overprovisioned (overcommitted), and how much is still free says how close it is to running out. Every platform gives each disk's size: Proxmox from the VM's config, vSphere from each disk's capacity, libvirt from `virsh domblkinfo`, Hyper-V from `Get-VHD`, oVirt from the disk's provisioned size, Nutanix from the VM's disk list, OpenStack from the flavor and the attached volumes. Snapshots, ISOs and backups on the same storage aren't counted. ### Storage forecasts Zaryn keeps one sample per storage (datastore, pool, volume or container) about every hour, for 90 days. The growth rate is a least-squares fit over the last 30 days, made once there are at least three samples spanning 12 hours; **full in about N days** is the free space divided by that rate. A storage that isn't growing says so. The forecast follows the trend; a sudden bulk copy won't show until it's sampled. ### Which VMs are filling it A storage filling up is one question; which VMs are filling it is the next. Where the platform says what each VM disk really takes on its storage (a thin disk grows toward its size as the guest writes), Zaryn samples that every hour too, for 35 days, and fits each VM's growth the same way. Each storage then lists the VMs growing fastest on it, and **Storage filling up** names the first one: `full in about 26 days at its pace over the last weeks (60.0 GiB a day); growing fastest: erp (12.0 GiB a day)`. A VM's own disks show what they take now (`120 GiB written`) and how fast they grow. ```diagram type: stats title: ds-ssd, over the last weeks caption: The storage grows 60 GiB a day. erp accounts for 12 of them; legacy isn't growing, so it isn't listed. --- 26 days | Until ds-ssd is full | 1.5 TiB free, at 60 GiB a day | 75 | warn +12 GiB | erp, a day | 120 GiB of its 200 GiB disk written | 60 | accent 0 | legacy, a day | not growing | 0 | good ``` | Platform | What each VM disk takes | |---|---| | Proxmox VE | The storage's volume list (`used`), where the storage type reports it (ZFS, Ceph RBD, LVM-thin and others); a container's root filesystem from the cluster's resources | | VMware vSphere | Each VM's committed bytes per datastore (SOAP), shared out over its disks there | | KVM (libvirt) | `virsh domstats --block`: a file's physical size, else the highest written offset | | Hyper-V | The virtual disk file's size (`Get-VHD`) | | oVirt / OLVM | The disk's actual size | | Nutanix, OpenStack | Not reported: the storage's own trend still works | ### What needs attention | Risk | Level | When | |---|---|---| | A VM's VLAN isn't carried by the switch port | high | A running VM's card leaves by a switch port that doesn't carry its VLAN | | Storage filling up | high under 7 days, medium under 30 | The forecast says it fills within 30 days; the VM growing fastest on it is named | | Storage nearly full | medium | Over 90% used, with no forecast yet | | Thin disks bigger than their storage | high when 80% used or more, medium below | The VM disks on it add up to more than its size | | All uplinks on one switch | medium | A host with running guests whose uplinks all reach one switch | | One uplink only | medium | A host with running guests on a single uplink | | Memory overcommitted | medium | Running guests were given more memory than the host has | ### What the collector sends The security model in one place. Every platform is read with a login that can only look; where a platform has no such role (Hyper-V, plain SSH for KVM), the collector itself is what keeps it read-only, because the commands it sends are fixed in its source. ```diagram type: sequence title: What crosses the wire lanes: Hypervisor | Collector | Zaryn alt: The collector sends read-only requests to the hypervisor, keeps the passwords itself, and sends only what it read to Zaryn over HTTPS --- Collector -> Hypervisor : read-only requests, nothing else Hypervisor --> Collector : the inventory note Collector : passwords and tokens stay here, in a file only it can read | good Collector -> Zaryn : what it read, outgoing HTTPS Zaryn --> Collector : nothing that can change a device ``` | Platform | Requests, and nothing else | |---|---| | Proxmox VE | `GET /api2/json/…` with an API token | | VMware vSphere | `POST /api/session` (login), `GET /api/vcenter/…`, `DELETE /api/session` (logout); SOAP on `/sdk`: `RetrieveServiceContent`, `Login`, `CreateContainerView`, `RetrievePropertiesEx` and `ContinueRetrievePropertiesEx`, `DestroyView`, `Logout` | | KVM (libvirt) | One SSH script: `virsh -r` (read-only connection) and `ip -j addr show` | | Hyper-V | One PowerShell script of `Get-` cmdlets, ending in `ConvertTo-Json` | | oVirt / OLVM | `GET /ovirt-engine/api/…`, the password on each request | | Nutanix AHV | `POST /api/nutanix/v3/…/list` and `POST /api/nutanix/v3/groups` (Prism's read calls are POSTs by design) | | OpenStack | `POST /v3/auth/tokens` (a token), `GET` on Nova, Neutron, Cinder and Keystone, `DELETE /v3/auth/tokens` (revoked) | Besides that: HTTPS keys are pinned on first contact (`zev site forget-key` after a certificate change on purpose) and SSH host keys kept like `known_hosts`; passwords and tokens stay on the collector, which sends Zaryn only what it read; passwords inside configs (cloud-init, switch secrets) are replaced before anything leaves the collector; nothing runs inside your VMs. ## Planning a migration in waves Leaving a hypervisor (VMware to Proxmox, an old cluster to a new one) goes in waves: a few VMs at a time, each wave moved, checked and kept before the next. The hard part is choosing the waves. Split two VMs that talk all day across the old and the new platform and every request crosses the link between them: slow, or broken if a firewall sits in the way. ```diagram type: flow title: How the waves are chosen loop: check each wave before the next --- Flows | Who talked to whom over the last week | accent | 📊 ~ grouped Groups | Keep-together sets, then the chattiest pairs | | 🔗 ~ packed Waves | At most the wave size, the quietest first | warn | 🌊 ~ one at a time Pilot wave | Move, check, keep | good | ✓ ``` **Plan a migration** at the bottom of the Virtual machines tab does it from what the site's [flows](/docs/en/network-flows) say, over the last week: 1. VMs that must stay together (the *keep together* sets, like `web1,db1; erp,erp-db`) start as one group. 2. The pairs of VMs that exchanged the most traffic are joined into one group while it stays within the wave size. 3. Groups are packed into waves of at most that many VMs. 4. The quietest waves come first: least traffic left split across waves, then least traffic inside, then fewest and smallest VMs. A good first wave is a pilot. Each wave lists: - its VMs, their size, addresses and warnings: not running (it moves cold), no address known or no guest agent/VMware Tools (whether it answers afterwards can't be checked automatically), large disks, several network cards; - the networks (bridges or port groups, with VLANs) the target must have before it starts; - traffic that stays split from other waves while it moves, with the services and volume; - what it needs outside the hypervisor: VMs on other hypervisors, devices, inside addresses, the internet. **Download as Markdown** gives the plan as a document for a change ticket. Without flows the waves are by size only, and the plan says so. ## History and changes Every VM and container config, and each host's network setup, is kept as the device's config: versions, diffs and drift work as for a router. Cloud-init passwords are replaced before anything leaves the collector. The site's **Changes** list says when a guest appears or disappears, moves to another host (live migration or HA), starts or stops, and when a host goes offline. The hypervisor is also on the site's **Map**, next to the switches it's cabled to. ## Limits - Proxmox VE 7 and 8; VMware vCenter 7.0 Update 2 and newer; libvirt 6 and newer; Windows Server 2019 and newer with Hyper-V (older ones work once OpenSSH Server is installed); oVirt and OLVM 4.4 and newer; Nutanix Prism Central 2022.6 and newer with AHV clusters; OpenStack with Keystone v3 and Nova API 2.47 or newer (Pike and newer). - Up to 2,000 guests per hypervisor device are read. - VLAN checks read Cisco IOS / IOS-XE and Huawei VRP switch configs; other switches leave the check unknown. Nutanix and OpenStack hosts' physical ports aren't in those platforms' APIs, so their uplinks come from LLDP or CDP only. - Uplinks, redundancy and switch failures need the switches to see the hosts over LLDP or CDP. - Storage forecasts need about a day of samples; Cinder pools need a reader the cloud's policy lets see them. - Reading only: Zaryn doesn't start, stop or move VMs, and [rollouts](/docs/en/network-rollouts) don't apply to hypervisors.