Path Explain and probes

"Why can't the office reach the web server?" Answering that usually means logging in to router after router, reading routing tables and access lists. **Path Explain** does it for you, from what the devices of a [site](/docs/en/network-sites) said last: - **Every hop.** The route each device uses (`172.16.20.0/24 via 10.0.2.2, OSPF`), the interface it leaves by, the ACLs and firewall rules it passes, and the rule that let it through. - **Where it stops, and why.** The device, the interface, and the exact rule that drops it, or the missing route, the routing loop, the interface that's down. - **The way back.** The reply is followed too. A reply that takes another way through a stateful firewall is dropped as invalid, the classic **asymmetric routing** problem, and Path Explain says so. - **The fix**, in the commands of that device's vendor: where to insert the permit rule, which static route to add and through which neighbor. **Probes** run on the site's collector: ping, a TCP port, an HTTP(S) URL, a DNS lookup, every 30 seconds to an hour. When one fails, Path Explain says whether the network is why. Zaryn only suggests. It never changes a device. ## What the collector reads for it The collector (zev 0.8 or later) reads two more things from each device, still with read-only commands: | Vendor | Routing table | Filters | |---|---|---| | Cisco IOS / IOS-XE | `show ip route` | ACLs on interfaces (`ip access-group`), from the config | | Huawei VRP | `display ip routing-table` | `traffic-filter` ACLs on interfaces, from the config | | MikroTik RouterOS | `/ip route print terse` | `/ip firewall filter`, with address lists and interface lists | | FRRouting, VyOS, Linux | `ip -j route` (the kernel's table) | iptables (`iptables-save`) and nftables (`nft -j list ruleset`) | On Linux, reading the firewall needs root. Let the collector's login run only those two commands, read-only: ```text # /etc/sudoers.d/zev-site netops ALL=(root) NOPASSWD: /usr/sbin/iptables-save, /usr/sbin/nft -j list ruleset ``` Without it, Path Explain says it couldn't read that device's firewall, and doesn't pretend it has none. A device's **Routes and filters** tab shows what was read. Search for an address to see the routes that hold it, and which one is used. ## Ask: why can't A reach B? On the site's **Paths** tab: - **From** and **To**: an IPv4 address, a device's name, or `collector` (the collector's own address). - **Protocol** and **port**: TCP or UDP with a port, ICMP (ping), or any IP. - **Enters at** (optional): the device the traffic comes in by, when the source isn't on one of the site's subnets (traffic from the internet, from a VPN) or when a subnet has two gateways. For example, from `192.168.10.20` (a laptop) to `172.16.20.10` (a server), TCP port 80: ```text Blocked by a filter sw1 drops it (outbound on GigabitEthernet0/2): SERVERS-OUT, rule 40 deny ip any any log gw bridge → ether2 route 172.16.0.0/16 via 10.0.1.2 (static) allowed · firewall, passing through · forward (no rule matched) sw1 Gi0/1 → Gi0/2 route 172.16.20.0/24 via 10.0.2.2 (ospf) dropped · outbound on Gi0/2 · SERVERS-OUT #40 deny ip any any log ``` ## Reading the answer | Answer | What it means | |---|---| | Reachable, both ways | It gets there, and the reply comes back. | | Leaves the site | It leaves toward a next hop that isn't one of the site's devices (an ISP, another site). Zaryn can't see past it. | | Gets there, but the reply doesn't come back | The request arrives. The reply is dropped on its way back (often asymmetric routing). | | Blocked by a filter | An ACL or firewall rule drops it. The answer quotes the rule. | | No route | A device has no route to the destination, and no default route. | | Discarded by a route | A route sends it to Null0 or a blackhole. | | Routing loop | Two or more devices send it back and forth. | | An interface on the way is down | The route uses an interface that's down or shut down. | | May be blocked | A rule Zaryn can't fully evaluate comes before the one that decides (an object group, a named set, a time range). The answer names it. | | Can't tell | Something is missing: a device's routes weren't collected, or the source isn't on any subnet of the site. | When there are **equal-cost paths** (ECMP), each one is followed and you can switch between them. **Show it on the map** highlights the devices on the way. **What each answer rests on** is listed under it: a device whose firewall couldn't be read, two gateways on the source's subnet, NAT on the way. ## The fix: the ACL rule, the route, the interface Under a failing answer, Zaryn gives the commands for that device: ```text # Cisco: just before the rule that drops it ip access-list extended SERVERS-OUT 35 permit tcp host 192.168.10.20 host 172.16.20.10 eq 80 # Huawei: a rule id below the one that drops it acl number 3001 rule 2 permit tcp source 192.168.10.20 0 destination 172.16.20.10 0 destination-port eq 3389 # MikroTik: placed before the rule that drops it /ip firewall filter add chain=forward action=accept protocol=tcp src-address=192.168.10.66 dst-address=172.16.20.10 dst-port=443 place-before=[find comment="quarantine"] # iptables / nftables iptables -I FORWARD 2 -s 10.81.1.10 -d 10.81.2.10 -p tcp --dport 8080 -j ACCEPT nft insert rule inet fw forward position 4 ip saddr 10.81.1.10 ip daddr 10.81.2.10 udp dport 53 accept ``` For **no route**, Zaryn looks for a neighbor on the same subnet that knows the way, and gives the static route through it (`ip route`, `ip route-static`, `/ip route add`, VyOS `set protocols static`). For an interface that's **shut down**, the command that brings it up. When there's no room for a rule number (Cisco sequence 20 and 21 with nothing between), the fix renumbers the list first (`ip access-list resequence`, Huawei's `step`). Read the commands before you use them. A permit for one host is the narrowest fix. You may want a wider one. ## When the reply takes another way Stateless ACLs (Cisco, Huawei) need a rule for the reply too, unless one allows `established` traffic. Stateful firewalls (MikroTik, iptables, nftables) let a reply back in when they saw the request. If the reply comes back through a firewall the request never went through, that firewall sees a reply to nothing it knows and drops it as **invalid**. MikroTik's default configuration does exactly that. Path Explain follows the reply through each device with what that device saw, so it finds this case. ## Watched paths After an answer, choose **Watch this path** and give it a name (`users-to-web`). Zaryn checks it again whenever a device on the way reports different routes, filters or interfaces. When the answer changes, **Changes** says so, with the device and the rule ("watched path users-to-web: Reachable → Blocked by a filter, at sw1: deny ip any any"). Someone's ACL change that broke a path shows up minutes later, not when users call. ## What it doesn't follow yet - **NAT.** When a device on the way does NAT, the answer says so and follows the original addresses. - **VRFs and policy routing.** Only the main (global) routing table is read. - **Cisco zone-based firewall** and **Huawei's `match-order auto`**. The answer says when a device has them. - **IPv6.** ## Probes On the site's **Health** tab, add a probe. The collector runs it as often as you choose (30 seconds to an hour): | Kind | Target | What's measured | |---|---|---| | Ping | an address or a name | round trip and loss (4 echoes) | | TCP port | an address or a name, and a port | how long the connection takes. **Refused** (the host answered, nothing listens) is told apart from **no answer** (dropped on the way) | | HTTP(S) | a URL | response time and status. Fine by default: 2xx and 3xx. Set the expected status (`200`, `2xx`, `200,401`) | | DNS lookup | a name, or `name@server` to ask one server | answer time. Optionally, an address the answer must hold | Each probe shows its last day (a bar per half hour), the share of checks that passed, and a chart for 6 hours, a day or a week. Results are kept for a week. A probe becomes **failing** after two failures in a row, and is back to normal after one success. Both show in **Changes**. When it starts failing, Zaryn runs Path Explain from the collector to the target: - If the network path is fine, the problem is at the target itself (the service is down, the port is closed). - If not, it names the device and the rule. **Open in Path Explain** shows the whole answer. Try a probe from the collector's shell, as the service runs it: ```bash zev site probe ping 10.0.0.10 zev site probe tcp 10.0.0.10:443 zev site probe http https://intranet.example/health --expect 200 zev site probe dns intranet.example@10.0.0.53 --expect 10.0.0.10 ``` ## Latency between sites: mesh and the health map With **mesh** on (Health tab), a site's collector pings the collectors of your other sites that have mesh on too. The **Sites** page then shows a **health map**: latency and loss between every pair of sites, with rows where the ping starts. A link between two sites going down shows on both. ## Limits Free: 10 probes and 10 watched paths per site. Premium: 300 probes and 200 watched paths per site. Mesh probes don't count.