Who talks to whom: flows and dependencies

## What flow export is Routers and firewalls see every connection that crosses them. With **flow export** they summarize those connections and send the summaries to a collector: for each conversation, the two addresses, the protocol and ports, and how many bytes and packets went by. Cisco calls it NetFlow, the IETF standard is **IPFIX**, MikroTik calls it Traffic Flow. No packet contents are sent, only who talked to whom, how much. Switches more often speak **sFlow**. Instead of summarizing every connection, a switch sends the first bytes (the headers) of one packet in every N, 1 in 1,000 say, with N alongside. Because it's done in the switch's hardware it costs nothing at full speed, and over a minute those samples, multiplied by N, are a good estimate of who talks to whom. ## What Zaryn does with it On its own, a flow record is a pair of IP addresses. Zaryn already knows the site: its routers and switches, its hypervisors, and every VM with its addresses. So the site's collector receives the flows, adds them up, and Zaryn names both ends: - **web1 (VM) → db (VM): postgresql (5432), redis (6379), 1.2 GB a day** - **erp (VM) → web1 (VM): https (443)** - **web1 (VM) → 8.8.8.8 (internet): dns (53)** That's the dependency map people usually draw by hand before a migration or after an outage: which machines must move together, what a maintenance window takes down, which VMs depend on something outside. ```diagram type: graph title: Flows become a dependency map alt: An ERP VM talks to a web VM over HTTPS; the web VM talks to a database over PostgreSQL and to the internet over DNS legend: path=Observed traffic caption: Routers and switches send summaries of connections; Zaryn names the machines behind the addresses. --- node erp | erp | 10 30 | vm node web1 | web1 | 40 30 | vm | accent node db | db | 72 14 | storage node net | internet | 72 46 | cloud link erp web1 | https 443 | path link web1 db | postgresql 5432 | path link web1 net | dns 53 | path ``` ## Turn it on On the collector (the setting lives only on that machine): ```bash sudo zev site flows on # NetFlow/IPFIX on UDP 2055, sFlow on UDP 6343 sudo zev site flows on --port 9995 --sflow-port 6343 ``` Restart the collector (`sudo systemctl restart zev-site`, or `zev site run` again) so it starts listening, then point the routers' flow export at it. It listens on IPv4 and IPv6, so exporters can reach it on either. MikroTik (RouterOS 7): ```text /ip traffic-flow set enabled=yes interfaces=all /ip traffic-flow target add dst-address=<collector> port=2055 version=ipfix ``` Cisco IOS-XE (Flexible NetFlow): ```text flow exporter ZEV destination <collector> transport udp 2055 export-protocol ipfix flow monitor ZEV-MON exporter ZEV record netflow ipv4 original-input flow monitor ZEV-MON6 exporter ZEV record netflow ipv6 original-input interface GigabitEthernet1/0/1 ip flow monitor ZEV-MON input ipv6 flow monitor ZEV-MON6 input ``` Linux routers and hypervisors: `softflowd -i eth0 -n <collector>:2055 -v 10`, or pmacct. `zev site flows status` shows the ports and the last upload: how many conversations, from how many exporters. ## sFlow Send the switches' sFlow to the collector on UDP 6343. A sampling rate of 1 in 1,000 to 1 in 4,000 is plenty for dependencies; on busy 10G and faster links the vendor's recommendation is fine. Arista EOS: ```text sflow sample 1024 sflow destination <collector> 6343 sflow source-interface Management1 sflow run ``` Cisco Nexus (NX-OS): ```text feature sflow sflow sampling-rate 4096 sflow collector-ip <collector> vrf management sflow collector-port 6343 sflow agent-ip <switch management address> sflow data-source interface ethernet 1/1 ``` Juniper Junos: ```text set protocols sflow collector <collector> udp-port 6343 set protocols sflow sample-rate ingress 1024 set protocols sflow interfaces ge-0/0/0 ``` Open vSwitch (Proxmox VE with OVS bridges, KVM hosts), on each host: ```bash ovs-vsctl -- --id=@s create sflow agent=vmbr0 target='"<collector>:6343"' sampling=1000 polling=20 -- set bridge vmbr0 sflow=@s ``` Linux hosts and hypervisors with plain bridges: [host sFlow](https://sflow.net) (`hsflowd`), with `collector { ip=<collector> udpport=6343 }` and `pcap { dev=vmbr0 }` in `/etc/hsflowd.conf`. Sampling on the hypervisor's own bridge also sees VMs talking to each other on the same host, which no router sees. A switch is listed under its sFlow agent address (the one it puts in each datagram), so one that sends from a management VRF or through NAT is still listed under its own address. ## What's kept The collector understands **NetFlow v5, NetFlow v9, IPFIX and sFlow v5**, over **IPv4 and IPv6**, with IPv4 and IPv6 addresses in the records. Sampled counts are multiplied back up. From sFlow it reads each sampled packet's headers (Ethernet, VLAN tags, IPv4 or IPv6, TCP, UDP or SCTP ports), or the switch's own decoded IPv4 or IPv6 record when it sends one; interface counter samples are skipped, since the site's devices already report their interfaces. Every minute it sends Zaryn the busiest 5,000 conversations, each reduced to client, server, the server's port and the protocol (the client's random port is dropped), with bytes, packets and flow counts. Zaryn sums them per hour and keeps 30 days. A conversation's direction is worked out from the ports: the side on the well-known port (below 49152, the lower one when both are) is the server. A reply sent as a separate flow lands on the same conversation. ## Where you see it - The site's **Dependencies** tab: every pair of endpoints that talked in the last hour, day, week or 30 days, the services between them and how much, each end named as a VM, device, hypervisor host, an inside address Zaryn doesn't know, or the internet. - Each VM's details, opened from its row: what it talks to and what talks to it. Addresses are named from what Zaryn knows: devices and their interfaces, hypervisor hosts, and VMs (their guest agent or VMware Tools addresses, their container config, or the ARP tables). IPv6 addresses are matched however they're written (`2001:DB8:0:0::5` and `2001:db8::5` are the same VM). Unique local IPv6 addresses (`fc00::/7`) and link-local ones count as inside, like the private IPv4 ranges. ## Limits - sFlow is sampled: a conversation of only a few packets may not show up at all, and small volumes are estimates. Its byte counts are frame lengths, Ethernet headers included. - Flows are only as complete as the routers that export them: traffic that never crosses an exporting router (two VMs on the same host and VLAN) isn't seen, unless the hypervisor's own bridge sends sFlow (above).