Underlay
The routed network between switches. Its only job is to reach every tunnel endpoint loopback over equal cost paths. I keep it plain and stable.
Data centre fabrics
VXLAN EVPN and ACI, explained packet by packet
I design data centre fabrics where any leaf can carry any segment, routing happens at the first hop and policy follows the workload. This page walks through VXLAN (Virtual Extensible LAN), the EVPN (Ethernet virtual private network) control plane, Cisco ACI (Application Centric Infrastructure) and the links between sites, with the commands I use to prove each layer.
Why overlays
Layer 2 everywhere was the problem, routing fixed it
Older data centres stretched VLANs across every switch, so one broadcast domain was also one failure domain. Spanning tree blocked half the links to stop loops. An overlay fabric routes between switches and carries the tenant segments inside tunnels instead.
Swipe sideways to see the whole diagram.
Stretched Layer 2. Spanning tree (STP) picks one root and blocks every other path, so half the uplinks carry nothing. The VLAN reaches every switch, so a loop or a broadcast storm in one rack can reach them all.
Fabric and overlay. The links between switches are routed, so every link carries traffic through equal cost multipath (ECMP) and there is no spanning tree inside the fabric. Tenant segments ride in VXLAN tunnels between leaves, so any segment can appear on any leaf.
Overlay
The tunnels and the control plane that tell each leaf where every host lives. Tenants, VRFs and gateways all live here.
Two blast radii
A tenant change cannot break the underlay, and an underlay fault shows up in every tenant. That split tells me where to look first when something fails.
VXLAN
Fifty bytes that carry a whole Ethernet frame
VXLAN, defined in Request for Comments (RFC) 7348, wraps the original Ethernet frame in UDP (User Datagram Protocol) and IP, so a routed network can carry it. The device that wraps and unwraps is the VTEP (VXLAN tunnel endpoint). On a Nexus switch it is the NVE (network virtualisation edge) interface. Step through how a leaf builds the packet.
Swipe sideways to see the whole diagram.
The virtual machine sends a normal Ethernet frame on VLAN 110, tagged by the host's virtual switch on a trunk port. Leaf 1 looks up the destination MAC address and finds it behind leaf 4.
Leaf 1 removes the VLAN tag and adds 8 bytes of VXLAN header carrying the 24 bit segment number, VNI (VXLAN network identifier) 10110.
8 bytes of UDP to port 4789. The source port is a hash of the inner headers, which spreads flows across the spines.
20 bytes of IP from leaf 1's tunnel loopback to leaf 4's. A VTEP must not fragment (RFC 7348), and many set the do not fragment bit, so the MTU (maximum transmission unit) must be big enough.
14 bytes of Ethernet to the next hop. The spines route on the outer header only.
1,500 + 14 inner Ethernet + 8 VXLAN + 8 UDP + 20 outer IP = 1,550 bytes of IP MTU, or 1,570 over IPv6. The outer Ethernet header is not part of the IP MTU. I set 9,216 on the fabric so even 9,000 byte host frames fit.
Overlay MTU calculator
A 1,500 byte packet in VXLAN over IPv4 needs an underlay IP MTU of at least 1,550 bytes. I set 9,216 on every fabric link and SVI (switched virtual interface). Then I prove it with a ping of 9,188 bytes of data, which is 9,216 minus 28 bytes of IP and ICMP (Internet Control Message Protocol) headers, with the do not fragment bit set.
Indicative sums for planning. Geneve options vary in length, which is why VMware asks for at least 1600 and recommends 1700. An outer VLAN tag adds 4 bytes to the frame, not to the IP MTU.
| Number | Value | Why it matters |
|---|---|---|
| UDP destination port | 4789 (Geneve uses 6081) | Firewalls and filters between tunnel endpoints must allow it. Some early Linux tunnels used 8472. |
| VNI size | 24 bits, 16,777,216 segments | Far more than 4,094 VLANs. Leaves still map each VNI to a local VLAN on their own ports. |
| Overhead | 50 bytes over IPv4, 70 over IPv6 | Sets the minimum underlay MTU. VXLAN has no fragmentation of its own. |
| Fabric MTU | 9216 on links, SVIs and the L3VNI (Layer 3 VNI) SVI | Covers 9000 byte hosts and NSX Geneve inside VXLAN. |
| Encryption | None in VXLAN itself | I keep the underlay unreachable from tenants, and encrypt links that leave a controlled space with MACsec (Media Access Control security), CloudSec (Cisco encryption between Multi-Site gateways) or IPsec (IP security). |
VLAN to VNI mapping
The VNI is fabric wide and the VLAN is local to a leaf. VLAN 110 on leaf 1 and VLAN 210 on leaf 4 can both map to VNI 10110. My standard is one mapping across the fabric, with local differences only as recorded exceptions.
Hardware or software VTEP
A leaf ASIC (application specific integrated circuit) encapsulates at line rate for anything plugged into it. A hypervisor, such as NSX, or a container network can be the VTEP instead. Then the fabric only sees the outer addresses, and the MTU must allow both layers.
VXLAN, NVGRE and Geneve
VXLAN has a fixed 8 byte header and suits switch fabrics. NVGRE (Network Virtualization using Generic Routing Encapsulation) uses GRE (Generic Routing Encapsulation) and gave poor load sharing. Geneve (RFC 8926) adds variable options and is used inside hypervisors such as NSX. A VXLAN variant, the Group Policy Option, uses reserved header bits to carry a 16 bit group tag. Cisco SD-Access uses it to carry the security group tag.
Underlay
Keep the underlay plain and predictable
The underlay gives every tunnel endpoint a route to every other one, over every spine. It runs the same way whatever the tenants do. The main choice is which protocol carries those routes, and how the EVPN sessions sit on top.
Swipe sideways to see the whole diagram.
IGP (interior gateway protocol) underlay, iBGP overlay. OSPF (Open Shortest Path First) or IS-IS (Intermediate System to Intermediate System) carries the loopbacks. Each leaf peers with two or more spines acting as route reflectors for EVPN. One AS (autonomous system) keeps automatic route targets working, and controllers template it easily. It is my default for a single pod.
eBGP everywhere (RFC 7938). One protocol for underlay and overlay, as large and multi vendor fabrics often use. Spines must keep routes they do not import (retain route-target all) and leave the next hop unchanged, and automatic route targets differ per AS, so I set them by hand or rewrite them.
Addressing plan for one pod
- Loopback 0: router ID and BGP (Border Gateway Protocol) source, one /32 per switch.
- Loopback 1: VTEP source, kept separate from the router ID. A vPC (virtual port channel) pair adds a shared secondary address.
- Fabric links: /31 point to point subnets, or IP unnumbered from loopback 0.
- IPv6: a matching plan, so the fabric is ready when tenants need it.
- Numbering: L2VNI (Layer 2 VNI) equals 10000 plus the VLAN, L3VNI equals 50000 plus the VRF (virtual routing and forwarding) number, the same in every pod.
What I set in every underlay
- MTU 9216 on every link, proved with full size pings.
- BFD (bidirectional forwarding detection) so a dead link is noticed in well under a second.
- Equal cost multipath across every spine, with different hash seeds per tier.
- At least two route reflectors on different spines.
- Graceful removal (maintenance mode) so a spine can be drained before work.
EVPN
BGP tells every leaf where each host lives
VXLAN on its own has to flood to learn. EVPN (Ethernet VPN, RFC 7432, with RFC 8365 for VXLAN) uses MP-BGP (multiprotocol BGP) to share MAC addresses, IP addresses and routes before traffic needs them. Step through what leaf 4 learns as a virtual machine boots and later moves.
Swipe sideways to see the whole diagram.
Each leaf sends a type 3 route (inclusive multicast Ethernet tag) for every VNI it carries. Together they build the list of VTEPs that must receive broadcast and unknown traffic for that segment.
VM A boots on leaf 1. The leaf learns the MAC address on its port and the IP address from ARP (Address Resolution Protocol) or DHCP (Dynamic Host Configuration Protocol). Then it sends a type 2 route with both, the L2VNI, the L3VNI and its router MAC.
The spines reflect the route to every leaf. Leaf 4 imports it because the route target matches, then installs the MAC behind leaf 1, an ARP entry it can answer locally and a /32 host route in VRF PROD.
The border leaf learns a default route from the firewall or WAN (wide area network) and sends it as a type 5 (IP prefix) route with the tenant L3VNI. Every leaf now knows the way out.
VM A moves to leaf 2. Leaf 2 sends a type 2 route with sequence number 1, which beats 0. Leaf 1 withdraws its route and leaf 4 points the MAC and host route at leaf 2.
| Route type | Name | What it carries | When I see it |
|---|---|---|---|
| 1 | Ethernet auto discovery | Which leaves share an Ethernet segment, for fast mass withdraw and load sharing | Multihoming with an ESI (Ethernet segment identifier) |
| 2 | MAC and IP advertisement | A host's MAC, optionally its IP, the VNIs, router MAC and a mobility sequence number | Every day. Drives ARP suppression and host routes. |
| 3 | Inclusive multicast Ethernet tag | "Send me broadcast and unknown traffic for this VNI", with the replication method | Every VNI on every VTEP. Builds the flood list. |
| 4 | Ethernet segment | Discovery of leaves on the same segment, used to elect a designated forwarder | Multihoming with an ESI |
| 5 | IP prefix | A subnet or external route in a VRF, with the L3VNI | Border leaves, silent host subnets and links between fabrics |
| 6 to 8 | Multicast routes | Tenant multicast interest and IGMP (Internet Group Management Protocol) join and leave sync | Only with tenant multicast. Cisco Tenant Routed Multicast (TRM) adds multicast VPN (MVPN) routes and a rendezvous point in the fabric. |
Route distinguisher and route target
The route distinguisher (RD) makes each leaf's route unique, so the route reflectors keep both copies for load sharing. It does not decide anything. The route target (RT) decides which VRF or VNI imports a route. On NX-OS, rd auto builds the RD from the router ID, and route-target both auto builds the RT from the AS and the VNI, for example 65101:10110. In an eBGP overlay every leaf has its own AS, so automatic RTs no longer match. I set them by hand or rewrite them. Forgetting send-community extended is the classic build fault: the route arrives without its RTs and nobody imports it.
Distributed anycast gateway
Every leaf with a subnet holds the same gateway IP and the same virtual MAC, set once with fabric forwarding anycast-gateway-mac. A host always finds its gateway on its own leaf, so there is no trip to a central router, no failover protocol, and a moved virtual machine keeps its gateway. DHCP relay needs a unique source loopback per leaf, or replies go to the wrong leaf.
ARP suppression
ARP is the largest source of broadcast traffic. With suppress-arp, a leaf answers ARP requests from its cache of type 2 routes, so the request never crosses the fabric. Unknown entries still flood, and IPv6 uses the same idea for neighbour discovery.
Silent hosts and duplicates
A host that never sends has no type 2 route. I advertise each subnet as a type 5 route, so traffic reaches a leaf that owns the subnet, which sends an ARP request and the host appears. Duplicate detection stops a MAC that moves too often: by default five moves in 180 seconds, usually a loop or a badly teamed server.
What a leaf configuration looks like
I generate leaf configuration from templates or the controller rather than typing it, but I read it the way an engineer would. Each block below maps to a layer of the fabric.
! features
nv overlay evpn
feature bgp
feature interface-vlan
feature fabric forwarding
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
! tenant VRF and its L3VNI (transit VLAN 3001)
vlan 3001
vn-segment 50001
vrf context PROD
vni 50001
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan3001
vrf member PROD
mtu 9216
ip forward
! a tenant segment with its anycast gateway
vlan 110
vn-segment 10110
interface Vlan110
vrf member PROD
ip address 10.10.1.1/24
fabric forwarding mode anycast-gateway
! the VTEP
interface nve1
host-reachability protocol bgp
source-interface loopback1
member vni 10110
suppress-arp
ingress-replication protocol bgp
member vni 50001 associate-vrf
! EVPN instance and BGP to the spine route reflectors
evpn
vni 10110 l2
rd auto
route-target import auto
route-target export auto
router bgp 65101
neighbor 10.254.0.1
remote-as 65101
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf PROD
address-family ipv4 unicast
advertise l2vpn evpnIllustrative NX-OS configuration. Addresses, AS numbers and VNIs are examples from my numbering standard.
Packet walks
Bridged and routed, hop by hop
Two virtual machines in the same subnet are bridged inside one VNI. Machines in different subnets are routed with symmetric IRB (integrated routing and bridging), where both leaves route and the tenant L3VNI carries the packet between them. Watch the headers change at each hop.
Swipe sideways to see the whole diagram.
VM A has already resolved VM B's address, answered locally by ARP suppression. It sends a normal frame tagged VLAN 110.
Leaf 1 finds VM B's MAC behind leaf 4 from a type 2 route. It removes the tag, adds VXLAN with VNI 10110, UDP to port 4789 and an outer IP header from its VTEP to leaf 4's.
The spine sees an ordinary IP packet between two loopbacks. It routes it, rewrites the outer MAC addresses and lowers the outer TTL (time to live). The inner frame is never read.
Leaf 4 sees its own address and port 4789, removes the outer headers, maps VNI 10110 to its local VLAN 110 and delivers the frame exactly as VM A sent it.
Swipe sideways to see the whole diagram.
VM C is in another subnet, so VM A sends the packet to its gateway. The anycast gateway MAC lives on leaf 1, the leaf VM A is plugged into.
Leaf 1 routes in VRF PROD and finds a /32 host route to VM C via leaf 4. It encapsulates in the tenant L3VNI 50001, with leaf 4's router MAC as the inner destination.
The spine does not know about tenants. It routes the outer header to leaf 4 over equal cost paths.
Leaf 4 maps VNI 50001 to VRF PROD, routes into VLAN 120 and delivers to VM C. The TTL dropped by two, and leaf 1 never needed VLAN 120. The reply mirrors this path.
| Model | How it routes | What each leaf needs | My view |
|---|---|---|---|
| Symmetric IRB | Ingress leaf routes into the L3VNI, egress leaf routes into the destination subnet | Only its own subnets, plus one L3VNI per VRF | The NX-OS default and my standard. It scales. |
| Asymmetric IRB | Ingress leaf routes and bridges into the destination VNI, egress leaf only bridges | Every subnet, SVI and ARP entry on every leaf | Simpler to read, but the tables grow with the whole fabric. |
A north to south walk, from a branch user to a virtual machine, enters at the border leaf as a type 5 or type 2 lookup. The border leaf hands each VRF to the firewall or WAN as VRF Lite: one subinterface and one eBGP session per VRF, with the VRF's L3VNI inside the fabric. It crosses the fabric in the tenant L3VNI and returns through the virtual machine's own leaf. I check that a stateful firewall on the way sees both directions, and that the MTU holds end to end.
BUM traffic
Broadcast, unknown and multicast, kept on a short lead
Some frames must reach every leaf in a segment: broadcasts such as ARP and DHCP discover, unicast to an unknown MAC, and multicast. A fabric can copy them at the sending leaf, let the underlay multicast them, or avoid most of them altogether.
Swipe sideways to see the whole diagram.
Ingress replication. The sending leaf makes one unicast copy for every VTEP that sent a type 3 route for the VNI. No multicast in the underlay, so it is simple and works across links between sites. The cost is extra copies on the sender's uplinks.
Underlay multicast. Each VNI maps to a multicast group, and the spines act as rendezvous points with anycast RP. One copy leaves the sender. It scales better for many VTEPs and heavy multicast, but the team must run PIM (Protocol Independent Multicast).
ARP suppression. Most broadcast traffic is ARP, and the leaf already knows the answer from type 2 routes. DHCP relay at the anycast gateway removes DHCP broadcasts too. Storm control on host ports caps what is left.
Flood and learn
VXLAN without a control plane floods every unknown frame to a multicast group and learns MACs from what arrives. It works, but it has no ARP suppression and no proper routing between tenants. I treat it as a legacy mode.
Scope the segments
A VNI configured only on the leaves that need it keeps the flood list short. Every extra leaf in the list is another copy of every broadcast.
Load sharing
Spines only see the outer headers, so every flow between two leaves would look the same. The UDP source port hash fixes that for normal flows. Very large single flows still take one path.
vPC and ESI
Dual attached servers, two ways to do it
Servers connect to two leaves so a leaf or cable can fail. Cisco fabrics usually pair leaves with vPC (virtual port channel). The open standard is EVPN multihoming with an Ethernet segment identifier (ESI), which needs no peer link.
Swipe sideways to see the whole diagram.
vPC pair. Both leaves share one anycast VTEP address, so remote leaves send to it and the underlay load shares to both. With advertise-pip, routed type 5 traffic uses each leaf's own address, so it lands on the leaf that has the route. Single attached orphan devices are reached over the peer link, so I keep them to a minimum.
EVPN multihoming. Both leaves advertise the same ESI. Type 4 routes elect one designated forwarder per VLAN for broadcast traffic, so the server never gets two copies. If a link fails, one type 1 route withdraws the whole segment at once. It suits multi vendor fabrics and more than two leaves.
Designated forwarder election
The default method orders the leaves by IP address. VLAN V goes to leaf number V mod N, counting from zero. Try it.
VLAN 110 mod 2 is 0, so the leaf with the lowest IP address forwards broadcast traffic for VLAN 110.
vPC rules I keep
- Peer link and keepalive on physically separate paths.
- Type 1 settings identical on both peers, checked with
show vpc consistency-parameters globalandshow vpc consistency-parameters interface port-channel 10. - Auto recovery on, so one peer can bring vPCs up alone after a double failure.
- Changes always applied to both peers in the same change.
- Upgrades one peer at a time.
The vPC failure cases themselves are on the data centre networks page.
Between sites
Join fabrics, not failure domains
With EVPN Multi-Site, each data centre or pod keeps its own fabric, underlay and route reflectors. Border gateways (BGWs) at each site exchange routes and carry only the segments I choose to extend. A fault or a bad change stays in one site.
Swipe sideways to see the whole diagram.
Leaf C3 advertises VM C. The site 2 gateways advertise those routes again across the link between sites, using their shared virtual address and their own router MAC. Where the sites use different AS numbers, they also rewrite the route targets. Site 1 gateways do the same into site 1.
Leaf A1 routes the packet in VRF PROD and sends it in a VXLAN tunnel to the site 1 gateways' shared address. Leaves only ever tunnel to their local gateways.
The gateway removes the tunnel and builds a new one to the remote gateways across the link between sites, encrypted with MACsec on each link or CloudSec between gateways. Every 100 km adds about 1 ms of round trip time.
The site 2 gateway builds the last tunnel to leaf C3, which routes into VM C's subnet. Three tunnels, not one, so neither site depends on the other's internals.
If a gateway loses every link to the other site, tracking withdraws its shared address so the fabric stops sending to it. A restore delay stops it flapping back too soon.
Distance and what it allows
40 km of fibre adds about 0.4 ms of round trip time. After allowing about 1 ms for equipment, that is fine for synchronous storage replication, a stretched vSAN cluster and ACI Multi-Pod.
Light in fibre travels about 200,000 km a second, so 5 µs per km each way, or 1 ms of round trip time per 100 km. Fibre routes run a third or more longer than a straight line, and optical, encryption and firewall equipment add more. These are indicative limits. Product limits come from the vendor's current documentation.
| Transport between sites | Typical use | What I check |
|---|---|---|
| Dark fibre | Metro primary links, with my own optics: LR (long reach) to 10 km, ER (extended reach) to 40 km, coherent 400ZR to about 120 km | Two physically diverse routes, MACsec with keys agreed by MKA (MACsec Key Agreement), jumbo MTU end to end |
| DWDM (dense wavelength division multiplexing) | Typically 40 to 96 wavelengths in the C band over one fibre pair, each 100G or 400G | Diverse paths, who owns the optical layer, latency per wavelength |
| MPLS (Multiprotocol Label Switching) L3VPN (Layer 3 VPN) | Any distance, regions and backup links | The carrier's MTU and routing limits, encryption over the top |
| Carrier Ethernet | Layer 2 services between metro sites | MTU and tag handling in the contract, MACsec across the WAN |
| Internet with IPsec | Last resort or small sites | Throughput of the encryption devices, and the MTU after overhead |
Worked case: eight pods in a hall, a second site 40 km away
Each pod is its own EVPN fabric with its own border gateways. A routed super spine or core in the hall joins the pods. Multi-Site gateways join the hall to the second site over two DWDM paths on diverse routes. Forty km of fibre adds about 0.4 ms of round trip time, so synchronous replication still fits, and a fault in one pod or one site stays there.
Stretched Layer 2: no by default
A stretched VLAN makes two sites one broadcast and failure domain. I first ask why: a cluster needing the same address, a heartbeat or a migration wave. If it is really needed, I extend only that VNI, with ARP suppression, storm control at the gateways and no spanning tree across the link, as a time bound exception with an owner and an end date.
BUM between sites
Gateways use ingress replication across the link between sites, because there is no multicast in the WAN. One gateway per VNI is elected to send and receive, so nothing loops. Storm control at the gateways stops a problem in one site becoming an outage in two.
Anycast or vPC gateways
Anycast border gateways (up to six per site) all forward, share one address and host no servers, which suits new pods. A vPC gateway pair can also host legacy Layer 2, which suits migration away from OTV (Overlay Transport Virtualisation).
Cisco ACI
One controller, policy as objects
Cisco ACI (Application Centric Infrastructure) runs Nexus 9000 switches in ACI mode under a cluster of APIC (Application Policy Infrastructure Controller) appliances. The fabric is configured as objects through one API (application programming interface). Traffic is allowed only where a contract says so.
Swipe sideways to see the whole diagram.
A tenant is an admin and policy container, here for the Payments business. Its VRF is a routing table. In an enforced VRF, the default, nothing talks without a contract.
A bridge domain is the Layer 2 forwarding domain, with its subnet. Its gateway exists on every leaf where the bridge domain is in use, much like an anycast gateway.
Endpoint groups (EPGs) hold endpoints that share a policy, classified by VLAN and port, by vCenter port group, or by attributes such as IP address or VM name. An application profile groups an application's EPGs.
A consumer EPG uses a contract that a provider EPG offers. Subjects and filters set the ports. Web reaches App on 8443 and App reaches Database on 1433, enforced in leaf hardware at line rate. Web to Database has no contract, so it is dropped.
An L3Out connects a VRF to the outside on border leaves with BGP, OSPF or static routes. An external EPG classifies outside addresses, here everything, so a contract can let users reach the web tier on 443. Leaves share those outside routes over MP-BGP, with the spines as route reflectors.
Swipe sideways to see the whole diagram.
Endpoint 1, a web server, sends its first packet on leaf 101. The leaf learns its MAC and IP address and which EPG it belongs to.
Leaf 101 reports the endpoint to the spines over COOP (Council of Oracle Protocol). The spines hold the endpoint database for the whole fabric. IS-IS in the infrastructure VRF carries the tunnel addresses underneath.
Leaf 103 has a packet for endpoint 1 but has never seen it. Instead of flooding, it sends the packet to the spine proxy address.
The spine looks up endpoint 1 in its database and forwards the packet to leaf 101. The ACI form of VXLAN carries the source EPG's class ID, so the policy check knows who sent it.
From the reply, leaf 103 learns where endpoint 1 lives and caches it, so later packets go straight to leaf 101. Contracts are checked on the leaves, never on the APICs.
APIC cluster
Three or more APICs hold the policy database in shards with three copies, so a majority is needed to make changes. If every APIC were lost, the fabric would keep forwarding but could not be changed. I place them on different leaves, racks and power feeds. A new switch is found by LLDP (Link Layer Discovery Protocol), gets a tunnel address from the APIC's TEP (tunnel endpoint) pool by DHCP, and joins once I register its serial number with a node ID.
Access policies
A port only carries an EPG's VLAN when the chain is complete: VLAN pool, physical or VMM (virtual machine manager) domain, attachable access entity profile (AAEP), interface policy group, interface profile and switch profile. The EPG must also be tied to that domain. A static binding without it raises a fault and passes no traffic.
Service graphs and PBR
Contracts are stateless filters. For flows that need a real firewall, such as card data, a service graph with policy based redirect (PBR) sends the contract's traffic through a firewall cluster, symmetrically in both directions.
ESGs, vzAny and preferred groups
Endpoint security groups (ESGs) group endpoints across bridge domains in one VRF. vzAny applies one contract to every EPG in a VRF, ideal for DNS (Domain Name System) and NTP (Network Time Protocol). A preferred group lets members talk freely. I treat the last two as governed exceptions.
| ACI object | What it is | The mistake I see most |
|---|---|---|
| Tenant | Admin and policy container. Built in tenants are common, infra and mgmt. | Too many tenants, then messy sharing between them |
| VRF | A routing table, enforced or unenforced | Left unenforced long after the migration ended |
| Bridge domain | Layer 2 domain with subnets and a pervasive gateway | Flooding left on, or subnet flags wrong for export |
| EPG | Endpoints that share policy | Endpoints landing in the wrong EPG through a VLAN mismatch |
| Contract | Subjects and filters between a consumer and a provider | Provider and consumer reversed |
| L3Out | Routed connection to the outside, with an external EPG | External EPG subnet flags for import and export set wrongly |
ACI at scale
Multi-Pod for one zone, Multi-Site for two
ACI joins fabrics in two ways. Multi-Pod stretches one fabric and one APIC cluster across pods. Multi-Site keeps separate fabrics, each with its own APICs, and an orchestrator pushes shared policy to them. What matters is whether one bad change could take out both places.
Swipe sideways to see the whole diagram.
Multi-Pod. One fabric across pods, such as two halls on a campus, joined by an inter pod network (IPN) running OSPF, PIM Bidir for broadcast traffic, DHCP relay and an MTU of at least 9150. Up to 50 ms of round trip time. One APIC cluster means one change domain, and APIC placement decides which side can still change policy if the IPN splits.
Multi-Site. Separate fabrics, each with its own APIC cluster, joined by an inter site network (ISN) with no multicast. Nexus Dashboard Orchestrator pushes tenants, VRFs, bridge domains, EPGs and contracts. Spines translate segment and class IDs between sites. A configuration error stays in one site.
Remote leaf
A pair of leaves at a small site run by the main APICs over a routed WAN. Useful for a small edge or lab, but it stays in the main fabric's change and fault domain, so it is not a substitute for a pod.
VMM integration
APIC creates a distributed switch in vCenter and EPGs appear as port groups. Policy is programmed only where virtual machines land. If NSX also runs, the fabric only sees Geneve, so I decide in one recorded decision where virtual machine policy lives.
Network centric first
Many migrations land with one VLAN equal to one bridge domain equal to one EPG, and an open VRF. Then flow data shows real traffic, and I move application by application to EPGs or ESGs with contracts, starting with regulated systems, to a dated plan so it does not stall.
Choosing
ACI or standard VXLAN EVPN: who will run it?
Both run on the Nexus 9000 family and both carry traffic in VXLAN. The difference is who owns the policy and who builds the control plane. I choose based on the estate, the skills of the team that will run it and how much lock in the organisation will accept.
| Factor | Cisco ACI | NX-OS VXLAN EVPN with Nexus Dashboard |
|---|---|---|
| Who builds the control plane | The switches run IS-IS, COOP and MP-BGP, all set up by APIC | Each switch runs BGP EVPN. The controller generates and checks configuration, and the fabric runs without it. |
| Standards | Cisco only | Open RFCs: 7348, 7432, 8365, 9135 and 9136 |
| Segmentation | Mature contracts in leaf hardware | VRFs, firewalls and ACLs, with group policy on newer releases and hardware |
| Automation | One REST (representational state transfer) API, with Terraform and Ansible | Templates, Terraform, Ansible and network as code |
| Skills | Object model, smaller skills pool | Common BGP and EVPN skills |
| Joining fabrics | Multi-Pod and Multi-Site through the orchestrator | EVPN Multi-Site border gateways |
| Blast radius | One APIC cluster per fabric | A template change can reach many switches, so pipelines need gates |
| Lock in | High. Leaving means a new platform. | Lower, but mixing vendors inside one fabric is still unwise |
Where ACI fits
New, multi tenant estates where the team will adopt the object model and wants allow list policy enforced in the fabric itself.
Where standard EVPN fits
Brownfield NX-OS estates, multi vendor plans, and teams that think in BGP. It also matches how many campus and cloud networks now work.
One fabric per pod
Whichever I choose, I keep each pod its own failure domain, joined by routing or by Multi-Site gateways, and I keep one vendor inside a pod with a standards based boundary between pods.
The hypervisor side of this choice, VMware NSX, has its own page: VMware NSX. The campus version of a VXLAN fabric is on the SD-Access campus fabric page, where LISP (Locator/ID Separation Protocol) replaces BGP EVPN as the control plane.
NDFC and upgrades
Intent in, configuration out, drift reported
Nexus Dashboard Fabric Controller (NDFC), which replaced DCNM (Data Center Network Manager), holds the fabric's intent and generates the configuration each switch should have. It then compares that with what is really running. That comparison is my drift detector.
Swipe sideways to see the whole diagram.
I describe the fabric as data: switch roles, networks, VRFs and interface policies. An existing NX-OS fabric can be imported and read back as intent.
The controller generates the expected configuration for each switch and compares it with what is running.
A preview shows the exact commands and the difference for each switch. That preview is the review evidence for the change ticket.
Deploy pushes the commands in a safe order, VRFs then networks then interfaces, and checks compliance again. Every switch should show in sync.
If someone changes a switch by hand, it shows as out of sync and the next deploy reverts it. My rule: no changes outside the controller, and freeform policies for anything it cannot model.
Upgrading a fabric safely
- Check the target against the recommended releases and the compatibility matrix.
- Run the pre upgrade checks and export the configuration.
- Stage the image, then upgrade in groups: odd and even, so a vPC pair, two spines or two route reflectors never go together.
- Drain each switch first with maintenance mode and BGP graceful shutdown.
- Check health, route counts and endpoint counts between groups, and stop automatically on a failure.
In ACI the APICs upgrade first, one at a time, while the fabric keeps forwarding. Then the switches upgrade in odd and even maintenance groups, after a pre upgrade validation and a configuration export.
Checks I automate
- Before and after snapshots with pyATS, compared automatically.
- Model based checks with Batfish before a change reaches a switch.
- Route counts per VNI and VRF against the source of truth.
- Full size pings between every VTEP pair after any MTU or link change.
- Nexus Dashboard or ACI pre change analysis where available.
The pipeline itself is on the network automation page.
Migration
Moving a live VLAN without a big bang
Most migrations into a new fabric extend a VLAN across a bridge, move the hosts, then move the gateway last. Each VLAN gets its own window, its own checks and one rehearsed way back.
Swipe sideways to see the whole diagram.
Weeks before, discovery lists every VLAN, gateway, dependency and silent host. The fabric side is built and checked with its gateway shut, a canary VLAN has crossed the bridge, and the method and rollback are approved. In the window, I take a snapshot first.
A trunk from the legacy core lands on a vPC pair of border leaves, carrying only the VLANs in scope, with storm control and planned spanning tree handling. VLAN 110 maps to VNI 10110, and I confirm MACs appear on both sides.
Servers or virtual machines move to fabric leaves. The gateway is still on the legacy core, so routed traffic crosses the bridge for now.
I shut the legacy gateway interface, then bring up the fabric's anycast gateway with the same IP. Either it reuses the old virtual MAC, or it sends gratuitous ARP. Never both gateways up at once.
Checks run: type 2 routes at the expected count, ARP entries on the leaves, reachability and application probes. If they have not passed by T+75 of a 120 minute window, I roll back in one rehearsed action.
Days later, when the legacy side shows no MACs, no flows and no DNS references, the trunk is removed, the old interface deleted and the records updated.
| Pattern | When it fits | Traps I check for |
|---|---|---|
| Extend and move the gateway | Most VLANs | Spanning tree root on the trunk, duplicate gateway IP, HSRP (Hot Standby Router Protocol) virtual MAC against the anycast MAC, MTU, silent hosts |
| Re address | Small, well understood applications | Hard coded addresses, firewall rules, DNS time to live, NAT (network address translation) |
| Firewall inserted segments | Zones with a firewall between them | VRF hand off at border leaves, asymmetric paths through active firewalls, rule accuracy |
| Multi-Site stitching | Moving between data centres | MTU across the link, one forwarder per VNI, matching VNIs, split brain |
| Special cases | Cluster heartbeats, iSCSI (Internet Small Computer Systems Interface) or NFS (Network File System), static neighbours, multicast applications | Each gets a named exception, its own window and its own checks |
Troubleshooting
Prove each layer, from the bottom up
A fault low in the stack shows up as a symptom in every layer above it, so I always start at the underlay. Pick a symptom to see which layer usually holds the cause, then open the matching checks below.
Swipe sideways to see the whole diagram.
Start at the underlay. Route to the remote VTEP through every spine, a full size ping between loopbacks, then the NVE interface, peers and VNIs.
The route is there but unused. Compare route targets on both leaves, check the L3VNI and the VRF association, and check that extended communities are sent.
Small works, big fails. Ping with the do not fragment bit at rising sizes until it fails, then check every link, SVI and circuit on that path, backup paths included.
Policy last, but often the cause. Read the health scores and faults, find both endpoints in the endpoint tracker, then check the contract and its direction.
A gateway that should have stepped back. Check link and fabric tracking on the border gateways, the shared address advertisement and the restore delay.
Hosts in the same VNI cannot reach each other
- Underlay route to the remote VTEP through every spine:
show ip route 10.254.2.14. - Full size test between loopbacks:
ping 10.254.2.14 source 10.254.1.11 df-bit packet-size 9000. - VTEP state and source loopback:
show nve interface nve1 detail. - Peers up and learned through the control plane:
show nve peers. - VNI state, type and replication mode:
show nve vniandshow nve vni ingress-replication. - Remote MAC pointing at the right VTEP:
show mac address-table vlan 110. - Test through the overlay itself:
ping nveandtraceroute nve.
Fix: correct the MTU, advertise the VTEP loopback, align VLAN to VNI mapping and replication mode from the template. Prevention: one template from the source of truth, and full size pings between every VTEP pair during acceptance.
EVPN route present in BGP but not installed
- Routes for the VNI with RD, RT, next hop and router MAC:
show bgp l2vpn evpn vni-id 10110. - What the leaf learned:
show l2route evpn mac allandshow l2route evpn mac-ip all detail. - Flood list membership from type 3 routes:
show l2route evpn imet all. - VRF to L3VNI association:
show nve vrf, thenshow ip route vrf PROD. - What the leaf answers locally:
show ip arp suppression-cache detail.
Fix: set matching route targets, or rewrite them in an eBGP overlay with retain route-target all on the spines; add send-community extended; bring up the L3VNI with its transit VLAN and member vni 50001 associate-vrf. Prevention: one route target scheme in the standard, and manual route targets at any multi vendor boundary.
Small packets work, large transfers hang
- Ping with the do not fragment bit between VTEP loopbacks at rising sizes, and host to host inside each VNI.
- Check the MTU on physical ports, SVIs and the L3VNI SVI, not only on the links.
- Test the backup path between sites too. It may only carry traffic after a failure.
- In ACI Multi-Pod, check the inter pod network MTU of at least 9150.
- Look for ICMP "fragmentation needed" being blocked by a firewall between sites.
Fix: 9216 on every fabric, super spine and inter site link and SVI. TCP (Transmission Control Protocol) MSS (maximum segment size) adjustment at any 1500 byte border. Prevention: the carrier's MTU written into the contract, and full size tests after every link change.
An ACI endpoint cannot reach a server
- Health scores, 0 to 100, at fabric, pod, switch, tenant and object level, then faults by severity.
- Endpoint tracker: where each endpoint is now and where it has been, including duplicates and moves. On the leaf,
show endpoint ip 10.20.1.21, and on a spine,show coop internal info ip-dbfor the copy the spines hold. - Zoning rules on the leaf with
show zoning-rule, and contract deny logs, to see whether policy dropped it. - Object browser or
moqueryagainst the object model, and the audit log for recent changes. - Atomic counters and SPAN (Switched Port Analyzer) when the drop is on the path rather than in policy.
Fix: add the contract or correct its direction, fix the static binding or VMM VLAN, correct the L3Out flags. Prevention: contract design reviewed by the design authority, policy as code and pre change analysis.
Inter site traffic black holed after a link fault
- Which gateway lost its links between sites, and is it still advertising the shared Multi-Site address?
- Link and fabric tracking state on that gateway, set with
evpn multisite dci-trackingandevpn multisite fabric-tracking. - Restore delay timers, so a recovering gateway does not attract traffic too soon.
- Broadcast rates at the gateways and which VNIs are extended, in case a storm is crossing.
Fix: enable tracking so the gateway withdraws its address, and set a restore delay. Prevention: two diverse paths each sized for the full load, BFD, and tests that pull a link and fail a gateway.
| Platform | Command or tool | What it tells me |
|---|---|---|
| NX-OS | show nve peers | Remote VTEPs, state up, learned through the control plane |
| NX-OS | show nve vni | Each VNI's state, Layer 2 or Layer 3, and how broadcast traffic is replicated |
| NX-OS | show bgp l2vpn evpn summary | Overlay sessions up and prefixes received |
| NX-OS | show l2route evpn mac all | Local and BGP learned MACs |
| NX-OS | show fabric forwarding ip local-host-db vrf PROD | Hosts the leaf learned locally |
| NX-OS | show vpc brief | Peer link, keepalive and consistency for a leaf pair |
| ACI | Health scores, faults, endpoint tracker | Where to look first, and where an endpoint really is |
| ACI | moquery, audit log, atomic counters | The object model as it is, who changed it, and where packets were lost |
| ACI | acidiag fnvread, show endpoint ip, show zoning-rule | Discovered switches, where an endpoint is learned, and the rules a leaf enforces |
Engagement
How I run a fabric programme, from brownfield import to refresh
A new fabric is a platform other teams will build on for years. I start with the workloads and the people who run them, measure the estate I am replacing, then design, prove and hand over a fabric that operations can change safely.
Swipe sideways to see the whole diagram.
I agree why the fabric is needed: growth, segmentation, faster change, a refresh or a merger. I write the goals as measures, such as time to deliver a new network or tenant, and agree who decides and who operates.
Functional requirements cover tenants, segments, gateways, external links and services such as firewalls and load balancers. Non functional requirements cover scale, convergence, MTU, availability and how fast a change can be made, each with a number.
I pull the facts from the switches themselves: VLANs, gateways, MAC and ARP tables, routing and spanning tree. Flow data and firewall logs show who talks to whom. Silent hosts, clusters and multicast applications become named special cases.
I design one standard pod: underlay protocol, AS numbers, route reflectors, VNI and VRF numbering, MTU, border hand off and links to other sites. The high level and low level designs go through the design authority, and each choice is recorded.
I build from templates, then test failure cases: a spine, a leaf, a vPC peer link, a border link and a route reflector. Full size pings with the do not fragment bit prove the MTU on every path before the first workload moves.
Operations get runbooks, a health check per layer, alerts that mean something and templates for the changes they make every week. Hypercare ends when the agreed exit criteria are met and drift reports are clean.
I keep the fabric on recommended releases, upgrade in groups so a vPC pair or two spines never go together, and track hardware tables, port use and end of life dates so growth and refresh are planned.
Functional requirements
What it must do.
- Carry any tenant segment on any leaf, with the gateway on the first leaf a host meets.
- Keep tenants apart in their own VRFs, with traffic between them passing a firewall.
- Connect the fabric to the WAN, internet and other sites at border leaves or border gateways.
- Insert firewalls and load balancers for the flows that need them.
- Let virtual machines move between racks without changing address or gateway.
- Join the old network during migration, with a tested way back.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Scale
- Leaves, VNIs, VRFs and host routes per pod, with headroom against the verified scalability guide.
- Availability
- No single device or link takes down a tenant. A spine, leaf or vPC peer can fail without loss beyond a few hundred milliseconds.
- Performance
- Oversubscription agreed per leaf type, for example 3:1 for general compute.
- MTU
- 9216 on every fabric link and tenant hosts at 1500 or 9000, proved end to end.
- Security
- Underlay unreachable from tenants, encryption on links that leave a controlled space, and logs sent to the SIEM (security information and event management).
- Change
- A new tenant network delivered from a template in hours, with drift reported automatically.
- Recovery
- Configuration backed up daily, and a rebuilt leaf back in service from its template within an hour.
These are typical starting targets for a fabric. Each one is agreed with the owners and proved in the factory test.
Who I talk to
- Network, data centre and security teams
- VMware, storage and container platform owners
- Owners of applications with special needs: clusters, multicast, fixed addresses
- Firewall and load balancer teams
- Facilities, for racks, power and cabling
How I find out
- Workshops by workload type and tenant
- Reading switch tables: VLANs, MAC, ARP, routing and spanning tree
- Flow analysis to find who talks to whom
- Walking the halls to confirm racks, cabling and optics
- Reviewing incidents and changes from the last year
Discovery tools
| Tool or source | What it tells me |
|---|---|
Switch configuration and show output | VLANs, gateways, HSRP, spanning tree roots and port channels as they really are. |
| NetFlow, sFlow or Nexus Dashboard flow telemetry | Traffic between applications, so segments and contracts follow real flows. |
| Firewall logs and rule bases | Which flows cross zones today and which rules are never hit. |
| vCenter inventory and RVTools | Virtual machines, port groups and VLANs per cluster. |
| IP address management and DNS | Subnets, owners and the addresses that are really in use. |
| Nexus Dashboard Fabric Controller brownfield import | An existing NX-OS fabric read back as intent, ready to compare with the design. |
Fabric faults sit in one of five layers: the underlay, the tunnel endpoints, the EVPN control plane, the forwarding tables or the policy. I prove each layer in that order, because a fault in a lower layer shows up as a symptom in every layer above it.
My approach
- Scope. One host, one segment, one leaf or everything? Bridged or routed? Since when, and what changed?
- Underlay. Route to the remote VTEP through every spine, then a full size ping with the do not fragment bit between VTEP loopbacks.
- Tunnel endpoints. NVE interface up, peers up and learned through the control plane, VNI state and replication mode.
- Control plane. EVPN sessions up, the type 2, 3 or 5 route present, and the route target imported where it should be.
- Forwarding. MAC table, ARP suppression cache and host routes in the VRF on the receiving leaf.
- Policy and record. ACLs, contracts or firewall rules last. Fix through the controller or a template, confirm with the same checks, and record the cause.
Faults I see most
- MTU too small on one link, so large transfers hang
- Route target mismatch: the route arrives but nobody imports it
send-community extendedmissing, so routes lose their communities- VLAN to VNI mapping different on two leaves
- Silent hosts that never send, so no type 2 route exists
- In ACI, a contract with provider and consumer reversed
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Optics, errors and FEC (forward error correction) counters on fabric links with show interface counters errors. |
| 2 · Data link | VLAN to VNI mapping, vPC consistency with show vpc consistency-parameters global, MAC entries with show mac address-table vlan 110. |
| 3 · Network | Underlay routes with show ip route, VTEP to VTEP ping 10.254.2.14 source 10.254.1.11 df-bit packet-size 9000, EVPN with show bgp l2vpn evpn summary. |
| 4 · Transport | NVE peers and VNIs with show nve peers and show nve vni, UDP 4789 not filtered between VTEPs. |
| 5 to 7 · Application | The flow itself: contract or ACL (access control list) hits, firewall logs, load balancer health and what the user sees. |
Worked fabric faults are in the troubleshooting section above, and my general method is on the troubleshooting page.
A fabric is only as good as the team that changes it. I agree the support model before the first workload moves.
Service model
Who owns the fabric, the controller, the firewalls and the platforms, with escalation paths to the vendor.
Health checks
A one page check per layer: underlay, NVE, EVPN, tables and policy, run before and after every change.
Telemetry
Interface errors, route counts, VTEP peers and table use streamed to monitoring, with alerts tuned in hypercare.
Change
Every change from a template or the controller, checked in a lab first, with a rollback point and a post check.
Drift
The controller or a pipeline compares running configuration with intent every day and reports the difference.
Problem management
A repeat fault gets a root cause, and the fix goes into the template or the controller policy so it cannot return.
I plan the fabric's life from day one, using the vendor's recommended releases and end of life notices.
- Plan. One approved software release per platform, held as data, with a policy on how far behind the latest we may sit.
- Upgrade. Validate first, then upgrade in groups: one vPC peer, one spine and one route reflector at a time.
- Patch. Cisco Product Security Incident Response Team (PSIRT) advisories for NX-OS and APIC are triaged by severity and exposure. Critical fixes go through an emergency change, the rest join the next wave.
- Review. Quarterly reviews of hardware table use, port capacity, oversubscription and growth.
- Retire. Hardware leaves only when no MACs, routes or flows remain, and every record is closed.
- Refresh. Pods are refreshed in the same pattern they were built, before the hardware leaves support.
Projects
Where I have built on these fabrics
Data centre fabrics across telecoms, industrial software, banking and consultancy work.
Tier 3 data centre on Cisco ACI and Nexus
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI and Nexus, including provider IP space and AS numbers managed with RIPE NCC.
Nexus transformation and AI ready data centres
Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN (software defined WAN) and SD-Access, and designed AI ready data centre infrastructure for high density compute clusters.
Data centre transitions at 99.99% uptime
Safeguarded over £200 million of transactional services through data centre transitions, holding 99.99% uptime throughout.
Software defined networking on Nexus
Cloud, SDN, disaster recovery and security architecture on Nexus, Viptela, Palo Alto, F5 and Aruba for the firm's enterprise clients.
Data centre refresh under MiFID II
Audit, high and low level design and implementation of local area network (LAN), WAN, SD-WAN and data centre infrastructure, replacing end of life equipment with no service outages. The work ran under the Markets in Financial Instruments Directive (MiFID II).
Routing and data centre depth
CCIE Routing and Switching written exam passed, CCNP Enterprise, CCNP Routing and Switching and CCDP, with VXLAN and Cisco Nexus and ACI among my core skills.
The physical side of a fabric, from leaf and spine sizing to out of band management, is on the data centre networks page. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.