Predictable
Every path between racks has the same number of hops, so performance does not depend on where a workload lands.
Data centre networks
Data centre networks designed to fail safely
I design the networks inside data centres: fabrics that keep working when parts fail, grow by pod, stay reachable when production is down, and keep security zones apart. Step through the diagrams, or break a fabric yourself and watch what survives.
Overview
From three tiers to leaf and spine
Data centre traffic changed direction. Most of it now flows between servers, not out to users, and the network design had to change with it.
Swipe sideways to see the whole diagram.
Three tier. The classic design was built for traffic heading out to users. Spanning tree blocks half the uplinks to stop loops, and traffic between servers in different aggregation blocks climbs through the core and back down. As traffic between servers grew, it became the bottleneck.
Leaf and spine. Every leaf connects to every spine. Any two racks are two hops apart, latency is predictable, and every link carries traffic through equal cost routing. To grow, I add a leaf for more ports or a spine for more bandwidth.
Scale by adding
Ports and bandwidth grow in known steps. The design for 4 leaves is the design for 40.
Routed, not bridged
The underlay is routed, with no spanning tree inside the fabric. A VXLAN EVPN overlay (Virtual Extensible LAN, with Ethernet VPN as the control plane) carries the tenants, as shown on the SD-WAN and SDN page.
The building under the fabric, from capacity and racks to power, cooling and cabling, has its own page. Data centre facilities
Failure domains
Break it and watch what survives
A failure domain is everything that fails together. Select any switch to fail it, or pick a scenario, and see how far the damage spreads.
Swipe sideways to see the whole diagram.
ServiceUp
Paths16 of 16
Capacity100%
Everything is healthy. There are 16 equal cost paths between server A and server B.
Tip: select a spine or a leaf in the diagram, or use Tab and Enter, to fail it or bring it back.
How I set failure domains
- Know what fails together. Domains nest: device, rack, row, pod, hall, site. No important service keeps all its copies in one of them.
- Look for hidden sharing. Two "independent" pods on one uninterruptible power supply (UPS) block, or two fibre routes in one duct, are one failure domain on paper only.
- Dual home what matters. Servers connect to a leaf pair and to A and B power, so one device or one feed never takes them down.
- Keep spare that is real. N+1 (one spare beyond what is needed) only works if the spare capacity is free. A cluster running at 95 percent has nothing left to absorb a failure.
- Limit the logical blast radius too. One controller or one automation run that touches every pod is a failure domain. I stage changes pod by pod.
- Test it, do not assume it. Acceptance includes failing a spine and pulling a power feed while the service is under load.
What users see when part of a pod fails
| What fails | What the fabric does | What users see | How I design for it |
|---|---|---|---|
| One link | The route is withdrawn and traffic moves to the other paths within a second, helped by BFD (bidirectional forwarding detection). | Nothing. | Alert on lost capacity, not just on link state. |
| One spine | Equal cost multipath (ECMP) drops to the remaining spines. With four spines, a quarter of the bandwidth between leaves is lost. | Nothing, if the fabric was sized for N minus 1. | At least four spines and two route reflectors. |
| One leaf of a pair | Dual homed servers carry on over their other link. The partner owns the shared tunnel endpoint. | Nothing for dual homed servers. Single homed servers go down. | Dual home everything, and keep each leaf under half loaded. |
| Both leaves of a pair | The racks behind them are cut off. | Workloads restart elsewhere if the cluster allows it. | Spread every cluster across racks and leaf pairs. |
| vPC peer link, keepalive up | The secondary suspends its vPC ports and switched virtual interfaces (SVIs). | Single attached servers on the secondary lose service. | No orphan ports, and a peer link split across line cards. |
| Both border leaves | East to west traffic carries on, but the pod loses its way out. | Users outside the pod cannot reach it. | Border pairs in separate racks, power and cooling zones. |
| All route reflectors | Every EVPN session drops. Graceful restart may keep the learned routes for about two minutes, then they are withdrawn and traffic between leaves stops. | A short pause, then a wide outage for anything that crosses the fabric. | Two or more reflectors on different spines, racks and power, never all in one failure domain, with BFD so a single loss is noticed fast. |
| One super spine plane | Each pod loses a quarter of its bandwidth to other pods. No pod loses connectivity. | Nothing, unless traffic between pods was near its limit. | Independent planes, changed one at a time. |
| The fabric controller cluster | Forwarding continues on the last good policy. Changes stop. | Nothing, until a change is needed. | Three nodes across racks, backed up, with a tested rebuild. |
| A grey failure | A link drops a few packets but stays up, so routing never reacts. | Slow or failing transfers. | Telemetry on error counters, BFD, and draining the suspect device. |
| One pod | Its services fail over to their partner pod in another hall. | A short disruption within the agreed tolerance. | Every important service runs in two pods. |
Indicative behaviour. The real figures depend on timers, platform and design, so I prove each row in acceptance testing.
Failure domains, from a disk to a region
| Layer | What can fail | How I contain it |
|---|---|---|
| Component | Power supply, fan, disk, optic, network card | Redundancy inside the device, and dual homing |
| Device | A leaf, a server, a controller | Pairs, N+1 clusters and dual controllers |
| Rack | A top of rack pair, a rack power trip, a local fire | Clusters striped across racks, with A and B power |
| Row or aisle | A containment cooling fault, a busway fault | Services spread across pods |
| Pod | A control plane problem, a bad change, a firmware bug | Every important service in two pods, and staged change |
| Hall | A UPS block, the cooling plant, a suppression discharge | Paired pods in different halls |
| Site | Utility and generator together, a flood, lost access | A second site, or metro active and active |
| Region | The grid, a legal or geopolitical event | Asynchronous recovery to another region, with data residency checked |
| Hidden and logical | One UPS block, one duct, one controller, one pipeline, one identity system | Find the sharing and write its blast radius into the design |
Correlated failure
Redundant copies can fail for the same reason at the same time. In 2019 a firmware bug made some enterprise SAS solid state drives fail after 32,768 hours of power on time, so drives installed together failed together. In July 2024 one faulty security software update crashed about 8.5 million Windows machines. I stagger firmware and changes, mix hardware batches across pairs and never change both halves of a pair together.
Active and active, or active and standby
Active and active fails over almost instantly and proves both sides every day, but each side must be able to carry everything. Two sites each running at 70 percent cannot absorb each other. Active and standby is simpler for stateful systems and often cheaper to license, but the standby must be tested. Leaf pairs, ECMP and anycast gateways are active and active. Firewalls and some databases usually run active and standby.
Quorum and witnesses
A cluster splits its brain when both halves think they are in charge. I use odd numbers of members, three or five, spread across failure domains, and put any witness or tie breaker in a third domain, never in either half's site. The data plane forwards, the control plane decides and the management plane configures, so losing a controller should pause changes, not traffic.
Scale out
Grow by pod, not by redesign
A pod is a standard block of leaves, spines and racks. Data centres grow by adding pods, joining them through a super spine, and joining sites at their border. How each pod moves from plan to refresh is on the design authority page.
Swipe sideways to see the whole diagram.
I size a pod to a physical unit, usually two rows sharing a contained aisle, as in the data hall layout. Its leaves, spines, racks, power and cooling are designed together and built the same way every time.
New pods join through a super spine. Each pod is a self contained block with its own spines, so growth adds capacity without touching what is already running.
Pods that back each other up sit in different halls, on different UPS blocks and cooling plant. A physical event in one hall cannot take out both copies of a service.
A second data centre has its own fabric. Border gateways join the sites with EVPN multi site, so each site fails on its own and only the routes that need to cross do so.
When something goes wrong in pod 2, the boundaries keep it there. That is the real point of pods. They make growth simple, and they limit how far one problem can spread.
Fabric maths
The numbers behind every pod
Oversubscription, spine count and port budgets decide what a fabric can carry and how it behaves when a spine fails. I work them out before anything is ordered. Try the numbers yourself, then follow a worked design for 64 racks.
Swipe sideways to see the whole diagram.
Server port speed
Uplink speed
Test it
| Links to each spine | 2 |
|---|---|
| Oversubscription | 1.5 : 1 |
| With one spine down | 2 : 1 (75% of uplink left) |
| Most leaves the spines allow | 32 |
| Server ports in the fabric | 1,536 |
Low oversubscription. Suits busy virtual machine and big data pods. Storage leaves usually sit closer to 1 : 1.
The formulas I use
- Oversubscription. Server facing bandwidth divided by uplink bandwidth. 48 × 25G is 1.2 Tb/s. With 8 × 100G up it is 1.5 to 1, with 6 × 100G it is 2 to 1 and with 4 × 100G it is 3 to 1.
- After a spine failure. Uplink capacity falls to (N minus 1) divided by N when the uplinks divide evenly. With two spines, one failure halves it, so each must run below 50 percent. With four, 75 percent is left. With eight, 87.5 percent. When they do not divide evenly, I plan for losing the spine that carries the most links.
- Most spines. Uplinks per leaf divided by the links from each leaf to each spine. Eight uplinks allow up to eight spines.
- Most leaves. Ports per spine divided by the links from each leaf to each spine. A 64 × 400G spine broken out to 256 × 100G takes up to 256 leaves, less the ports kept for border leaves, the super spine and growth.
- Count what is used. Only traffic that leaves the leaf matters, and only ports really cabled count. A leaf where power allows 16 servers uses 16 of its 48 ports, so its real ratio is much lower.
- A second ratio. Spine to super spine is a second oversubscription point. I size it from how much traffic really crosses between pods.
- Control plane and tables. Port maths is only half the limit. Each leaf also has ceilings for tunnel endpoint (VTEP) peers, VXLAN network identifiers (VNIs) and routing instances (VRFs), media access control (MAC) addresses and host routes. I check every figure against the vendor's verified scalability guide for the exact release, and keep headroom for growth.
Ratios I start from
| Pod or role | Typical ratio at the leaf | Why |
|---|---|---|
| General virtual machines | About 3 : 1 | Most traffic is bursty and not all servers peak together. |
| Storage leaves | Close to 1 : 1 | Storage traffic is heavy and sensitive to loss and delay. |
| Artificial intelligence (AI) front end and big data | Close to 1 : 1, or 2 : 1 at most | Large flows between many nodes at once. |
| Graphics processing unit (GPU) back end | Exactly 1 : 1 | Training stalls if any link is short of bandwidth. |
| Between pods | 2 : 1 to 3 : 1 | Most traffic stays inside its pod. |
Indicative starting points. I record the chosen ratio for each pod type in the cost model, because it drives the number of spines and optics.
A worked design for 64 racks
Swipe sideways to see the whole diagram.
Before I draw anything, I ask how many servers each rack holds, which network cards they use and how storage connects. I also ask how much traffic stays east to west, how many kilowatts each rack may draw and how big a failure the business can accept. Here: 64 racks, every server dual homed at 25G, and 1.5 to 1 at the leaf.
The 64 racks become four pods of 16 racks, two rows of eight around one contained aisle. Each pod is its own failure and change domain, and lines up with the hall's power and cooling blocks.
Each rack has a leaf pair, so a pod has 32 leaves. A leaf gives 48 × 25G to servers, 1.2 Tb/s, and 8 × 100G up, 0.8 Tb/s, with two links to each of four spines. That is 1.5 to 1 at the leaf.
Each spine has 32 × 400G ports. Sixteen are broken out into 64 × 100G for the leaves, eight go up to the super spine and eight stay spare. Down is 6.4 Tb/s and up is 3.2 Tb/s, so 2 to 1 between pods. That is fine because most traffic stays inside a pod.
Spine 1 of every pod joins plane 1, spine 2 joins plane 2, and so on. Four pods × four spines × eight links is 128 links of 400G. Four 64 port super spines use 32 ports each, which leaves room for four more pods.
The alternative is one fabric of 128 leaves on eight large spines. It has fewer layers, but every leaf shares the same spines, control plane and change windows. I choose pods, because a fault or a bad change then stays inside 16 racks.
Pod design
Every leaf has a job
Inside a pod every switch is the same hardware, but each leaf pair is built from a role template: compute, storage, service, border, multi site gateway or GPU. The role sets its oversubscription, buffers, policy and blast radius. Select a role to see the traffic it carries.
Swipe sideways to see the whole diagram.
Compute leaves. Each compute rack has a leaf pair. Every leaf is a VXLAN tunnel endpoint (VTEP) and the anycast gateway for its servers, so a virtual machine reaches its storage across one spine.
Storage leaves. Arrays sit on their own leaf pair with lower oversubscription, and deeper buffers where many servers read at once. Explicit congestion notification (ECN) is tuned for storage traffic.
Service leaves. Firewalls, load balancers and intrusion prevention attach here. Traffic between two VRFs leaves the compute leaf, is inspected on the service leaf, then comes back.
Border leaves. The pod's front door to the WAN, the core and the internet edge. I hand off per VRF, summarise the pod's prefixes outwards and accept only agreed routes inwards.
Multi site border gateways. They join this fabric to the other data centre over the data centre interconnect (DCI), so only the routes that must cross do so. The detail is on the VXLAN EVPN and ACI page.
GPU leaves. GPU servers have a front end on the normal fabric and a separate back end for training traffic, rail optimised and non blocking. That is on the AI infrastructure page.
What the spine does, and what it never does
Transport only
Spines route the outer IP header between tunnel endpoints. They hold no tenant VRFs, no MAC addresses and no segments. They often act as EVPN route reflectors and, where multicast is used, as anycast rendezvous points.
Nothing attaches to a spine
A firewall, load balancer or WAN link on a spine makes that spine special. Traffic to the service must use it, every spine upgrade becomes an outage and the two hop rule breaks. Services go on service leaves and external links on border leaves.
Shallow or deep buffers
Most leaves and spines use shallow, on chip buffers of tens of megabytes, with ECMP and ECN. I use deep buffer switches, with gigabytes off chip, at the border, the DCI and the WAN edge, where speeds change and bursts arrive. Deep buffers add delay when they fill, so I choose by role.
The underlay
The underlay is the routed network between leaves and spines. The usual choices are external Border Gateway Protocol (eBGP), Open Shortest Path First (OSPF) and Intermediate System to Intermediate System (IS-IS). Request for Comments (RFC) 7938 describes the eBGP design for large data centres.
Swipe sideways to see the whole diagram.
eBGP underlay. Following RFC 7938, each leaf, or leaf pair, gets its own autonomous system number (ASN) and the spines share one. Border Gateway Protocol (BGP) then carries the loopbacks with no flooding, clear policy and one protocol from end to end. It needs automation and a good ASN plan.
Why the spines share an ASN. A route that went up through spine 1 and down to leaf 2 would come back up to spine 2 with spine 1's ASN already in its path. Spine 2 sees its own ASN and rejects it, so the leaf is never used as a transit and BGP does not hunt through failed paths.
OSPF or IS-IS. Open Shortest Path First (OSPF) area 0, or Intermediate System to Intermediate System (IS-IS) level 2, on point to point links is simple and familiar. The leaves then run internal BGP for EVPN to the spines as route reflectors. Flooding grows with the fabric, which is fine for one pod.
| Underlay | Strengths | Watch outs |
|---|---|---|
| eBGP (RFC 7938) | Scales to large fabrics, no flooding, clear policy, one protocol end to end, works across vendors. | Needs automation and an ASN plan, multipath relax and BFD. |
| OSPF area 0 | Simple and familiar to most operations teams. | Flooding grows with the fabric. iBGP for EVPN rides on top. |
| IS-IS level 2 | Runs directly over layer 2 and is extensible. Cisco ACI also uses IS-IS for its own underlay. | Less familiar, so skills matter. |
I decide with the operations team that will run it, and record the choice and the reasons in a decision record.
Shared, pod local or both
| Where it lives | Services | Why |
|---|---|---|
| Shared across pods | Network: the super spine, the border to the WAN and DCI, the internet edge and perimeter firewalls, and the out of band core. Operations: jump hosts and privileged access, IP address management (IPAM), the configuration management database (CMDB), security information and event management (SIEM) and the backup vault. | One instance serves every pod, so it gets the most conservative design and the slowest change. |
| Inside each pod | Spines and leaves, border and service leaves, compute and storage, the pod management cluster, pod out of band switches and console servers, telemetry collectors, spares | It fails, changes and retires with the pod. |
| Both | Domain name system (DNS) resolvers on anycast addresses, Network Time Protocol (NTP) and Precision Time Protocol (PTP) time, domain controllers per site, key management across sites, local dynamic host configuration protocol (DHCP) relay | Local copies keep a pod running if the shared copy is out of reach. |
Naming and addressing
Pod names and addresses, generated from the pod number
Names and address blocks follow from a few numbers, so nobody invents them by hand. IP address management (IPAM), in NetBox or Infoblox, holds the result as the source of truth. The scheme below is an example.
| Leaf name | LDN1-H2-P03-LEAF-R05A |
|---|---|
| Pod infrastructure block | 10.19.0.0/16, one route at the super spine |
| Router IDs and BGP (loopback 0) | 10.19.0.0/24 |
| Tunnel endpoints (loopback 1) | 10.19.1.0/24 |
| Point to point links (/31 each) | 10.19.8.0/22 |
| Out of band management | 10.19.4.0/23 |
| Spine ASN, leaf pair ASN | 4201003000, 4201003005 |
| IPv6 for the pod | 2001:db8:1:300::/56, from the site /48 2001:db8:1::/48 |
Each site has a /12 from the 10.0.0.0/8 private range, so site 1 holds 10.16.0.0/12 and room for 15 pods, with the first /16 kept for site shared services. The ASN uses the private four byte range 4200000000 to 4294967294, encoding site, pod and rack. The IPv6 prefix uses the documentation range 2001:db8::/32. CMDB means configuration management database and SIEM means security information and event management.
Platforms I compare
| Platform | Typical leaves and spines | Run by |
|---|---|---|
| Cisco Nexus 9000 | 9300 family leaves, 400G GX2 class spines | NX-OS with VXLAN EVPN and Nexus Dashboard, or ACI mode with the APIC controllers |
| Arista | 7050X leaves, 7060X shallow buffer or 7280R and 7800R deep buffer switches | EOS with CloudVision |
| Dell PowerSwitch | 25G leaves and 400G spines | Enterprise SONiC or OS10 |
I compare EVPN features in the certified release (multihoming, type 5 routes, Multi-Site), automation, telemetry, support, lead times, power per port and cost per usable port. One vendor per fabric, joined to others at the border or the DCI gateway, proven in a lab.
Nexus
Two switches, one logical peer
Cisco Nexus has been at the core of many data centres I have worked in, from Manx Telecom and Civica to my current work at XIDA. Virtual port channel (vPC) is the feature that matters most day to day, and the one most often misunderstood when something fails.
Swipe sideways to see the whole diagram.
Two Nexus switches present a single port channel to the server. Both links forward traffic, so nothing is left idle by spanning tree.
The peer link keeps MAC (hardware address) and state tables in step, and carries traffic for anything attached to only one switch. The keepalive is a separate heartbeat, usually over the management network, used to tell a dead peer from a broken link.
If Nexus 2 fails, the server simply loses one member of its port channel and carries on over Nexus 1. No routing change, no outage.
If only the peer link fails, the keepalive proves both switches are alive. The secondary then suspends its vPC ports, so traffic stays on one consistent path instead of two switches acting alone. This is why I never run the keepalive over the peer link.
Virtual device contexts
On the Nexus 7000 family, one physical switch can be split into virtual device contexts (VDCs): separate logical switches, each with its own configuration and processes. They still share the chassis, supervisors and power, so they are not separate hardware failure domains. That range is now end of sale, so VDCs mostly appear in estates being migrated.
Overlay Transport Virtualisation (OTV)
Layer 2 extended between data centres over a routed core, with loop protection built in. Today I would use VXLAN EVPN multi site for the same job.
FabricPath
An earlier multipath layer 2 fabric that removed spanning tree inside the data centre. VXLAN EVPN has taken over that role, but the design lessons carry across.
The parts of a vPC pair
| Part | What it does | Defaults and rules |
|---|---|---|
| vPC domain | Joins two switches into one logical partner for link aggregation (LACP). | Domain ID 1 to 1000. Pairs that connect back to back need different IDs. |
| Peer link | A port channel trunk that carries Cisco Fabric Services (CFS) sync, flooding, orphan traffic and failure traffic. | At least two 10G or faster links, on different line cards, carrying vPC VLANs only. |
| Peer keepalive | A separate layer 3 heartbeat that tells a dead peer from a broken link. | Over mgmt0 in the management VRF, user datagram protocol (UDP) port 3200, every second, five second timeout. Never over the peer link. |
| Member port | A port channel to a downstream device, with the same vPC number on both peers. | The downstream device just runs normal LACP. |
| Orphan port | A port in a vPC VLAN on one peer only. | The most exposed port in the design. Avoid it, or use orphan port suspend. |
| Roles | Primary and secondary decide who suspends on a peer link failure. | Lower role priority wins, default 32667, then lower system MAC. Roles do not preempt. |
| vPC system MAC | Both peers use it as their LACP system ID. | 00:23:04:ee:be followed by the domain ID in hexadecimal. |
The loop prevention rule
Swipe sideways to see the whole diagram.
Server A sends a broadcast, such as an address resolution protocol (ARP) request. The port channel hashes it onto its link to Nexus 1.
Nexus 1 floods it out of its own member port to server B, and across the peer link so that Nexus 2 can reach anything only it can reach.
A frame that arrives over the peer link is never sent out of a vPC member port if the other peer's member for that vPC is up. Nexus 1 has already delivered to B, so Nexus 2 sends it only to orphan C. This rule is why routing over a vPC breaks unless it is designed for.
If Nexus 1 loses its link to B, the rule is lifted for vPC 20 and Nexus 2 delivers to B. A peer link that is busy in normal running means hairpinning, which is a design problem.
Fail one part at a time
Swipe sideways to see the whole diagram.
Healthy. Both switches forward for servers A and B. Server C is an orphan, attached only to Nexus 2. The keepalive runs over the management network, every second with a five second timeout by default, using UDP port 3200.
Peer status : peer adjacency formed ok Keepalive status : peer is alive Consistency status : success vPC role : primary
Peer link down, keepalive up. The keepalive proves Nexus 1 is alive, so the secondary suspends its vPC member ports and the switched virtual interfaces (SVIs) for vPC VLANs. A and B carry on through Nexus 1. Orphan C loses service, which is why I dual home everything or use orphan port suspend so the server fails over.
Peer status : peer link is down Keepalive status : peer is alive Nexus 2 vPC role : secondary, vPCs suspended
Keepalive down only. Nothing changes for traffic, so this fault is easy to miss. But the pair can no longer tell a dead peer from a broken link, so a second fault would cause a split brain. I alert on it and treat it as urgent.
Peer status : peer adjacency formed ok Keepalive status : peer is not reachable
Peer link and keepalive both down. Each switch thinks the other is dead and acts as primary: two gateways, two spanning tree roots and MAC addresses flapping. I restore the keepalive first, then the peer link, so the roles settle cleanly.
Peer status : peer link is down Keepalive status : peer is not reachable vPC role : primary, on both switches
Primary reloads. Nexus 2 becomes the operational primary and carries everything. Roles do not preempt, so when Nexus 1 returns it stays secondary, and delay restore holds its vPC ports down until routing and ARP are ready.
Peer status : peer link is down Nexus 2 vPC role : operational primary
One member link down. Server A loses its link to Nexus 1 but stays up on Nexus 2. The loop prevention rule is lifted for vPC 10, so traffic for A that arrives at Nexus 1 crosses the peer link. This is why I size the peer link for the failure case.
id Port Status Consistency 10 Po10 up success (one member down) 20 Po20 up success
Both lose power, only Nexus 2 returns, auto recovery on. After the reload delay, 240 seconds by default, Nexus 2 assumes its peer is gone and brings the vPCs up on its own.
Auto recovery : enabled (timeout 240s) Nexus 2 vPC role : primary, vPCs up
Same failure, auto recovery off. Nexus 2 waits for a peer that never comes, so the vPCs stay down and the servers have no service. This is why auto recovery is on in every design I sign.
Auto recovery : disabled vPC status : down, peer never detected
Consistency checks
| Type | Examples | What happens on a mismatch |
|---|---|---|
| Type 1, global | Spanning tree mode, Multiple Spanning Tree (MST) region, Bridge Assurance, Loop Guard, some quality of service (QoS) and maximum transmission unit (MTU) settings | With graceful consistency, the default on current releases, only the secondary suspends its members. A global mismatch hits every vPC at once, so I change both peers in the same window. |
| Type 1, interface | LACP mode, speed, duplex, switchport mode, native VLAN, MTU, spanning tree port type | That vPC is suspended on the secondary. |
| Allowed VLANs | A VLAN allowed on one peer and not the other | Only the mismatched VLANs are suspended. |
| Type 2 | An SVI on one peer only, an access control list (ACL) or QoS difference, depending on release | A warning, with no suspension. It still needs fixing. |
I check with show vpc consistency-parameters global, show vpc consistency-parameters interface port-channel 10 and show vpc brief, alongside show vpc peer-keepalive and show vpc orphan-ports.
The settings I turn on, in order
feature lacp feature vpc vpc domain 10 role priority 100 peer-keepalive destination 10.0.0.2 source 10.0.0.1 vrf management peer-switch peer-gateway auto-recovery delay restore 150 ip arp synchronize interface port-channel1 switchport mode trunk spanning-tree port type network vpc peer-link interface port-channel10 switchport mode trunk vpc 10
Peer gateway
Each peer routes frames sent to the other peer's own MAC address. Some storage arrays and load balancers reply to the router's real MAC instead of the gateway's virtual MAC, and without this they break. I also turn off IP redirects on vPC SVIs.
Peer switch
Both peers present one spanning tree bridge ID, so the root does not move when the primary fails. Both need the same spanning tree priority.
Auto recovery and delay restore
Auto recovery lets a lone survivor bring its vPCs up after a dual power loss. Delay restore holds vPC ports down after the peer link returns, 30 seconds by default, so routing and ARP are ready first. ip arp synchronize copies ARP entries over the peer link, so a returning peer can route at once.
Gateways on a vPC pair
HSRP (Hot Standby Router Protocol) or VRRP (Virtual Router Redundancy Protocol) runs as normal on the SVIs, but both peers forward frames sent to the virtual MAC. I keep the active gateway on the vPC primary. VXLAN EVPN replaces this with a distributed anycast gateway.
Routing to a vPC attached router
Routing protocol peering across a vPC needs layer3 peer-router, and support depends on platform and release. I prefer dedicated layer 3 links for routers and firewalls.
What I watch
Keepalive status, peer link use, consistency status and orphan ports. An unnoticed keepalive loss is the failure that worries me most.
Two switches as one: the options compared
| Option | Control planes | Where I use it |
|---|---|---|
| EtherChannel | One switch, or one logical switch | Bundling links between any two devices, with LACP. |
| StackWise | One, for a stack of access switches | Campus access closets. |
| Virtual Switching System (VSS) and StackWise Virtual | One, two chassis merged into one logical switch | Catalyst campus cores and distribution. |
| vPC | Two independent switches cooperating over the peer link | Nexus data centre pairs, including VXLAN EVPN leaf pairs with a shared virtual tunnel endpoint. |
| EVPN multihoming with an Ethernet segment identifier (ESI) | Independent switches, coordinated by BGP | The standards based alternative to vPC on VXLAN EVPN fabrics. |
vPC keeps two control planes, so a software fault on one switch does not take down both. The price is two configurations to keep aligned and failure behaviour that the team must understand.
How I build vPC pairs
- Keepalive on its own path. Over mgmt0 through the out of band switch, or a dedicated routed link in its own VRF. Never over the peer link.
- Peer link sized for failure. At least two links on different line cards or chips, carrying vPC VLANs only.
- Same software, same config. Matching NX-OS versions and settings, and unique domain IDs.
- Dual home everything. Single attached devices go on the primary with orphan port suspend.
- One peer at a time. Changes and upgrades go to one peer, then are verified, then the other.
- Watch it. Alerts on keepalive, peer link use, consistency and orphans.
Older Nexus estates
Features I meet when I migrate
Many older data centres still run Nexus 7000 or 5000 platforms, with virtual device contexts (VDCs), fabric extenders (FEX), Overlay Transport Virtualisation (OTV) and FabricPath. These features solved real problems in their day, and knowing how they behave is what makes a migration off them safe.
Swipe sideways to see the whole diagram.
Virtual device contexts. Each VDC has its own configuration, processes, VLANs, VRFs and administrators, and each port belongs to one VDC. The admin VDC is for management only, and FCoE (Fibre Channel over Ethernet) needs its own storage VDC. Traffic between contexts needs a physical cable or another device. The chassis, supervisors and power are shared, so a supervisor failure hits them all.
Fabric extenders. A Nexus 2000 fabric extender (FEX) is a remote line card configured from its parent, with ports such as Ethernet101/1/1. It does no local switching, so server to server traffic hairpins through the parent. Host ports have bridge protocol data unit (BPDU) Guard on by default. If a FEX goes offline, I check its fabric uplinks first.
Overlay Transport Virtualisation. OTV extended layer 2 over any IP network. An IS-IS control plane advertised MAC addresses, so unknown unicast was not flooded, each site kept its own spanning tree, and one authoritative edge device (AED) per VLAN prevented loops. VXLAN EVPN Multi-Site has replaced it.
FabricPath. Switches got switch IDs and IS-IS computed shortest paths, giving equal cost layer 2 paths with a time to live (TTL) in the header and no spanning tree inside. vPC+ dual homed classic Ethernet devices. Like OTV it was Cisco only, and VXLAN EVPN has taken its place.
| Feature | What I check | Commands |
|---|---|---|
| VDC | Which ports and resources each context holds, the licence for extra contexts, the high availability (HA) policy per context. Moving a port between VDCs wipes its configuration. | show vdc, show vdc membership, show vdc resource, switchto vdc AGG |
| FEX | Fabric uplinks, whether the FEX is single homed or dual homed to a vPC pair, and its image state. | show fex, show fex 101 detail, show interface fex-fabric |
| OTV | The join, overlay and internal interfaces, the site VLAN, which edge is authoritative for each VLAN. | show otv, show otv adjacency, show otv route |
| FabricPath | Switch IDs, the IS-IS topology and vPC+ pairs. | show fabricpath switch-id, show fabricpath route |
The Nexus 7000 range is end of sale, so I plan these features out as part of the move to VXLAN EVPN.
Interconnect
Pods, the WAN and the other site
Inside a pod, every leaf is two hops from every other. Between pods, through the border and across to another data centre, I decide exactly what crosses each boundary: which routes, which segments, and ideally no layer 2 at all.
Swipe sideways to see the whole diagram.
Super spine planes. Spine 1 of every pod joins plane 1, spine 2 joins plane 2, and so on. Each plane is independent, so I can maintain one at a time. With the RFC 7938 pattern all super spines share one ASN, each pod's spines share another, and each leaf or leaf pair has its own.
A plane fails. Each pod loses one of its four uplink groups, a quarter of its bandwidth to other pods, but no pod loses connectivity. Traffic inside each pod is not touched.
Border leaves. The handoff to the WAN or core is one subinterface and one eBGP session per VRF, or a Multiprotocol Label Switching (MPLS) layer 3 VPN. External routes come in as EVPN type 5 routes, usually a default per VRF. The pod's prefixes go out summarised, never thousands of host routes. Both directions of a flow stay on the same firewall cluster, and I set the Transmission Control Protocol (TCP) MSS (maximum segment size) for the WAN's smaller MTU (maximum transmission unit).
Between data centres. Border gateways join the sites over two dense wavelength division multiplexing (DWDM) paths on different physical routes. Each site keeps its own fabric and failure domain, and only agreed routes cross. Light in fibre adds about 5 microseconds per kilometre, so 100 km adds about 1 ms of round trip time.
| Boundary | What crosses | What it contains |
|---|---|---|
| Leaf to spine, inside a pod | Tunnelled traffic between tunnel endpoints, two hops | One pod |
| Pod to super spine | Summarised pod prefixes and EVPN routes for shared segments | Faults inside each pod |
| Border leaf to WAN and core | One handoff per VRF, summaries out, defaults in | The fabric from the WAN |
| Site to site | Agreed routes, through Multi-Site border gateways | Each data centre from the other |
Moving off a legacy data centre
- Baseline with tools. Collect configurations, import them into NetBox, model the routing with Batfish and gather flow data. Conversations alone miss what the network really does.
- One target standard. One fabric design, data model, repository, pipeline and evidence pack, whatever the starting point.
- Patterns, then waves. Group workloads by migration pattern, prove each pattern in a lab and a pilot, then move in waves ordered by dependency and change window.
- Layer 2 only for a while. A temporary layer 2 bridge from the old network is an approved exception with an end date, not part of the design.
- Govern by evidence. The change board sees the configuration difference, the checks and the rollback for every wave.
- Retire deliberately. No MAC addresses learned on the old side for an agreed number of days, no flows, no DNS records pointing at it, then power off with a rollback horizon, and remove it from IPAM last.
Out of band
A way in when the fabric is down
In band management fails exactly when it is needed most: during a fabric outage or a bad change. A separate out of band network keeps every device reachable, whatever state production is in.
Swipe sideways to see the whole diagram.
Each rack has a small out of band switch with 1G copper to every management port: switches, server management controllers and power strips.
A console server in the network rack has a serial cable to every switch. If a switch has lost its management address or its configuration, its console still answers.
The rack switches and console servers join a pod aggregation pair, which uplinks to the site out of band (OOB) core. One address block per pod keeps it tidy.
The OOB network never routes over the fabric it manages, and it does not depend on production DNS or sign in without a local fallback. Engineers reach it only through its own firewall, over a virtual private network (VPN), and a jump host with multi factor authentication (MFA) and recorded sessions.
A 4G or broadband link gives a path in that does not depend on the site WAN. It lands on the same OOB firewall and jump host, stays locked down until it is needed, and every use raises an alert.
A bad change takes the fabric down and production stops. I sign in through the jump host, reach the console server over the out of band network, and fix the spine from its console.
How I build the management network
- Built first. Out of band goes live before production, so the fabric can be built and fixed through it.
- No hidden dependencies. Its own switches, routing and firewall, fed from both the A and B power sides, with local fallback for name lookup and sign in.
- Management ports only. Server management controllers are never reachable from user or internet networks.
- Tested on a schedule. The backup path and the break glass accounts are tested regularly, and the results are kept.
A management VRF is not out of band
On Nexus, mgmt0 sits in its own management VRF by default. That separates it logically, but if the cabling runs through the production fabric, it fails with the fabric. I want both: the management VRF and a physically separate network.
Zero touch provisioning first
Zero touch provisioning (ZTP) runs over the out of band network, so it is live before the first leaf arrives.
Last resort access
When the site is cut off from the WAN, console servers on a separate circuit or a cellular link give a way in. It stays locked until invoked, needs two approvals and a privileged access management (PAM) session, records everything, alerts the security team, and is tested every quarter.
Shared services
The foundations everything else needs
A fabric can be perfectly redundant and still go dark because DNS, time or a key manager failed. I design these shared services as carefully as the network, and I prove the order everything starts in.
| Service | If it fails | How I make it resilient |
|---|---|---|
| DNS | Almost everything fails, including sign in. | Anycast resolvers in every pod or site, several authoritative servers, and caching. |
| Identity (Active Directory, Lightweight Directory Access Protocol (LDAP), identity provider) | Nobody can sign in, including administrators. | Two domain controllers per site, local break glass accounts in a vault, resilient privileged access. |
| Time (NTP and PTP) | Kerberos fails beyond five minutes of skew, certificates look invalid, trading timestamps break. | Three or four independent sources, a local source per pod, satellite and terrestrial references, holdover oscillators. |
| DHCP | New devices, network boot with the preboot execution environment (PXE) and zero touch provisioning (ZTP) fail. Existing leases carry on. | Failover pairs, long leases and reserved infrastructure addresses. |
| IPAM | Automation stops. Running services are fine. | High availability, backups and pipelines that fail safe. |
| Key management service and hardware security modules | Encrypted virtual machines and storage may not start after a reboot. | A hardware security module cluster across sites, never hosted on the platforms it serves. |
The cold start
Swipe sideways to see the whole diagram.
After a full power loss, everything restarts in an order: power and cooling, out of band access, time, DNS and IPAM, identity, keys, then the fabric, platforms and workloads. Each rung needs the ones below it.
Circular dependencies only show up in a cold start. A key manager virtual machine on the encrypted cluster that needs its keys, domain controllers on a cluster whose management needs a domain login, or DNS behind a firewall whose management needs DNS.
I put the foundations in a separate shared services pod, outside the pods they serve, with local break glass accounts and local name resolution as a fallback.
A full pod cold start is part of factory acceptance and is repeated in production on a schedule. It is timed, and the result is kept as evidence for the regulator.
Upgrades without an outage
Swipe sideways to see the whole diagram.
Traffic between leaves spreads across all four spines by equal cost multipath.
NX-OS graceful insertion and removal (GIR) puts the spine into maintenance mode. BGP graceful shutdown lowers its routes first, so traffic moves before anything is torn down.
With no traffic on it, the spine is upgraded. The other three carry the load, so I plan every fabric for N minus 1 at peak.
Routing sessions up, counters flat, telemetry clean. Only then does it leave maintenance mode, and only then does the next spine start. In service software upgrade (ISSU) exists on some platforms, but a rolling upgrade is the normal pattern in leaf and spine.
- One at a time. One leaf of a pair, one spine, one plane. In Cisco ACI I use odd and even maintenance groups.
- Drain first. Maintenance mode or graceful shutdown before work, so traffic moves cleanly.
- Stagger upgrades. Running the same version on both halves of a pair is right. Upgrading both at once is not.
- Mixed versions supported. Every step of a rolling upgrade must be a supported combination.
Segmentation
Zones that meet only at a firewall
Segmentation decides who can talk to whom, so one compromised server cannot reach everything. I set a small number of security zones once, and every pod inherits them. The wider zero trust approach is on the security page.
Swipe sideways to see the whole diagram.
Each security zone gets its own VRF (virtual routing and forwarding), a separate routing table, across every leaf. Without a route there is no reach, whatever the firewall rules say.
Zones meet only at a firewall cluster on the border leaves. It inspects and logs every flow that crosses, and the rule base has named owners.
Production asks shared services for name lookup (DNS). A rule allows it, so the firewall passes it through to the shared services VRF.
A test server tries to reach the card data zone. There is no rule, so the firewall drops it and logs the attempt. This boundary is what keeps the scope of PCI DSS, the Payment Card Industry Data Security Standard, small.
Within production, micro segmentation decides which workloads talk. Web servers reach the application tier, never the database directly, so one compromised server cannot reach everything.
Zone names, VRFs and rules are part of the pod standard, so every new pod arrives with the same boundaries. It follows the zero trust ideas on the security page.
The zone matrix
Swipe sideways to see the whole diagram.
Internet to DMZ: allowed. HTTPS from the internet reaches only the demilitarised zone (DMZ), where reverse proxies and load balancers take the connection.
DMZ to production: allowed. The reverse proxies reach the application tier on its published port, and nothing else crosses.
Internet to production: blocked. Nothing from the internet reaches production directly. Every request comes through the DMZ.
Management to production: allowed. Administrators reach servers only from jump hosts in the management zone, over secure shell (SSH) or HTTPS, with every session recorded.
Production to management: blocked. No production server can open a session into the management zone, so a compromised server cannot reach the tools that run the estate.
| From | To | Allowed | Rule owner |
|---|---|---|---|
| Internet | DMZ | HTTPS to the reverse proxies | Security team |
| Internet | Production, PCI | Nothing | Security team |
| DMZ | Production | The published application port only | Application owner |
| Production | Shared services | DNS, NTP and sign in | Platform team |
| Production | PCI | Named payment calls only | PCI service owner |
| Non production | Production, PCI | Nothing | Security team |
| Management (OOB) | All zones | SSH and HTTPS from jump hosts | Infrastructure team |
| Any zone | Management (OOB) | Nothing can open a session in | Infrastructure team |
An indicative example. In a real design every rule has a named owner and a review date.
Identity, not addresses
Inside a zone, policy follows the workload rather than its IP address: Cisco ACI (Application Centric Infrastructure) contracts between endpoint groups, a distributed firewall in the hypervisor, or Kubernetes network policy on labels. The rule says web may reach app, and it holds wherever the workload moves.
Engagement
How I run a data centre network programme, from first rack to refresh
A data centre fabric touches facilities, platforms, security and operations at once. I run every programme the same way: agree the outcome, turn it into numbers, prove the current state with tools, then design, build and hand over a fabric the operations team can run and keep current.
Swipe sideways to see the whole diagram.
I start by agreeing what the fabric is for: new capacity, a migration off old Nexus platforms, better resilience or lower cost per rack. Each goal becomes a measure, and each decision gets one accountable owner in a RACI (responsible, accountable, consulted, informed).
Functional requirements say what the fabric must connect and separate. Non functional ones give numbers: oversubscription, failover time, ports per rack, growth and change windows. Every number is traced to a test in factory and site acceptance.
I collect every configuration, import it into NetBox and model the routing with Batfish. Flow data shows what really talks to what, and the cabling and power audit shows what the drawings got wrong. Workshops come second.
I design one pod type with role templates for compute, storage, service and border leaves, the underlay and overlay, the address and ASN plan, the out of band network and the zones. The design authority approves it before anything is ordered.
Racks are built and burned in at an integration centre, configured from the source of truth and failure tested: links, leaves, spines and power feeds. Live services then move in waves through change control, each with a rollback deadline.
Before handover the operations team gets runbooks for the common faults, alerts tuned on real telemetry and a tested escalation path to the vendor technical assistance centre (TAC). Hypercare ends only when the agreed exit criteria are met.
A fabric needs a software baseline, upgrades in waves and a refresh cycle tied to the vendor's end of life dates. I keep a capacity trigger per pod type, so the next pod is ordered before the current one is full.
Functional requirements
What it must do.
- Connect every server rack with dual homed links to a leaf pair.
- Separate production, PCI card data, non production and management zones, meeting only at a firewall.
- Give every workload the same gateway wherever it runs in the pod.
- Join pods through a super spine, and sites through border gateways.
- Provide an out of band network that works when the fabric is down.
- Build every pod of a type from the same templates and data.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Capacity
- Oversubscription agreed per pod type, for example 3 to 1 for general virtual machines and 1 to 1 for GPU back ends.
- Resilience
- No single device, link or power feed takes a dual homed server down. Capacity after one spine failure stays above peak demand.
- Recovery
- Traffic moves within about a second of a link or spine failure.
- Scale
- Racks per pod, pods per hall and ports per rack agreed, with growth headroom.
- Security
- Zones enforced by VRF and firewall, management reachable only from jump hosts, every change logged.
- Operability
- Configuration generated from the source of truth, drift reported weekly, upgrades without an outage.
- Cost
- Cost per usable port and per rack tracked in the cost model.
These are typical starting targets. Each programme agrees its own numbers, and each one is traced to an acceptance test.
Who I talk to
- Server, virtualisation and storage teams
- Facilities: power, cooling, space and cabling
- Security architects and the risk team
- Network operations and the service desk
- Application owners for the critical services
How I find out
- Workshops by pod type and workload
- Configuration collection across every switch, router and firewall
- Flow analysis to find what really talks to what
- Physical audits of racks, ports, cabling and power
- End of life review against vendor dates
Discovery tools
| Tool or source | What it tells me |
|---|---|
| Configuration backups and NetBox import | Every device, port, VLAN and address in one source of truth. |
| Batfish | A model of the real routing and ACLs, so I can test a change before it is made. |
| NetFlow, sFlow or fabric telemetry | Which servers and applications talk to each other, and how much. |
show lldp neighbors and show cdp neighbors | The real cabling, compared with the drawings. |
| DCIM and rack audits | Space, power per rack and which ports are free. |
| Vendor end of life notices | Which platforms and software trains are out of support, and when. |
In a fabric I check the underlay and the overlay separately. Most faults are a link or optic, a vPC or port channel problem, or a missing route or segment, in that order.
My approach
- Define. Which servers, VLANs, VRFs or pods are affected, since when, and what changed?
- Physical. Interface state, errors, optics, cabling and power on the leaf, spine and server.
- Layer 2. Port channel and vPC state, consistency checks, MAC learning and spanning tree at the edge.
- Underlay. BGP, OSPF or IS-IS sessions, loopbacks reachable, ECMP paths and BFD.
- Overlay. EVPN routes, tunnel endpoint peers, the anycast gateway and the VRF route tables.
- Fix and record. Fix through change on one pod first, prove it with the same show commands, and add the cause to the pod standard.
Faults I see most
- A dirty optic or failing cable causing errors but staying up
- A vPC consistency mismatch suspending ports
- An orphan port losing service after a peer link failure
- A missing VLAN or VNI on one leaf
- An MTU mismatch breaking large frames over the overlay
- Asymmetric routing through a stateful firewall
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | show interface errors and counters, show interface transceiver details, cabling and power. |
| 2 · Data link | show port-channel summary, show vpc brief, show mac address-table, show spanning-tree. |
| 3 · Network | show ip route vrf TENANT1, show bgp ipv4 unicast summary for an eBGP underlay, show ip ospf neighbors, show bfd neighbors. |
| Overlay | show nve peers to confirm the tunnel endpoints are up. The full overlay method, with the EVPN tables, is on the VXLAN EVPN and ACI page. |
| 4 to 7 · Application | Load balancer and firewall sessions, DNS, and what the application team really sees. |
My general method, with worked faults, is on the troubleshooting page.
I agree who supports what before the first pod goes live, so nobody learns their role during an outage.
Service model
Who owns each layer: network operations, platform teams, facilities, the vendor and any managed service provider, with response times.
Monitoring
Streaming telemetry on errors, buffer drops, BGP and vPC state, with alerts tuned so each one means something.
Runbooks
Step by step guides for a failed leaf, optic or power supply, a vPC fault and adding a rack.
Change
Changes generated from the source of truth, checked before and after, applied to one pod at a time.
Spares
Spare leaves, optics and power supplies on site, matched to the support contract times.
Problem management
Repeat faults, such as a bad batch of optics, get a root cause, a known error record and a change to the pod standard.
A data centre fabric lives for five to seven years. I plan its whole life from day one.
- Baseline. One approved software version per platform, checked in the lab before it is used.
- Upgrade. Rolling upgrades, one leaf of a pair and one spine at a time, drained first.
- Patch. Security advisories triaged by risk. Critical fixes go through an emergency change, the rest join the next wave.
- Review. Quarterly reviews of capacity, oversubscription, optics and drift from the standard.
- Refresh. A new pod is built beside the old one, workloads move in waves, and the old pod is drained and retired.
- Dispose. Configurations wiped, assets removed from the CMDB and IPAM, and hardware returned or recycled through a certified partner.
Projects
Where I have built and run them
Data centre networks across telecoms, banking, the public sector and industrial software.
Network transformation on Cisco Nexus
Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access and AWS, Azure and Google Cloud environments.
Data centre transitions at 99.99% uptime
Safeguarded over £200 million of transactional services through data centre transitions, holding 99.99% uptime throughout.
Tier 3 data centre on Cisco ACI and Nexus
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI (Application Centric Infrastructure) and Nexus, including provider IP space and autonomous system (AS) numbers managed with RIPE NCC, the European internet registry.
Europe, Middle East and Africa refresh
Audit, design and implementation across LAN, WAN, SD-WAN and data centre infrastructure, replacing end of life equipment with no service outages under MiFID II.
Public sector network on Nexus
Public sector network modernisation on Meraki, Nexus, FortiGate and Check Point, leading 7 engineers and third party suppliers.
Enterprise data centre after a merger
Post merger infrastructure integration across a multi site estate and the enterprise data centre, on Cisco, Palo Alto and VMware.
Backed by VMware Certified Professional, Data Centre Virtualisation, alongside CCNP Enterprise and CCDP. The Network Rail build and my power, cooling and cabling work are on the data centre facilities page. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.