SD-Access campus fabric

A campus fabric where identity sets the policy

I design and build Cisco SD-Access fabrics with Catalyst Center and ISE (Identity Services Engine). This page walks through each plane, every role and the traffic between them, from the first switch powering on to the fault finding when something goes wrong.

  • SD-Access
  • Catalyst Center
  • LISP
  • VXLAN
  • TrustSec
  • Cisco ISE
  • 802.1X and MAB
  • LAN Automation
A campus fabric carrying identity Two edge switches register their endpoints with a control plane node and send traffic to each other in VXLAN across a routed underlay. A laptop tagged as staff and a camera tagged as cameras keep their groups wherever they connect. ROUTED UNDERLAY Control planeLISP Edge Aanycast gateway Edge Banycast gateway staff · SGT 10 camera · SGT 40 VXLAN with the group tag
Live view · endpoints registered, traffic tagged by group
Overview

A campus fabric where identity sets the policy

SD-Access is Cisco's campus fabric. It splits the network into four planes, each doing one job, so that a user or device gets the same access wherever it connects. I design it from the underlay up. The SD-WAN (software defined wide area network) page has a shorter summary of how it fits with the WAN (wide area network).

The four planes of an SD-Access fabric Catalyst Center, a control plane node and Cisco ISE sit above two edge switches joined through an intermediate switch. In the underlay view, IS-IS runs over routed links and Catalyst Center automates the switches. In the control plane view, each edge registers its endpoints with the control plane node. In the data plane view, a laptop reaches a printer through a VXLAN tunnel between the edges. In the policy view, ISE gives the laptop a group and the far edge enforces group policy. Catalyst Centerautomation Control planeLISP Cisco ISEidentity, groups Edge 1access switch Intermediateunderlay only Edge 2access switch Laptopstaff Printerprinters group IS-IS over routed links, no spanning tree Each edge registers what sits behind it VXLAN between edge loopbacks Staff to printers: permit, by group not by IP VXLAN

Swipe sideways to see the whole diagram.

Underlay. A plain routed network: point to point links, a loopback on every switch, and IS-IS (Intermediate System to Intermediate System) as the routing protocol. There is no spanning tree between fabric switches. Catalyst Center builds and checks it, and nothing clever lives here on purpose.

Control plane. LISP (Locator/ID Separation Protocol) keeps a live directory of every endpoint: which switch it sits behind. Edges register what they learn and ask the control plane node when they need to reach something, instead of flooding.

Data plane. Traffic between edges travels in VXLAN (virtual extensible LAN) between the switches' loopbacks. Any subnet can live anywhere without stretching VLANs (virtual local area networks), and every edge acts as the default gateway.

Policy plane. Cisco ISE (Identity Services Engine) gives each user or device a scalable group tag (SGT) when it connects. The tag travels inside the VXLAN header, and the destination edge allows or blocks traffic by group pair, not by IP address.

Animation

Three things with similar names

Catalyst SD-WAN joins sites across the WAN. SD-Access is the campus fabric. Catalyst Center, formerly DNA Center (Digital Network Architecture Center), is the controller that builds, runs and assures SD-Access. They meet at the border, where each fabric virtual network becomes an SD-WAN service VPN (virtual private network).

Two levels of segmentation

A virtual network (VN) is a VRF (virtual routing and forwarding table): a separate routing table, used for large groups such as staff, guests, IoT (Internet of Things) and OT (operational technology). Inside a VN, scalable group tags give fine grained, identity based rules. Traffic between VNs leaves the fabric through a firewall.

When I recommend it

Large campuses with many device types, a need for identity based policy, frequent moves and a team ready to run ISE and Catalyst Center. For a small site, a well designed routed access campus is often enough.

How it compares with a traditional campus

Traditional VLAN campusSD-Access fabric
SegmentationVLANs, ACLs (access control lists) and firewall rules written per subnetVirtual networks for large groups, scalable group tags inside them
Where a device livesIts subnet is tied to a closet or a stretched VLANAny subnet on any edge, with the same gateway everywhere
Loops and failure domainsSpanning tree across the access and distribution layersRouted underlay, no spanning tree between fabric switches
Wired and wirelessSeparate policy, wireless often tunnelled to a controllerOne policy, wireless traffic switched at the edge
ChangeSwitch by switch configurationIntent set once in Catalyst Center and pushed to every switch
Costs and dependenciesFamiliar and simple to licenceAdvantage tier Catalyst licences, a dependency on ISE, and pool and policy design up front
Underlay

An underlay that is boring on purpose

The underlay is just a routed network that carries the fabric. Catalyst Center can build it for me with LAN (local area network) Automation, so every switch is addressed and configured the same way. I can also build it by hand with OSPF (Open Shortest Path First), as long as the rules are the same.

Building the underlay with LAN Automation Catalyst Center reaches two seed switches set up by hand. New switches with no configuration get a temporary address from a DHCP pool on the seeds that points them to Catalyst Center. Catalyst Center then gives every link a /31 subnet and every switch a loopback from the underlay pool, IS-IS comes up on every link, and the network is ready for the fabric with a 9100 byte MTU. Catalyst CenterLAN Automation Seed 1set up by hand Seed 2set up by hand New switch 1no configurationaddressedfabric ready New switch 2no configurationaddressedfabric ready New switch 3no configurationaddressedfabric ready Seeds reachable, IS-IS ready DHCP on the seed points to Catalyst Center Addresses come from the underlay pool Routed access, no spanning tree MTU 9100 on every link, ready for the fabric /31/31/31/31Loopback0 .11/32Loopback0 .12/32Loopback0 .13/32

Swipe sideways to see the whole diagram.

Step 1
  1. I configure one or two seed switches by hand, normally the distribution or border pair, and make them reachable from Catalyst Center. Everything below them will be built automatically.

  2. New switches arrive with no configuration. They get a temporary address from a DHCP (Dynamic Host Configuration Protocol) pool on the seed, which also tells them where Catalyst Center is. Their Plug and Play agent calls home, and Catalyst Center can upgrade the software before anything else.

  3. Catalyst Center takes addresses from the underlay pool I defined: a /31 for every point to point link and a /32 Loopback0 on every switch. The loopback becomes the switch's address in the fabric, its routing locator (RLOC).

  4. LAN Automation turns on IS-IS on every link. The access layer is now routed, so there is no spanning tree between fabric switches and every uplink carries traffic with equal cost paths.

  5. The system MTU (maximum transmission unit) is 9100 bytes everywhere, so VXLAN's 50 extra bytes never cause trouble. If I need Layer 2 flooding or native multicast later, I add underlay multicast now.

Rules I never bend

Routed point to point links, a /32 loopback per switch reachable everywhere, equal cost paths, MTU 9100 end to end, and no spanning tree between fabric nodes.

Size the pools for the end state

Underlay, endpoint and wireless pools are planned for the final size of the campus, not today's. A /24 that looks generous today runs out when the IoT devices arrive.

Check it like any routed network

show isis neighbors, show ip route for every loopback, and a ping with a large size and the do not fragment bit between loopbacks to prove the MTU.

Fabric roles

Every switch has one clear job

A fabric is easier to run when every device has a defined role. I pick the roles for each site by size and resilience: two control plane nodes and two borders as standard, and as many edges as there are closets.

The roles in an SD-Access fabric A fabric with a border node, a control plane node and a fabric wireless controller at the top, an intermediate switch in the middle, and two edge switches below. One edge has a fabric access point, the other connects to an extended node that serves an IoT ring. Each view highlights one role and shows its traffic. Borderinternal, external or both Control planeLISP map server Fabric WLCwireless controller Intermediateroutes only Edgeaccess switch Edgeaccess switch Extended nodeindustrial switch Fabric AP IoT ring Laptop Gateway, authentication and enforcement The only way in and out of the fabric Knows where every endpoint lives Control through the WLC, data at the edge Extends the fabric to plant rooms to WAN

Swipe sideways to see the whole diagram.

Edge node. The access switch. It is the default gateway for every VN, with the same anycast address on every edge. It authenticates users and devices with 802.1X or MAB (MAC Authentication Bypass), applies the group tag, registers endpoints with the control plane and enforces group policy. Typically a Catalyst 9200, 9300 or 9400.

Border node. The way in and out. An internal border connects to known networks, such as the data centre. An external border is the default exit to everything else, such as the internet through a firewall, and an internal and external border does both. A Layer 2 border joins a fabric pool to an old VLAN during migration. Each VN is handed off separately, usually as VRF lite with BGP (Border Gateway Protocol) to a fusion device or firewall.

Control plane node. The LISP map server and map resolver: a database of every endpoint and the edge it sits behind. I always deploy two per fabric site, on Catalyst 9500 or 9600 switches, or on Catalyst 8000V virtual routers for large sites.

Fabric wireless. Access points keep CAPWAP (Control and Provisioning of Wireless Access Points) control traffic to the wireless LAN controller, but client data goes straight into the fabric at the edge in VXLAN. The controller registers each wireless client with the control plane node, so wired and wireless share one policy.

Extended node. A Catalyst IE (Industrial Ethernet) industrial switch or a compact switch in a plant room or ceiling, onboarded and managed by Catalyst Center. Its ports join VNs like edge ports. A policy extended node can also enforce group policy; a plain one relies on its parent edge.

Animation
RoleWhat it doesTypical platform
Control plane nodeLISP map server and resolver: where every endpoint livesCatalyst 9500 or 9600, or Catalyst 8000V. Two per site
Border nodeConnects the fabric to everything else, one handoff per VN. Internal, external, or bothCatalyst 9500 or 9600, or a router
Edge nodeGateway, authentication, tagging, registration, enforcementCatalyst 9200, 9300 or 9400
Intermediate nodeRoutes underlay traffic only, knows nothing about the fabricAny routed switch
Fabric wireless controllerCAPWAP control for access points, registers wireless clientsCatalyst 9800 series
Extended nodeExtends fabric ports to places an edge cannot reach, in REP (Resilient Ethernet Protocol) rings where neededCatalyst IE 3300 or 3400, compact Catalyst switches

Fabric terms in one line each

TermWhat it means
INFRA_VNThe virtual network for access points and extended nodes. It lives in the global routing table, not a VRF.
DEFAULT_VNThe user virtual network Catalyst Center creates for you. I add my own VNs for each large group.
Anycast gatewayThe same gateway address and MAC (media access control) address on every edge, so a device keeps its gateway wherever it connects.
Critical VLANWhere a port puts devices when ISE cannot be reached, so they keep a known, limited level of access.
Device trackingSISF (switch integrated security features) learns each endpoint's addresses from DHCP, ARP (Address Resolution Protocol) and neighbour discovery, so the edge can register it with LISP.
pxGridISE's publish and subscribe service, which shares who and what is on the network with firewalls and other tools.
SXP (SGT Exchange Protocol)It carries IP address to group bindings over TCP (Transmission Control Protocol) to devices that cannot read tags inline.
Plug and PlayHow a new switch finds Catalyst Center and gets its configuration, with LAN Automation building the underlay.
LicencesSD-Access needs the Advantage tier of the switch and Catalyst Center licences. Essentials is not enough.
LISP

A directory instead of flooding the campus

The control plane is what makes the fabric scale. Instead of every switch learning every address by flooding, each edge registers what it has and asks for what it needs. The instance ID in these commands is an example: each VN has its own.

How LISP finds an endpoint Edge A learns a laptop and registers it with the control plane node. When the laptop sends to a printer behind edge B, edge A asks the control plane node, caches the reply and sends the traffic in VXLAN to edge B. Traffic for an address outside the fabric goes to the border. When the laptop moves to edge B, edge B registers it and edge A is told it has moved. Control plane database Edge A map cache 10.10.1.20 at .11 10.10.1.20 at .12 10.10.1.50 at .12 10.10.1.50 at .12 8.8.8.8 not found 10.10.1.20 now at .12 Control planemap server, resolver Edge ARLOC 10.0.250.11 Edge BRLOC 10.0.250.12 Borderoutside the fabric Laptop10.10.1.20 Printer10.10.1.50 Laptopmoved here Device tracking learns 10.10.1.20 on Gi1/0/5 Map register: 10.10.1.20 is behind edge A Map request: where is 10.10.1.50? Map reply: behind 10.0.250.12, cached VXLAN straight to edge B No entry for 8.8.8.8: send it to the border Edge B registers the laptop, the control plane tells edge A VXLAN

Swipe sideways to see the whole diagram.

Step 1
  1. The laptop connects to edge A and gets an address. Device tracking on the switch (SISF) learns its IP and MAC address from DHCP and ARP. In LISP terms the laptop is an endpoint identifier (EID), and edge A's loopback is its routing locator (RLOC).

  2. Edge A sends a map register to the control plane node: 10.10.1.20 sits behind 10.0.250.11. Each VN has its own LISP instance, so the same address can exist in two VNs without clashing.

  3. When the laptop sends to the printer, edge A checks its map cache. It has no entry, so it sends a map request to the control plane node instead of flooding the whole campus.

  4. The control plane node answers with a map reply: 10.10.1.50 is behind 10.0.250.12. Edge A caches the answer, so later packets go straight out without asking again. On current releases the borders subscribe to the control plane node, so they learn every registration as it happens.

  5. Edge A wraps the packet in VXLAN with edge B's loopback as the outer destination. The underlay only routes between loopbacks and never learns endpoint addresses.

  6. An address the control plane node does not know, such as an internet address, gets a negative reply. Edge A sends that traffic to the border, which acts as the proxy for everything outside the fabric.

  7. When the laptop roams to edge B, edge B registers it again. The control plane node tells edge A, so any edge still sending to the old place is corrected. The laptop keeps its address and its gateway.

Checking the control plane

What I want to knowCommand on IOS XE
Did the edge learn the endpoint?show device-tracking database
Is the edge talking to the control plane node?show lisp session
What has this edge registered?show lisp instance-id 4099 ipv4 database
What does the control plane node know?show lisp instance-id 4099 ipv4 server
What has this edge resolved?show lisp instance-id 4099 ipv4 map-cache
Data plane

VXLAN carries the packet and the identity

Inside the fabric every packet travels in VXLAN between edge loopbacks. The header is small, but it carries two things the old campus could not: which virtual network the packet belongs to, and which group sent it.

What the fabric adds to every frame An ordinary Ethernet frame, then the same frame inside VXLAN. The edge adds an outer Ethernet header, an outer IP header, a UDP header on port 4789 and an 8 byte VXLAN group policy header, 50 bytes in all. The VXLAN header carries the network identifier and the group tag. A third view shows a 1,550 byte packet dropped on a 1,500 byte underlay link but fitting on a 9,100 byte link. Ethernet14IP20TCP20Dataup to 1,460An ordinary Ethernet frame: 1,514 bytes at most Outer Ethernet14Outer IP20UDP8VXLAN-GPO8Ethernet14IP20TCP20Dataadded by the edge: 50 bytesVNI: which network, 24 bitsGroup policy ID: the SGT, 16 bitsUDP port 4789. The source port is a hash of the inner flow,so the underlay spreads flows across equal cost paths.A 1,514 byte frame becomes 1,564 on the wire Underlay link with MTU 15001,550 byte packet from the edgedroppedUnderlay link with MTU 91001,550 byte packet from the edgefitsPings work, large transfers and TLS stall

Swipe sideways to see the whole diagram.

An ordinary frame. What the laptop sends: an Ethernet header, an IP header, a TCP header and up to 1,460 bytes of data.

In the fabric. The edge adds 50 bytes: outer Ethernet, outer IP between the two edges' loopbacks, UDP (User Datagram Protocol) on port 4789 and the VXLAN header. SD-Access uses the VXLAN group policy option (VXLAN-GPO), which carries the 24 bit network identifier (VNI) and the 16 bit group tag, so identity travels with every packet.

Why MTU matters. With 50 extra bytes, a full size packet becomes 1,550 bytes. On an underlay link left at 1,500 it is dropped, and the symptom is confusing: pings and small requests work, file copies and TLS (Transport Layer Security) sessions stall. That is why the underlay runs at 9,100.

Animation

A packet walk inside one virtual network

A packet walk inside one virtual network A laptop behind edge A sends to a printer behind edge B in the same virtual network. The laptop uses the anycast gateway on edge A. Edge A finds the printer's edge in its map cache, wraps the packet in VXLAN with the network identifier and the laptop's group tag, and the underlay carries it over two equal cost paths. Edge B removes the header, checks the group policy and delivers the packet. Edge A10.0.250.11 Intermediate 1underlay Intermediate 2underlay Edge B10.0.250.12 LaptopVN CORP · SGT 10 PrinterVN CORP · SGT 30 The gateway 10.10.1.1 is on every edge Map cache: the printer is behind 10.0.250.12 Outer 10.0.250.11 to .12 · VNI for CORP · SGT 10 Edge B checks the matrix: staff to printers, permit equal cost paths IP VXLAN VXLAN IP

Swipe sideways to see the whole diagram.

Step 1
  1. The laptop sends to its default gateway, 10.10.1.1. Every edge answers for that address with the same MAC address, so the gateway is always one hop away.

  2. Edge A finds the printer in its LISP map cache: it sits behind edge B, whose loopback is 10.0.250.12. If it were not cached, edge A would ask the control plane node first.

  3. Edge A wraps the packet in VXLAN: outer source 10.0.250.11, outer destination 10.0.250.12, the network identifier for VN CORP and the laptop's group tag 10. The intermediate switches route on the outer header only, and spread flows over both paths.

  4. Edge B removes the VXLAN header and reads the source tag. It checks the group policy matrix for staff (10) to printers (30), finds a permit, and delivers the packet. A denied pair would be dropped here, at the destination.

Anycast gateway

Every edge holds the same gateway IP and MAC address for each pool. A device that moves, or a phone that roams between access points, always has its gateway on the switch it is connected to: no first hop redundancy protocol, no hairpin and no new ARP.

Layer 2 and Layer 3 networks

Each pool gets a Layer 2 network identifier for traffic inside the subnet, and each VN gets a Layer 3 identifier for routed traffic. ARP is answered using LISP rather than flooded.

When something needs broadcast

Broadcast is dropped by default. Devices that discover by broadcast, such as BACnet (building automation and control networks) controllers or some printers, need Layer 2 flooding on their pool, which uses underlay multicast. A protocol based fix, such as a BACnet broadcast management device, also works.

Multicast in the fabric

Multicast in the fabric runs one of two ways. With head end replication, the source edge copies each packet to every receiving edge as unicast VXLAN, so the underlay needs nothing extra. With native multicast, the underlay runs Protocol Independent Multicast in source specific mode (PIM SSM) and replicates, which scales far better when many edges have receivers.

Group policy

Rules written by who, not where

In SD-Access the policy is a matrix of groups, not a list of IP addresses. A camera is a camera wherever it is plugged in, and the rules follow it.

Group based policy: classify, carry, enforce A laptop logs in at edge A and Cisco ISE returns a group tag. The tag travels inside every VXLAN packet across the fabric, and SXP or inline tagging carries it to a firewall outside the fabric. Edge B downloads the group policy matrix from ISE and enforces it on traffic arriving for its own endpoints. Cisco ISEgroups and matrix Edge Aclassifies Edge Benforces Firewalloutside the fabric Laptopstaff Login as staff: ISE returns SGT 10 SGT 10 rides inside every VXLAN packet Inline tags at the border, or SXP from ISE, carry it beyond the fabric Edge B downloads the matrix: 10 to 30 permit SGT 10

Swipe sideways to see the whole diagram.

Step 1
  1. When the laptop authenticates, ISE returns a scalable group tag (SGT) with the access result: 10 for staff. Servers and fixed devices can be tagged statically by subnet, VLAN or port.

  2. The ingress edge writes the tag into the VXLAN header of every packet, so the destination knows who sent it without trusting the source IP address.

  3. Outside the fabric, SXP (SGT Exchange Protocol) shares IP to tag mappings with firewalls and older switches, or links carry the tag inline. The firewall can also learn tags from ISE over pxGrid and write rules by group.

  4. Each edge downloads only the rows of the matrix it needs and enforces them as group ACLs (SGACLs) on traffic for its own endpoints. The ingress edge only classifies, which keeps policy in one place.

Try the group policy matrix

Each row is a source group and each column a destination. Select a cell to see the rule and why I set it that way. This is an example design, not a real customer's policy.

From \ toFinance appVideo recorderBuilding systemsPrint serverInternet
Staff
Contractors
Cameras
Building sensors
Printers
Guests

Default rule for any pair not listed: deny. I start from deny and open only what each group needs.

Macro and micro segmentation

Virtual networks keep large groups apart completely: staff, guests, IoT and OT never share a routing table. Group tags then control which groups may talk inside a VN. I use as few VNs as I can, because traffic between VNs has to leave the fabric.

A short matrix

A handful of groups per VN and a matrix that fits on one page. Complexity is what stops a fabric from scaling, not hardware limits. Every cell has an owner and a reason.

Tags are a security control

Whoever can change a device's group can change its access. I restrict who can edit authorisation policy in ISE, and the tag assignments are reviewed like firewall rules.

Onboarding

Getting every device into the right group

Onboarding decides everything that follows: which VN a device joins and which group it carries. I design it with Cisco ISE, and roll it out in phases so nothing important is locked out on day one.

Onboarding a user or device with ISE A device connects to an edge switch, which relays authentication to an ISE policy service node over RADIUS. A laptop proves itself with a certificate using 802.1X and EAP-TLS and is placed in a pool with a group tag. A device without a supplicant falls back to MAC authentication and profiling. ISE can change a live session with a change of authorisation, and if ISE cannot be reached the port falls back to a limited critical state. Devicelaptop or camera Edge switchauthenticator ISE PSNpolicy service node EAP over LANRADIUS ResultLink up: the switch asks for an identityNo traffic passes yet, apart from what the template allows ResultEAP-TLS: the laptop proves itself with a certificateThe switch relays EAP inside RADIUS to ISE ResultAccess accept: VN CORP pool, SGT 10Optional extras: a downloadable ACL, a session timeout ResultNo supplicant: 802.1X times out, MAB sends the MACProfiling from DHCP, LLDP and CDP: camera, SGT 40 ResultChange of authorisation (CoA)ISE reauthorises the live session: quarantine or a new group ResultISE unreachable: critical authenticationPorts fall into a known, limited state instead of going dark

Swipe sideways to see the whole diagram.

Step 1
  1. When the link comes up, the edge switch, as authenticator, asks the device for an identity using EAP (Extensible Authentication Protocol) over LAN. Until the device is authorised, the port only allows what its authentication template permits.

  2. A managed laptop answers with a certificate using EAP-TLS, the strongest method. The switch relays EAP inside RADIUS (Remote Authentication Dial In User Service) to an ISE policy service node. PEAP (Protected EAP) with a password still works, but it is weaker and newer Windows protections break it.

  3. ISE checks who and what the device is, and returns the pool or VLAN for its VN and its group tag. It can add a downloadable ACL or session timers.

  4. A camera or printer cannot do 802.1X. After it times out, MAC authentication bypass (MAB) sends its MAC address, and ISE profiles it from DHCP, LLDP (Link Layer Discovery Protocol), CDP (Cisco Discovery Protocol) and other clues. The result is always tightly scoped, because a MAC address is easy to copy.

  5. When posture or a profile changes, or a security tool flags the device, ISE sends a change of authorisation (CoA) and the switch reauthorises the session: into quarantine, or into a better group.

  6. Existing sessions stay up until they reauthenticate. New devices would fail, so I set critical authentication on the templates: ports fall into a known, limited state rather than going dark. Two policy service nodes per region make this rare.

Authentication templates in Catalyst Center

Authentication templateWhat it doesWhen I use it
Closed802.1X with MAB fallback. Nothing passes until the device is authorisedThe target state for every user and device port
Open (monitor mode)Traffic passes whatever the result, while ISE logs what it would have doneThe first weeks, to find every device before enforcing
Low impactA small set of traffic, such as DHCP and DNS (Domain Name System), passes before authorisationSites moving from open to closed
No authenticationA static assignment with no checksOnly for documented exceptions: APs (access points), extended nodes and devices that cannot authenticate

Placing ISE

ISE personaJobHow I place it
Policy Administration Node (PAN)Where policy is written and storedA primary and secondary in different data centres
Monitoring and Troubleshooting (MnT)Logs, live sessions and reportsA primary and secondary in different data centres
Policy Service Node (PSN)Answers RADIUS, runs portals and profilingClose to users, at least two per region. Switches try the local PSN (policy service node) first, then a remote one
pxGridShares context such as user, group and posture with other toolsEnabled where firewalls or security tools consume tags

The same ideas in Aruba ClearPass

Cisco ISEAruba ClearPass equivalent
Policy sets and authorisation rulesServices, role mapping and enforcement policies
Authorisation profile: VLAN, dACL, SGT (scalable group tag)Enforcement profile: VLAN, user role, dACL
Guest portalGuest module
BYOD (bring your own device) onboardingOnBoard, with its own certificate authority
PostureOnGuard
PAN, MnT and PSN nodesPublisher, standby publisher and subscribers

IoT in phases

I start every IoT device class in open mode with profiling on, build the endpoint groups and the policy, then move one class at a time to closed with MAB. Unknown devices go to a quarantine group with DHCP and DNS only, and raise an alert.

Certificates where possible

Cameras, gateways and industrial switches that support it get a device certificate, so they can use EAP-TLS instead of MAB. The certificate comes from the factory (IEEE 802.1AR) or is enrolled with SCEP (Simple Certificate Enrollment Protocol) or EST (Enrollment over Secure Transport).

Test the failure

Every quarter I test what happens when a PSN, or all of ISE, is unreachable, and check the reconnect storm when it comes back. Sleepy IoT devices need longer timers.

Guests

Guests get a fabric SSID (service set identifier) mapped to a guest VN, with central web authentication on ISE. The detail is on the wireless page.

Between VNs

Crossing between networks, only through a firewall

Virtual networks keep large groups completely apart. When traffic must cross between them, it leaves the fabric at the border and passes a fusion device, usually a firewall, where I can inspect and log it.

Traffic between virtual networks goes through a fusion firewall A camera in the IoT virtual network sends to a recorder in shared services. The edge has no route for it inside the VN, so it sends the traffic to the border. The border hands each VN to a fusion firewall on its own VLAN with its own BGP session. The firewall checks policy by group and routes the traffic into shared services, which are leaked back into every VN. FABRIC SITE EdgeVN IOT and CORP BorderVRF lite handoff Camera · SGT 40 Fusion firewallbetween VNs Shared servicesDHCP, DNS, recorder IOT · VLAN 3001CORP · VLAN 3002BGP per VN Camera onboarded into VN IOT with SGT 40 No route in VN IOT: the edge sends it to the border The border hands VN IOT to the firewall on its own VLAN Firewall policy by group: cameras to recorder, permit Shared services leaked into each VN through the fusion VXLAN

Swipe sideways to see the whole diagram.

Step 1
  1. The camera joins VN IOT with group tag 40. Inside that VN it can only reach what the group policy matrix allows. Other VNs do not exist as far as it is concerned.

  2. The recorder is not in VN IOT, so the edge has no LISP entry for it and sends the traffic to the border, still in VXLAN and still tagged.

  3. The border hands each VN to the fusion device on its own VLAN and its own BGP session: VRF lite. Keeping the VNs apart all the way to the firewall is what makes this a security boundary.

  4. The fusion firewall checks the flow. With tags from ISE over pxGrid, the rule can say cameras may reach the recorder on its streaming port and nothing else. It then routes the traffic into the shared services network.

  5. DHCP, DNS, ISE and the recorder must be reachable from every VN. The fusion device leaks their routes into each VN, so every device can get an address while the VNs still never reach each other directly.

Fusion router or firewall?

A fusion router only leaks routes. A fusion firewall leaks routes and enforces policy between VNs. I use a firewall wherever the VNs differ in trust, such as IoT or OT against corporate.

DHCP needs care

The DHCP server sits in shared services, so the relay on each edge's anycast gateway and the route back into the VN must both work. SD-Access also relies on DHCP option 82 so the offer returns to the right edge, and the server must hand it back unchanged.

Keep VNs few

Every new VN is another handoff, another BGP session and another firewall interface. I add a VN only when a group needs a separate routing table, and use group tags for everything else.

Sites and transits

One policy across many buildings

A fabric site is one set of control plane nodes, borders and edges, usually one building or campus, and a failure domain I am happy to accept. Sites are joined by a transit, and managed together as a fabric domain with shared VNs and policy.

Joining fabric sites together Two fabric sites, each with a control plane node and a border. With IP transit, each virtual network is handed off as a VRF with its own BGP session across the WAN. With SD-Access transit, VXLAN runs end to end across a WAN with controlled MTU, using transit control plane nodes. With SD-WAN, each virtual network maps to an SD-WAN service VPN and the group tag is carried in the overlay. FABRIC SITE 1 FABRIC SITE 2 Control plane Control plane Bordersite 1 Bordersite 2 WAN routersVRF per VN Transitcontrol plane SD-WANedgeSD-WANedge one VRF and BGP session per VN VXLAN end to end VN carried as a service VPN Each VN handed off as a VRF, the tag carried inline or by SXP One fabric domain, VXLAN across an MTU controlled WAN Each VN maps to an SD-WAN service VPN, the tag rides along VXLAN WAN routersVRF per VN SD-WANedgeSD-WANedge

Swipe sideways to see the whole diagram.

IP transit. Each border hands every VN off as VRF lite with BGP, and the WAN carries a VRF per VN. The group tag travels inline where the WAN supports it, or by SXP. It works over any WAN, which is why it is the most common choice.

SD-Access transit. LISP and VXLAN run end to end between sites, with transit control plane nodes holding the summary of every site. Policy and tags stay native, but the WAN must be private, fast and able to carry the larger MTU, such as campus fibre or metro Ethernet.

With SD-WAN. At the border, each VN becomes an SD-WAN service VPN, so segmentation runs from the campus across the WAN to other sites and the cloud. The group tag can be carried in the SD-WAN overlay too. More on the SD-WAN page.

Animation

Size each site

I keep each site inside the scale limits published for the release: endpoints per control plane node, edges per site, and VNs. Several smaller sites joined by a transit are easier to run than one huge one.

Plan the pools

Separate pools for each device class, sized for the end state, often a /22 or /21 where a /24 looks enough today, with IPv6 (Internet Protocol version 6) on the roadmap. Pools are hard to change once devices live in them.

Size the services too

Catalyst Center runs as a three node cluster for resilience. ISE policy service nodes are sized for the RADIUS rate at peak, not the device count, because sleepy IoT devices send a new MAB request every time they reconnect.

Wireless in the fabric

Fabric SSIDs map to pools and groups, access points tunnel control to the controller, and client data joins the fabric at the edge. I plan 2.4 GHz capacity for IoT separately from staff Wi-Fi, with identity PSK (iPSK) for devices that cannot do 802.1X.

Operations

Running the fabric from one place

Catalyst Center is where the fabric is designed, built and watched. I follow its four stages in order, and I plan the migration from the old campus as carefully as the fabric itself.

StageWhat I set up in Catalyst Center
DesignSite hierarchy of areas, buildings and floors. Network settings for AAA (authentication, authorisation and accounting) with ISE, DHCP, DNS, NTP (Network Time Protocol), syslog and SNMP (Simple Network Management Protocol). Credentials, IP pools, wireless profiles and templates.
PolicyVirtual networks, scalable groups from ISE, the group based access matrix and application QoS (quality of service) policy.
ProvisionDiscover or Plug and Play the switches, assign them to sites, run LAN Automation, create the fabric site, give each switch its role and set host onboarding per port.
AssuranceHealth scores for devices, clients and applications from streaming telemetry, Client 360 with the onboarding timeline of each client, path trace across the fabric, Endpoint Analytics feeding device profiles to ISE, and issues raised against learned baselines.

How I migrate a live campus onto SD-Access

  1. Find out how devices really talk. Before moving anything I list each device class and whether it needs broadcast, multicast, a static address or a hard coded gateway.
  2. Build the fabric alongside. The new fabric runs in parallel, joined to the old network at the border with a Layer 3 handoff, plus a Layer 2 border for VLANs that must stay stretched during the move.
  3. Move by device class and closet. Not building by building. Each move has a rollback, which for a closet can be as simple as patching back.
  4. Open first, enforce later. Authentication starts in monitor mode and moves to closed mode once the profiling and policy are proven.
  5. Cut the old gateway last. The old gateway stays until the last device in that subnet has moved, then the Layer 2 handoff is removed and recorded as closed.
  6. Agree windows with OT. Plant and building systems move only in agreed windows, with device by device checks for anything that must not fail.

When a device misbehaves

  1. Detect. An Assurance anomaly, a firewall or IDS (intrusion detection system) alert, Cyber Vision or Secure Network Analytics. IoT networks are quiet, so unexpected traffic out is a strong signal.
  2. Contain. ISE Adaptive Network Control quarantines the endpoint, and a CoA moves it into a quarantine group at once. A firewall or SIEM (security information and event management) can trigger this automatically over pxGrid.
  3. Investigate. Path trace, flow records, and the device's firmware version and certificate.
  4. Recover. Rebuild or replace the device, rotate its credentials or certificate, then release it from quarantine. If the traffic was legitimate, I fix the profile or the policy instead.
  5. Learn. Update the runbook, and tighten the group policy or move the device class behind a gateway.
Troubleshooting

Finding a fabric fault plane by plane

I troubleshoot a fabric the same way every time: is the underlay healthy, has the edge learned and registered the device, does the policy allow it, and does the path out of the fabric work? Choose a symptom to see where it usually fails. Catalyst Center Assurance shows each client's onboarding timeline, which often shows the cause before I log in to a switch.

An SD-Access fault finder Six checks in order: port and power, authentication, pool and DHCP, LISP registration, group policy and the path between virtual networks. For each symptom, test traffic flows as far as the stage that fails and stops there, and a panel shows what I see and what I check. Portlink and PoE Auth802.1X or MAB PoolDHCP, flooding LISPregistration PolicySGACL Inter VNborder, fusion What I seeWhat I check Symptom: Authorised, but no IP address Session authorised, pool returned, no DHCP offershow device-tracking database Symptom: Connected with the wrong access Authorised with the wrong group or endpoint profileshow authentication sessions interface Gi1/0/12 details Symptom: New devices fail after an ISE outage RADIUS server marked dead, critical VLAN appliedshow aaa servers Symptom: Devices cannot discover each other BACnet Who-Is broadcasts never reach the other poolLayer 2 flooding setting on the pool in Catalyst Center Symptom: Has an address, reaches nothing No registration on the control plane nodeshow lisp instance-id 4099 ipv4 server Symptom: One pair of groups blocked SGACL drop counters rising for this pairshow cts role-based counters Symptom: Cannot reach another VN Reaches the border, no route back from the fusionshow ip route vrf IOT

Swipe sideways to see the whole diagram.

Authorised, but no IP address. The session is up, so I check the pool: the returned VLAN name must exist on this edge for this VN. Then DHCP: the relay on the anycast gateway, the route from the DHCP server back into the VN through the fusion device, and option 82 handling on the server. show device-tracking database shows whether the switch saw any address at all.

Wrong access. show authentication sessions interface Gi1/0/12 details shows the method, the result and the group. In ISE live logs I check which rule matched. Usually the device was profiled as something else, or matched a rule above the one intended. Fix the profile or rule, then send a CoA.

After an ISE outage. Existing sessions survived, but every port that reauthenticated failed. The fix is design: two policy service nodes per region with dead server detection, critical authentication on the templates, and timers long enough that a short outage is invisible. show aaa servers shows the RADIUS state.

Discovery fails. Some building and IoT devices find each other by broadcast, which the fabric drops by default, and devices in different pools are in different broadcast domains anyway. I enable Layer 2 flooding on that pool, which needs underlay multicast, or use a protocol fix such as a BACnet broadcast management device.

Reaches nothing. The edge learned the device but the control plane node does not know it. I check show lisp session on the edge, then show lisp instance-id 4099 ipv4 server on the control plane node. No registrations usually means the edge cannot reach the control plane node across the underlay.

One pair blocked. show cts role-based counters shows drops for that source and destination group. I check the matrix cell in ISE and that the edge has downloaded the current policy with show cts role-based permissions. Catalyst Center path trace shows the hit too.

Cannot reach another VN. Traffic between VNs leaves through the border to the fusion device. I check the border's BGP session for that VN, then the routes on the fusion device: show ip route vrf IOT. A missing leak or a firewall rule is the usual cause.

Animation

Commands by check

What I want to knowCommand on the edge or border
Is the device authenticated, and with what result?show authentication sessions interface Gi1/0/12 details, or show access-session on IBNS (Identity Based Networking Services) 2.0
Did the switch learn its address?show device-tracking database
Is the edge registered with the control plane node?show lisp session, show lisp instance-id 4099 ipv4 database
Which policy is downloaded, and what is hitting it?show cts environment-data, show cts role-based permissions, show cts role-based counters
Is the underlay healthy?show isis neighbors, ping 10.0.250.12 source Loopback0 size 1550 df-bit
Is the border handing off this VN?show ip bgp vpnv4 vrf IOT summary, show ip route vrf IOT
Is power the problem?show power inline, show interfaces status
Engagement

How I run an SD-Access programme, from discovery to steady running

A campus fabric changes how identity, policy and operations work, not just the switches. I run it as a programme with the security, desktop and facilities teams, because they own most of the devices that will join it.

How an engagement runs, from first conversation to lifecycle Seven stages from left to right: engage (stakeholders), requirements (FR and NFR), discovery (facts first), design (HLD and LLD), build and test (lab and pilot), support (handover), lifecycle (keep current). For each stage the panel shows who I work with and what comes out of it. After the lifecycle stage the loop returns to the start. Engagestakeholders RequirementsFR and NFR Discoveryfacts first DesignHLD and LLD Buildlab and pilot Supporthandover Lifecyclekeep current Step 1 · EngageAgree what segmentation must achieveWho I work withWhat comes outNetwork and security leadsDesktop and identity teamsFacilities and OT ownersGoals for segmentation and accessSites and buildings in scopeRisks, licences and constraintsDecision log and RACI Step 2 · RequirementsWho may reach what, and how wellWho I work withWhat comes outApplication ownersSecurity and riskService deskVirtual networks and groupsGroup policy matrix, first draftScale, resilience and timersOnboarding rules per device class Step 3 · DiscoveryEvery device class and how it talksWho I work withWhat comes outSite and facilities contactsDesktop and print teamsOT and building engineersEndpoint inventory by classBroadcast and multicast needsCurrent VLANs, ACLs and gatewaysSwitch and licence readiness Step 4 · DesignSites, pools, VNs and the matrixWho I work withWhat comes outDesign authorityIdentity architectsCisco or partner specialistsFabric site and transit designIP pools and VN handoffsISE deployment and policy designMigration and rollback plan Step 5 · Build and testOne closet, then one buildingWho I work withWhat comes outEngineers and partnerPilot users and ownersChange advisory boardLab tested fabric and policyFailure test resultsPilot sign off by device classWave plan by closet Step 6 · SupportAssurance, runbooks and a clear ownerWho I work withWhat comes outService desk and NOCIdentity and security teamProblem managerAssurance dashboards and alertsOnboarding runbooks per classPolicy change processHypercare exit criteria Step 7 · LifecycleSoftware, ISE and policy stay currentWho I work withWhat comes outVendor account teamIdentity teamDesign authorityCompatibility matrix trackingUpgrade waves for switches and ISEQuarterly policy reviewEnd of life register

Swipe sideways to see the whole diagram.

Step 1
  1. I start with what the business needs from segmentation: containing threats, onboarding IoT safely, or meeting an audit finding. I agree who owns ISE, who owns the device groups and who decides on policy (a RACI: responsible, accountable, consulted, informed).

  2. Functional requirements become virtual networks, groups and a draft policy matrix. Non functional ones set the scale per site, the recovery time when a node fails, what happens when ISE is unreachable, and how quickly a new device must be onboarded.

  3. I build an inventory of every device class from switch tables, ISE profiling in monitor mode and walk rounds. For each class I record how it gets an address, whether it needs broadcast or multicast, and who owns it. Surprises here are what derail fabric rollouts.

  4. The high level design (HLD) covers fabric sites, transits, roles, VNs and the fusion design. The low level design (LLD) covers pools, handoff VLANs, BGP, ISE personas, templates and the full matrix. Both go to the design authority.

  5. I prove the design in a lab, including node, ISE and transit failures, then pilot one closet in open authentication. Device owners sign off each class before enforcement, and the rollout moves closet by closet with a rollback for each.

  6. Operations get Catalyst Center Assurance, runbooks for the common onboarding faults and a clear owner for policy changes. Service desk staff learn to read the onboarding timeline, which answers most "my device will not connect" calls on its own.

  7. Catalyst Center, switch software and ISE must stay on versions that are supported together. I plan upgrades in waves against the compatibility matrix, review the policy matrix every quarter and retire groups nobody uses.

Functional requirements

What it must do.

  • Authenticate every user and device port, with 802.1X where possible and MAB with profiling where not.
  • Keep staff, guests, IoT, building systems and OT in separate virtual networks.
  • Control access between groups inside each network with a policy matrix.
  • Inspect every flow between virtual networks at a firewall.
  • Give wired and wireless users the same policy wherever they connect.
  • Onboard new switches and access points automatically.

Non functional requirements

How well it must do it, with a number wherever one can be agreed.

Availability
No single node failure takes down a site: two control plane nodes and two borders per site.
Recovery
Traffic moves to the surviving border or path within seconds of a failure.
Resilience to ISE
Ports fall into a known critical state if ISE is unreachable, with two policy service nodes per region.
Scale
Each site within the published limits for endpoints, edges and VNs, with 30% headroom.
Onboarding
A new device class can be profiled, grouped and enforced within an agreed number of days.
Operability
Health, onboarding and policy hits visible in Catalyst Center, and every policy change logged.

These are typical starting targets for a fabric site. Each organisation sets its own, and each number is traced to a failover or onboarding test in the pilot.

Projects

Where I have designed and built it

SD-Access has been part of my campus work for years, usually alongside Catalyst SD-WAN, with ISE behind both.

AVEVA · 2020 to 2021

A global LAN and WAN rebuild

Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center through the reverse merger, modernising every site worldwide.

Vodafone · 2024

Fabric planning at scale

Shaped SD-WAN, SD-Access and Catalyst Center infrastructure planning so backbone, data centre and edge environments could carry high throughput, low latency workloads.

XIDA · 2021 to 2024

Fabric designs for clients

Produced multi vendor SD-WAN and SD-Access designs for enterprise clients.

XIDA · 2025 to now

Discovery to provisioning

Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access and AWS (Amazon Web Services), Azure and Google Cloud environments.

iSystems Integration · three engagements, 2015 to 2019

Architecture and delivery

Directed architecture and delivery across cloud, disaster recovery, Catalyst SD-WAN, SD-Access and security, and led and mentored a team of 4 engineers.

Home lab

Cisco ISE and wireless

My lab runs Cisco ISE for network access control and a virtual wireless controller on its ESXi hosts, which I use to test authentication and onboarding policy. See the lab.

Backed by CCNP (Cisco Certified Network Professional) Enterprise and the Cisco Certified Specialist exams in Enterprise Core, Enterprise Design and Enterprise Advanced Infrastructure Implementation. See all credentials