Collision and broadcast domains
Every switch port is its own collision domain, and at full duplex there are no collisions at all. A VLAN is one broadcast domain. Routers, and Layer 3 switches, stop broadcasts.
Switching and campus LAN
How a switched network stays fast and loop free
Every campus I design rests on the same Layer 2 ideas: how a switch learns, how VLANs keep traffic apart, how spanning tree stops loops, and how links and gateways stay up when something fails. Each idea has a diagram you can step through, and the last section shows how I find faults.
How switches learn
A switch learns from every frame it sees
A switch forwards Ethernet frames by MAC (media access control) address. It builds its table by itself, from the source address of every frame that arrives, and floods only when it has to. Everything else on this page builds on this one idea.
Swipe sideways to see the whole diagram.
A switch learns from the source address of every frame. When host A sends its first frame, the switch records A's MAC address against port 1 in its MAC address table, also called the CAM (content addressable memory) table.
The switch has never seen B, so it sends the frame out of every other port in the VLAN. This is unknown unicast flooding. B keeps the frame, and C and D drop it.
B replies. The switch learns B on port 2, and it already knows where A is, so the reply goes out of port 1 only.
From now on, frames between A and B pass only between their two ports. Other ports never see them. An entry ages out after 300 seconds of silence, the Cisco default.
A broadcast, such as an ARP (address resolution protocol) request from C, always reaches every port in the VLAN. The switch learns C on port 3 as it goes. A VLAN is a broadcast domain, and that is why a loop hurts so much: the same broadcast keeps circling.
Frames have no time to live
An IP packet carries a TTL (time to live) that drops it after a set number of hops. An Ethernet frame has nothing like it, so a frame in a Layer 2 loop circles until the loop is broken.
Multicast in a VLAN
Without help, a switch floods multicast like a broadcast. IGMP (Internet Group Management Protocol) snooping sends each group only to ports that joined it. It needs an IGMP querier in the VLAN. Routing of multicast is on the routing page.
VLANs and trunks
Many networks on the same switches
A VLAN (virtual LAN) splits one physical switch network into separate broadcast domains, usually one VLAN per IP subnet. An access port belongs to one VLAN. A trunk carries many VLANs between switches, and marks each frame with an 802.1Q tag so the far end knows where it belongs.
Swipe sideways to see the whole diagram.
PC A sends an ordinary frame with no VLAN information. The switch knows its port is in VLAN 10, so the frame belongs to VLAN 10.
To cross the trunk, Switch 1 inserts a 4 byte 802.1Q tag holding VLAN ID 10. VLAN 20 frames cross the same cable with their own tag, so the two never mix.
Switch 2 reads the tag, removes it, and sends the frame only to VLAN 10 ports. PC B receives an ordinary frame. It never knows the trunk exists.
One VLAN per trunk crosses untagged: the native VLAN, VLAN 1 by default on Cisco. If the two ends disagree, frames leak between VLANs. I set an unused native VLAN, such as 999, and allow only the VLANs each trunk needs.
The tag
The tag sits after the source MAC address. It holds a TPID (tag protocol identifier) of 0x8100, a 3 bit priority (PCP, the Layer 2 class of service), a DEI (drop eligible indicator) bit and a 12 bit VLAN ID. That gives 1 to 4094 usable VLANs, because 0 and 4095 are reserved. Adding or removing the tag means the switch recalculates the FCS (frame check sequence) at the end of the frame.
How I build a trunk
Static trunk mode, with switchport nonegotiate to turn off DTP (Dynamic Trunking Protocol). Native VLAN set to an unused ID. An explicit allowed list. I always change that list with switchport trunk allowed vlan add, because leaving out add replaces the whole list.
Why DTP stays off
DTP lets a port negotiate into a trunk. That is convenient, but an attacker who negotiates a trunk can reach every VLAN on it. Together with an untagged native VLAN, it also opens the door to double tagging, a form of VLAN hopping.
VTP and its one big risk
VTP (VLAN Trunking Protocol) copies VLANs from a server switch to clients. The highest revision number wins, so a lab switch with the same domain name and a higher revision can wipe every VLAN on a site. I run VTP transparent or off, or VTP version 3 with one primary server.
Routing between VLANs
There are three ways. A router on a stick uses one subinterface per VLAN, which is simple, but one link carries everything. An SVI (switched virtual interface, interface Vlan10) on a Layer 3 switch routes in hardware. A routed port, with no switchport, suits point to point links.
Voice VLAN
A phone and a PC often share one cable. switchport access vlan 10 and switchport voice vlan 110 put the PC in the data VLAN, untagged, and the phone's traffic in the voice VLAN, tagged. The switch tells the phone which VLAN to use over CDP (Cisco Discovery Protocol), or over LLDP-MED (LLDP with media endpoint discovery) for other makes.
Private VLANs
A private VLAN splits one subnet into separate pieces. Ports in an isolated VLAN can only reach the promiscuous port, usually the gateway or firewall. Ports in a community VLAN reach each other and the promiscuous port. Both sit inside a primary VLAN. It suits shared server and hosting segments. VTP must be transparent, or version 3.
SVI autostate
On Cisco, an SVI stays down unless its VLAN exists and is up on at least one port or trunk. A gateway that will not come up after a change often means the VLAN is missing from every trunk.
Spanning tree
One loop free path, with a spare ready to go
Redundant links are good, but in Layer 2 they make loops. Spanning tree (STP) elects a root, works out the best path to it from every switch, and blocks just enough ports to break every loop. If a link fails, a blocked port takes over.
Swipe sideways to see the whole diagram.
Each switch starts by claiming to be the root. It sends BPDUs (bridge protocol data units) every 2 seconds, carrying its bridge ID: a priority plus its MAC address.
The lowest bridge ID wins. I never leave this to chance: priority 4096 on the primary distribution switch and 8192 on its peer. With Rapid PVST+ (Rapid Per VLAN Spanning Tree Plus) the VLAN number is added, so 4096 shows as 4106 for VLAN 10.
Every other switch picks one root port: the lowest total cost to the root. Ties go to the lowest sender bridge ID, then the lowest sender port priority, then the lowest sender port number. In the diagram, RP is a root port, DP a designated port and ALT an alternate port.
Each link gets one designated port, the end nearest the root. Both ends of the link between Distribution 2 and the access switch are cost 4 from the root, so the lower bridge ID wins. The access switch's port becomes an alternate port and blocks.
When the link to Distribution 1 fails, RSTP (Rapid Spanning Tree, 802.1w) moves the alternate port straight to forwarding, usually in under a second. Classic 802.1D would wait 30 to 50 seconds.
Three versions you will meet
| Version | Standard | How it works | Where I use it |
|---|---|---|---|
| STP | IEEE 802.1D | States: blocking, listening (15 s), learning (15 s), forwarding. Timers: hello 2 s, max age 20 s, forward delay 15 s. A direct failure takes about 30 s, an indirect one about 50 s. | Only where old kit forces it. |
| RSTP and Rapid PVST+ | IEEE 802.1w, Cisco per VLAN | Roles: root, designated, alternate, backup. States: discarding, learning, forwarding. Proposal and agreement on point to point links converge in about a second. A neighbour is lost after three missed hellos, 6 s. | Most campus networks. One instance per VLAN lets odd and even VLANs use different roots. |
| MST | IEEE 802.1s | Multiple Spanning Tree maps many VLANs to a few instances. Instance 0, the internal spanning tree, carries the BPDUs. A region needs the same name, revision and VLAN map on every switch. | Large estates, and where per VLAN instances hit platform limits, such as 128 on the Catalyst 3750. |
Where the root goes
The root belongs where traffic meets, normally the distribution or core pair. I set priorities in multiples of 4096 rather than rely on spanning-tree vlan 10 root primary, which sets 24576, or 4096 below the current root, only once. The root and the active gateway for each VLAN sit on the same switch.
Topology changes
In 802.1D a switch sends a TCN (topology change notification) up its root port, the root flags the change for 35 seconds, and every switch cuts MAC ageing to 15 seconds. In RSTP, only a non edge port moving to forwarding counts, and MAC entries are flushed at once. Either way, a burst of flooding follows. A rising change count usually means a flapping access port without PortFast.
Configuring MST
spanning-tree mode mst, then under spanning-tree mst configuration: name CAMPUS, revision 1 and instance 1 vlan 10-20, the same on every switch in the region. spanning-tree mst 1 priority 4096 places the root for that instance. A port facing a Rapid PVST+ switch is a boundary port and runs PVST simulation. I keep the CIST (common and internal spanning tree) root inside the region. If a better PVST+ BPDU arrives at the boundary, the port blocks as PVST simulation inconsistent.
Long and short path costs
Short costs are 16 bit values: 19 for 100 Mb/s, 4 for 1 Gb/s and 2 for 10 Gb/s, so faster links barely differ. Long costs, from 802.1t, are 32 bit: 20,000 for 1 Gb/s, 2,000 for 10 Gb/s and 200 for 100 Gb/s. spanning-tree pathcost method long sets them, and every switch must match. Some newer IOS XE releases default to long, so I check show spanning-tree summary first.
The MST region trap
Add a VLAN to the instance map on one switch only, and that switch forms its own region. Traffic then crosses the boundary on the common spanning tree and paths change. I change the map on every switch in one planned window.
STP guards
Each guard has one place it belongs
Spanning tree trusts every BPDU it hears. The guard features stop the wrong device taking over the tree, and stop a broken link turning into a loop. Placement matters as much as the feature, so pick a guard to see where I put it.
Swipe sideways to see the whole diagram.
PortFast and BPDU Guard, on every host port. PortFast sends a host port straight to forwarding and stops it causing topology changes. BPDU Guard err-disables the port the moment a BPDU arrives, so a desk switch cannot join the tree. I set both globally with spanning-tree portfast edge default and spanning-tree portfast edge bpduguard default (without edge on older IOS), and I do not auto recover BPDU Guard, so someone goes and looks.
Root Guard, on distribution downlinks. If a switch below starts sending a better BPDU, Root Guard puts that port into root inconsistent state instead of letting the root move. It recovers by itself when the superior BPDUs stop. It never goes on a port that should lead towards the root.
Loop Guard and UDLD, on uplinks. If BPDUs stop arriving on a root or alternate port, perhaps because one fibre strand failed, Loop Guard holds the port in loop inconsistent state rather than forwarding into a loop. UDLD (UniDirectional Link Detection) catches the one way link itself. In normal mode it only logs a one way link and leaves the port up. In aggressive mode it retries 8 times, a second apart, then err-disables. It must run on both ends. Never put Loop Guard and Root Guard on the same port.
Storm control, on access ports. It caps broadcast and multicast at a set share of the link, so a faulty NIC (network interface card) or a loop at the edge cannot flood the VLAN. I avoid BPDU Filter on an interface: it stops sending and receiving BPDUs, which turns spanning tree off on that port and overrides BPDU Guard.
Hardening the access port
Beyond the guards: port security or 802.1X, DHCP (Dynamic Host Configuration Protocol) snooping, dynamic ARP inspection and IP source guard, unused ports shut and placed in a dead VLAN. How these work is on the security fundamentals page.
Bridge Assurance on Nexus
On Nexus switch to switch links, Bridge Assurance expects BPDUs in both directions. If they stop, the port blocks. It must be on at both ends, and it does a similar job to Loop Guard.
Recovering an err-disabled port
Causes include BPDU Guard, port security, UDLD, EtherChannel misconfiguration, DHCP snooping or ARP inspection rate limits, and storm control. show interfaces status err-disabled gives the reason. I remove the cause, then shutdown and no shutdown. Automatic recovery, after 300 seconds by default, suits some causes but hides a rogue switch.
EtherChannel
Several links that act as one
EtherChannel bundles up to eight active links into one logical port channel. Spanning tree sees one link, so nothing blocks, and the bundle survives a member failing. Traffic is shared by a hash, not round robin, which surprises people.
Swipe sideways to see the whole diagram.
Hashing. The switch hashes header fields, such as source and destination IP addresses, to pick a member. The same flow always takes the same link, so packets stay in order. One large flow never goes faster than one member, and a few flows can spread unevenly. I pick the hash with the most variety, for example port-channel load-balance src-dst-ip, adding Layer 4 ports where the platform allows.
A member fails. The hash is worked out again over the remaining members, and the failed link's flows move across within moments. Spanning tree does not notice, because the port channel stays up.
A mismatch. These pairs work: LACP active with active or passive, PAgP desirable with desirable or auto, and on with on. These never work: passive with passive, auto with auto, and on with LACP or PAgP. Members must also match speed, duplex, trunk mode, allowed VLANs and native VLAN. Mode on skips negotiation, so a cabling mistake can loop or black hole traffic. I use LACP active at both ends.
LACP and PAgP
LACP (Link Aggregation Control Protocol) is the IEEE standard, 802.3ad and now 802.1AX. PAgP (Port Aggregation Protocol) is Cisco's own. LACP allows 16 members: 8 active and 8 hot standby. Fast rate sends a PDU (protocol data unit) every second and times out in 3. Slow rate sends one every 30.
How I configure it
I configure the port channel interface first and let the members inherit it, then bundle with channel-group 1 mode active. Checks: show etherchannel summary (P means bundled, s suspended, I stand alone) and show lacp neighbor, where every member must see the same partner.
Servers and NIC teaming
MAC flapping on server ports usually means the server's NIC teaming and the switch disagree. IP hash teaming needs a static channel. LACP needs a LAG (link aggregation group) on the distributed switch. Originating port teaming needs no channel at all.
Stacks and VSS
Two or more switches, one to manage
Stacking, virtual switching and vPC all let one EtherChannel land on two physical boxes, so no uplink sits blocked. The difference is how many control planes they keep, and that decides how they fail.
Swipe sideways to see the whole diagram.
StackWise. Access switches joined by stack cables behave as one switch with one configuration and one management address. A Catalyst 9300 stack holds up to 8 members on StackWise-480. The active switch is elected by stack priority, 1 to 15 with 1 the default, then other tie breakers ending with the lowest MAC address. A standby takes over with SSO (stateful switchover). One failed cable leaves a half ring at half the bandwidth, so I treat it as urgent.
VSS and StackWise Virtual. With VSS (Virtual Switching System) or StackWise Virtual, two chassis act as one switch over a virtual switch link. One control plane is active and one is hot standby, but both forward traffic. Access switches bundle one link to each chassis as a multichassis EtherChannel, so nothing blocks. VSS is the Catalyst 6500, 6800 and 4500-E version. StackWise Virtual is the Catalyst 9000 version, on the core and distribution models such as the 9500 and 9600. If the link between them fails, dual active detection, such as enhanced PAgP or a fast hello link, shuts the old active down.
vPC (virtual port channel). Two Nexus switches keep separate control planes and configurations, joined by a peer link and a keepalive over a separate path. The server simply runs LACP. Because each switch is configured on its own, consistency checks matter. vPC failure cases and design live on the data centre networks page.
One family, many names
MC-LAG (multichassis link aggregation) is the general term for one bundle across two switches: Nexus vPC, VSS, StackWise Virtual and Junos MC-LAG. The downstream device just runs LACP. LLDP (Link Layer Discovery Protocol, 802.1AB) is the vendor neutral way to see who is connected, where CDP is Cisco's own.
Building a VSS
Both chassis get the same domain with switch virtual domain 100 and their own switch number. The VSL (virtual switch link) is a port channel, ideally two or more 10 Gb/s links on different modules, marked with switch virtual link 1. switch convert mode virtual then reloads both as one system. If the VSL fails and dual active detection fires, the old active enters recovery mode and shuts every port except the VSL until the link returns.
A split stack
Two failed stack cables can split a stack in two. Both halves then run the same configuration and addresses, and duplicate address messages appear. When they rejoin, an election runs and the losing half reloads, which is a second outage.
Replacing a member
A spare must run the same software, with a low stack priority and the right member number set while it stands alone. A spare with a higher priority, or an old configuration, can take over the stack. show switch and show switch stack-ports confirm it joined cleanly.
Gateway redundancy
The default gateway that never goes away
Hosts have one default gateway address. First hop redundancy protocols share that address between two routers or Layer 3 switches, so if one fails the other answers for it. The part most often missed is tracking the uplink, not just the router.
Swipe sideways to see the whole diagram.
HSRP (Hot Standby Router Protocol) gateways send hellos every 3 seconds and wait 10 seconds before deciding a peer is dead. The highest priority, 100 by default, becomes active. Users point at the virtual IP address, never at a real router.
Gateway 1 answers ARP for the virtual IP with a virtual MAC address, so user traffic goes to it and up to the core.
Gateway 1's own uplink fails, but it is still alive, so the hellos carry on. Tracking spots the lost uplink and lowers its priority from 110 to 90. The default decrement is only 10, which is a common reason tracking seems to do nothing.
Gateway 2 now has the higher priority, and because preempt is configured, it takes over. Preempt is off by default in HSRP. It sends a gratuitous ARP so the switches learn the virtual MAC on its port.
User traffic now leaves through Gateway 2. I keep the active gateway and the spanning tree root on the same switch for each VLAN, so traffic never takes an extra hop.
| HSRP | VRRP (Virtual Router Redundancy Protocol) | GLBP (Gateway Load Balancing Protocol) | |
|---|---|---|---|
| Owner | Cisco | Open standard (RFC 9568 for version 3, which replaced RFC 5798) | Cisco |
| Roles | Active, standby, listen | Active (formerly master) and backup | One gateway, the AVG (active virtual gateway), answers ARP, and up to four AVFs (active virtual forwarders) forward |
| Load sharing | One active per group, so I split VLANs across groups | One active router per group | Built in: the AVG hands out up to four virtual MACs in turn |
| Timers | Hello 3 s, hold 10 s | Advert 1 s | Hello 3 s, hold 10 s |
| Preempt | Off by default | On by default | AVG: off by default. AVF: on by default, after a 30 second delay |
| On the wire | UDP (User Datagram Protocol) 1985. Version 1 uses 224.0.0.2, groups 0 to 255 and virtual MAC 0000.0c07.acXX. Version 2 uses 224.0.0.102, groups 0 to 4095, IPv6 and virtual MAC 0000.0c9f.fXXX | IP protocol 112 to 224.0.0.18, virtual MAC 0000.5e00.01XX | UDP 3222 to 224.0.0.102, virtual MAC 0007.b400.XXYY |
Tracking done properly
I track the uplink line protocol, or better, an IP SLA (service level agreement) probe to something beyond the next hop. The decrement must take priority below the peer: 110 minus 20 gives 90, under 100. A short delay before preempting lets routing settle first.
vPC and anycast gateways
On a Nexus vPC pair, both switches forward traffic sent to the HSRP virtual MAC, so it works active and active. In a VXLAN EVPN fabric (Virtual Extensible LAN with an Ethernet VPN control plane) the gateway moves to every leaf as a distributed anycast gateway. That is on the VXLAN EVPN page.
When it flaps
HSRP flapping between active and standby usually means lost hellos: the VLAN missing from the trunk between the pair, a loop, high CPU or a one way link. Both showing active means the Layer 2 path between them is broken. show standby brief is where I start.
Campus design
Where Layer 2 ends decides how far a fault spreads
The biggest campus design choice is where switching stops and routing starts. It sets the size of every failure domain, how much spanning tree you depend on, and how easily devices can move.
Swipe sideways to see the whole diagram.
Three tier. Access switches connect users. Distribution pairs hold the gateways and the spanning tree root for their block. The core only routes between blocks, fast and simple. Each distribution block is its own failure domain. I use it on large multi building sites.
Collapsed core. On one building or a mid sized site, the core and distribution jobs share one pair, often a StackWise Virtual pair so the access uplinks form one EtherChannel and nothing blocks. Fewer boxes, but one larger Layer 2 domain.
Routed access. Every access switch routes, with OSPF or IS-IS on its uplinks. Spanning tree has nothing to block, equal cost paths use every uplink and a fault stays inside one closet. The cost is that a VLAN cannot span closets, so devices that move keep their address only with an overlay such as SD-Access.
How I build a site today
- Requirements first. Headcount, device density, phones and access points needing PoE (power over Ethernet), OT (operational technology) systems, growth and the local carriers decide the size, not the catalogue.
- A few site templates. Large, medium, small and manufacturing, each with a fixed design. Across a 100 site estate, a small set of templates is what makes support possible.
- No blocked uplinks. Stacked access with uplinks on different members, bundled to a StackWise Virtual pair or a routed distribution layer.
- Gateway, root and routing together. The active gateway, the spanning tree root and the preferred routing path for each VLAN sit on the same switch.
- Resilient branches. Two WAN (wide area network) transports with routing or SD-WAN failover, two routers with HSRP or VRRP on bigger sites, and a UPS (uninterruptible power supply) for closets that carry security and OT devices.
- Power budgeted. PoE per port and per switch, with headroom. 802.3af gives 15.4 W at the switch, 802.3at (PoE+) 30 W, and 802.3bt 60 W or 90 W. A device announces a class from 0 to 8 when it connects, and the switch reserves that class's maximum from its budget, unless a fixed value is set with
power inline static max. CDP or LLDP can then negotiate the real figure, which frees budget for other ports. A Wi-Fi 6E access point on 802.3af runs with features turned off.
Campus QoS
Mark it at the edge, trust it everywhere after
QoS (quality of service) only matters when a queue builds, usually where a fast link feeds a slower one. In the campus, the job is to classify and mark traffic as close to the source as possible, decide which devices to trust, and give voice a protected queue.
Swipe sideways to see the whole diagram.
The phone marks its voice packets EF (expedited forwarding), which is DSCP (differentiated services code point) value 46. A PC can mark its traffic EF too, by mistake or on purpose, to jump the queue.
The access port is the trust boundary. It trusts the phone, often only after CDP or LLDP confirms one is attached, and rewrites the PC's marking to 0. Everything beyond the access switch trusts the marking it receives.
At the uplink, EF goes into a strict priority queue. I cap that queue at about a third of the link so voice cannot starve everything else. Other classes get guaranteed shares.
An access point maps DSCP to Wi-Fi priorities (WMM, Wi-Fi Multimedia), so EF reaches the voice access category on the air. RFC 8325 sets out the mapping. One consistent policy end to end is what makes it work.
| Class | Marking | DSCP value | Treatment |
|---|---|---|---|
| Voice | EF | 46 | Strict priority, capped at about a third of the link |
| Interactive video | AF41 (assured forwarding) | 34 | Guaranteed bandwidth, or priority on some designs |
| Call signalling | CS3 (class selector) | 24 | Small guaranteed queue |
| Business critical data | AF31 or AF21 | 26 or 18 | Guaranteed bandwidth |
| Network control | CS6 | 48 | Protected, so routing and HSRP hellos survive congestion |
| Best effort | Default | 0 | The remaining bandwidth |
| Scavenger | CS1 | 8 | Whatever is left, such as bulk backups or updates |
AF values follow a pattern: AFxy has DSCP 8x plus 2y, where y is the drop precedence. Queuing on the WAN edge, with priority and class based queues, shaping and policing, is on the routing page.
Platforms
The Cisco platforms I design around
These are the Catalyst and ISR (Integrated Services Router) families I design around. The 3750 family runs in my home lab, alongside 3560G switches and 2811 routers. Older models are past end of support, so I use them to learn and test, and I design new sites on current platforms.
| Platform | Software | Stacking or pairing | What to know |
|---|---|---|---|
| Catalyst 3750, 3750-E and 3750-X | Classic IOS, last train 15.0(2)SE on the 3750 and 3750-E, 15.2(4)E on the 3750-X | StackWise 32 Gb/s or StackWise Plus 64 Gb/s, up to 9 members. StackPower on the 3750-X | QoS with mls qos commands. Limit of 128 spanning tree instances. Long past end of support. |
| Catalyst 3850 | IOS XE, last train 16.12 | StackWise-480, up to 9 members, with active and hot standby | QoS with MQC (the modular QoS command line). Cannot stack with a 3750. Reached end of support in 2025. |
| Catalyst 6500 and 6800 | IOS 15 SY | VSS between two chassis | Modular cores and distribution. Instant Access lets 6800IA switches act as remote line cards. |
| Catalyst 9300, 9400, 9500 and 9600 | IOS XE 17 | StackWise-480 or 1T on the 9300, StackWise Virtual on the core models | Current platforms. Programmable through NETCONF (Network Configuration Protocol) and RESTCONF (its web based equivalent), and the base for SD-Access. |
| ISR, ISR G2 and ISR 4000 routers | IOS 12.4 and 15, with technology licences from the G2, then IOS XE on the 4000 | HSRP or VRRP pairs | The ISR 4000 separates control and data planes and forwards on dedicated cores. |
IOS XE habits
IOS runs as a process on a Linux kernel, with install mode for upgrades and patching without a full reload on many releases. Many commands carry over from classic IOS, but QoS, platform and boot commands differ.
NX-OS habits
On Nexus, features stay off until enabled (feature lacp, feature hsrp), management sits in its own VRF (virtual routing and forwarding instance), and show commands work in configuration mode. Configuration checkpoints and rollback come built in.
Protecting the CPU
Control plane policing (CoPP) rate limits traffic sent to the switch CPU, so a flood cannot starve routing, HSRP or BFD (Bidirectional Forwarding Detection). High CPU with protocol flaps usually means something is being punted to it.
Troubleshooting
Switching faults I can find quickly
Most switching faults leave clear evidence in the logs and counters, if you know where to look. Here is how a Layer 2 loop builds and how I contain it, then the faults I see most, with the commands I use.
Swipe sideways to see the whole diagram.
Someone patches a desk switch into two wall ports on different access switches. Those ports have BPDU Filter configured, so spanning tree never sees the loop. A single broadcast starts to circle.
Every pass copies the broadcast again, so links fill within seconds. MAC addresses flap between ports, CPU hits 100% and HSRP and routing neighbours drop. Management sessions fail too, so I go to the console or the out of band network.
I contain first and diagnose second. The MAC flap messages and the topology change count point at a port. I shut it, starting at the edge, and note the time for the incident record.
Without BPDU Filter, BPDU Guard on every access port would have shut that port the moment it saw a BPDU. Storm control limits the damage from what BPDU Guard cannot see. The cause goes into a problem record.
Fault finder
Pick a symptom to see the likely causes, the commands I run and the fix. Commands are for Cisco IOS and IOS XE unless I say otherwise.
Everything is slow and CPU is highLayer 2 loop or broadcast storm
- What I see
- The whole site is slow, switch CPU is near 100%, links are full of broadcast and multicast, and neighbours are flapping.
- Likely causes
- BPDU Filter on an interface, a desk switch or hub bridging two ports, a VM (virtual machine) bridging two NICs, a one way fibre, spanning tree turned off on a VLAN, or PortFast on a link between switches.
- What I check
show processes cpu sorted,show interfaces | include line protocol|broadcasts,show logging | include MACFLAP|SPANTREE,show spanning-tree detail | include ieee|occurr|from|is execto follow the topology changes to their source.- Fix
- Shut the port the evidence points to. With no access at all, shut downlinks one at a time from the core console while watching CPU and broadcast rates.
- Prevention
- BPDU Guard on access ports, Loop Guard or UDLD on uplinks, Root Guard on downlinks, storm control, no interface BPDU Filter.
MAC flapping messages in the logOne address seen on two ports
- What I see
%SW_MATM-4-MACFLAP_NOTIFnaming one MAC address moving between two ports.- Likely causes
- One server MAC on two ports means NIC teaming without a matching channel. A VM address moving once after vMotion is normal. Many addresses, including the gateway, flapping between two uplinks means a loop. The gateway MAC alone flapping means HSRP dual active or a vPC fault.
- What I check
show mac address-table address 0050.56aa.0101,show logging | include MACFLAP, then CDP or LLDP to see what sits on each port.- Fix
- Match teaming to the switch: IP hash with a static channel, LACP with a LAG, or originating port with no channel. Treat a loop as a loop.
- Prevention
- A server connection standard agreed with the virtualisation team.
A VLAN does not reach the far switchTrunk mismatch or missing VLAN
- What I see
- Hosts in one VLAN cannot reach their gateway or each other across switches. Other VLANs work.
- Likely causes
- The VLAN is not in the allowed list on one end, not created on a switch, pruned, or one end is not trunking at all. A recent change may have used
switchport trunk allowed vlan 40withoutadd, replacing the list. - What I check
show interfaces trunkon both ends (allowed, active and forwarding columns),show vlan brief,show interfaces Gi1/0/49 switchport.- Fix
- Add the VLAN with
switchport trunk allowed vlan add 40, create it where missing, and on a vPC pair add it to the peer link and both peers too. - Prevention
- Use
addorremoveevery time, and set areload inor configuration revert timer before changing a remote uplink.
Odd reachability and a native VLAN warningNative VLAN mismatch
- What I see
%CDP-4-NATIVE_VLAN_MISMATCH, hosts in one VLAN reaching another VLAN's subnet, or VLANs blocked as port VLAN ID inconsistent.- Likely causes
- The two ends of a trunk use different native VLANs, so untagged frames land in different VLANs at each end.
- What I check
show interfaces trunkon both ends,show spanning-tree inconsistentports,show logging | include NATIVE_VLAN.- Fix
- Set the same unused native VLAN on both ends.
- Prevention
- A trunk template with native VLAN 999, or
vlan dot1q tag nativeso every frame is tagged.
A user's port keeps going downerr-disabled
- What I see
- The port shows err-disabled, often with the reason
bpduguardorpsecure-violation. - Likely causes
- A small switch, docking station or VM bridging NICs sends BPDUs, or a desk move breaks sticky MAC port security.
- What I check
show interfaces status err-disabled,show logging | include BPDUGUARD|ERR_DISABLE,show port-security interface Gi1/0/12, then I go and look at what is plugged in.- Fix
- Remove the device, then
shutdownandno shutdown. For a desk move, clear the sticky entries on the old port. - Prevention
- No automatic recovery for BPDU Guard, and a desk move process that clears port security.
The link is up but slowDuplex mismatch
- What I see
- The link works but is slow, and worse on large transfers. Late collisions and CRC (cyclic redundancy check) errors climb.
- Likely causes
- One end hard set to full duplex, the other on auto, which falls back to half duplex at 10 or 100 Mb/s.
- What I check
show interfaces Gi1/0/5 | include duplex|CRC|collisionon both ends,show interfaces status(a-fullmeans negotiated), and%CDP-4-DUPLEX_MISMATCH.- Fix
- Auto on both ends, or hard set both ends the same.
- Prevention
- A standard of auto negotiation on both ends. Gigabit copper needs auto negotiation anyway.
A port channel member will not bundleEtherChannel suspended or stand alone
- What I see
- In
show etherchannel summarya member shows s (suspended) or I (stand alone), or the ports are err-disabled for channel misconfiguration. - Likely causes
- I means no LACP partner, so the far end is not configured or is cabled elsewhere. s means the member's configuration differs (VLANs, speed, mode). A different partner ID means the cable goes to another switch. Mode on facing a plain port triggers the misconfiguration guard.
- What I check
show etherchannel summary,show lacp neighbor,show interfaces Gi1/0/50 switchport,show logging | include EC-5. On Nexus:show port-channel summary.- Fix
- Reset the member with
default interface, add it back withchannel-group 1 mode active, and match the far end. - Prevention
- Configure the port channel interface and let members inherit, with LACP active on both ends.
One VLAN is slow, idle ports see other people's trafficUnicast flooding
- What I see
- High output on many access ports in one VLAN, and a capture on an idle port shows unicast for other hosts.
- Likely causes
- Asymmetric routing: HSRP is active on one switch but return traffic arrives through the other. That switch's ARP entry lasts 4 hours, but its MAC entry ages out after 300 seconds, so it floods.
- What I check
show ip arp 10.1.10.25(present) andshow mac address-table addressfor the same host (missing) on the second switch,show standby brief,show spanning-tree root.- Fix
- Align the active gateway, the spanning tree root and the routing for each VLAN, or set the ARP timeout below the MAC ageing time on the SVI.
- Prevention
- A design standard that keeps gateway, root and routing together.
VLANs disappear across a siteVTP revision overwrite
- What I see
- VLANs vanish and access ports go inactive, all at once.
- Likely causes
- A switch with the same VTP domain and a higher configuration revision joined a trunk and overwrote the VLAN database.
- What I check
show vtp statuson several switches: a jump in the revision and the address that made the last change.- Fix
- Restore the VLANs on the real server and set the newcomer to transparent.
- Prevention
- VTP transparent or off, or VTP version 3 with a primary server.
Seeing the traffic itself
SPAN
SPAN (Switched Port Analyzer) is port mirroring on one switch. monitor session 1 source interface Gi1/0/5 both and monitor session 1 destination interface Gi1/0/48 copy a port's traffic to an analyser. The destination port stops acting as a normal port while the session runs.
RSPAN
RSPAN (remote SPAN) carries the copy to another switch in a VLAN set aside with remote-span. The source switch uses destination remote vlan 900, and the switch with the analyser uses source remote vlan 900. The VLAN must be allowed on every trunk between them.
ERSPAN
ERSPAN (encapsulated remote SPAN) wraps the copy in GRE (generic routing encapsulation), so it can reach an analyser anywhere across a routed network. On the Catalyst 9000, monitor capture can also capture on the switch itself.
My general method, from defining the problem to closing the record, is on the troubleshooting page.
Projects
Where I use these every day
Campus switching is where I started, hands on, and it still sits under every design I lead.
Three sites on physical Catalyst kit
My home lab runs Catalyst 3750 and 3560G core switches with VLANs and trunks at three sites, so I can break and rebuild Layer 2 designs safely.
EMEA network refresh
Audit, high and low level design and implementation across LAN, WAN and data centre, replacing end of life kit with no service outages.
Global LAN and WAN modernisation
Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center, and upgraded every site worldwide remotely during the lockdowns.
Where I learnt it hands on
LAN, WAN, VoIP (voice over IP), virtualisation and security across the group, including the warehouse and food depot networks.
Backed by CCNP Enterprise, CCNP Routing and Switching, CCNA Routing and Switching, and the CCIE Routing and Switching written exam. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.