Cisco IOS and Catalyst

Cisco IOS and Catalyst, from first boot to rollback

I have built networks on Cisco IOS (Internetwork Operating System) throughout my career, from telecoms networks to Catalyst 9000 campuses, ISR (Integrated Services Router) branches and SD-WAN (software defined wide area network) edges. This page shows how the software is built, how devices boot and upgrade, how stacks survive failures and how packets are forwarded. It also covers how I harden, automate and troubleshoot them.

IOS and IOS XE
Catalyst 9000
ISR and Catalyst 8000
Install mode upgrades
StackWise stacks
Packet forwarding
Model driven telemetry
Hardening

Start with the IOS family Where I have used it

Live view · a branch on Catalyst and IOS XE

The IOS family

One command line, several operating systems underneath

Cisco IOS (Internetwork Operating System) has run Cisco routers and switches for decades. Today the name covers several systems. Classic IOS is still on older kit. IOS XE runs current Catalyst switches and routers, with IOS itself as one process, IOSd (the IOS daemon), on a Linux kernel and an ASIC (application specific integrated circuit) doing the forwarding. CatOS once ran the big Catalyst chassis. Knowing which one a device runs tells me how it boots, how it upgrades and how it fails.

Swipe sideways to see the whole diagram.

Classic IOS is one image that runs as a single program. Routing, the CLI (command line interface), drivers and every other task share one memory space, and a scheduler lets each task run until it hands back control. It is small and fast, but one faulty task can corrupt memory and reload the whole device. The 2811 routers and 3750 switches in my lab run it.

IOS XE runs on a Linux kernel. IOSd, the IOS daemon, still holds routing and the CLI, so the commands look the same, but it is one process among many. FED (the forwarding engine driver) writes routes into the ASIC (application specific integrated circuit), platform managers look after power and fans, and a database feeds the programming interfaces. A fault in one process need not take the device down.

CatOS ran the switching side of older Catalyst chassis such as the 4000, 5000 and 6500. In hybrid mode the supervisor ran CatOS for ports and spanning tree, while its routing card, the MSFC (multilayer switch feature card), ran IOS. That meant two command lines, two configs and session 15 to jump between them. Native mode put one IOS image on both, and CatOS is long retired.

Animation

Cisco operating systems today

Operating system	How it is built	Where it runs	What it is like to use
IOS (classic)	One image, one shared memory space	ISR G1 and G2 routers such as the 2811 and 2911; Catalyst 2960, 3560 and 3750 switches	The original `show` and `configure terminal` CLI. Changes apply the moment I press Enter.
IOS XE	IOSd and other processes on a Linux kernel	Catalyst 9000 and C9000 switches, Catalyst 8000 and ISR 1000 and 4000 routers, ASR (Aggregation Services Router) 1000, Catalyst 9800 wireless controllers	The same CLI as IOS, plus install mode packages, programmable interfaces and a guest shell.
IOS XR	Distributed processes on Linux	Service provider routers: ASR 9000, NCS (Network Convergence System) and Cisco 8000	A candidate config and `commit`, closer to Junos than to IOS.
NX-OS	Modular processes on Linux	Nexus data centre switches	An IOS style CLI with `feature` commands, checkpoints and vPC. It has its own page.
CatOS	Its own image, separate from IOS	Catalyst 4000, 5000 and 6500 supervisors (retired)	`set`, `clear` and `show` commands, with no configure mode.

Read the version first

show version tells me the operating system, the release, the uptime and why it last reloaded. Half of all upgrade and support decisions start from that one screen.

Same commands, different behaviour

A command that works on a 3750 may work differently, or not at all, on a Catalyst 9300. Hardware programming, defaults and licensing changed with IOS XE, so I test configs on the target release.

Process faults show up differently

On IOS XE, show processes cpu platform sorted shows Linux processes as well as IOSd. A busy FED process points at the ASIC, a busy IOSd at routing or the CLI.

Platforms

Catalyst switches and routers, matched to the job

Current Cisco campus and branch kit almost all runs IOS XE, but the hardware underneath differs. Catalyst 9000 switches forward in UADP (Unified Access Data Plane) ASICs or Cisco Silicon One. Branch routers forward in software on multicore processors, and WAN (wide area network) aggregation routers use the QFP (Quantum Flow Processor). The hardware sets the limits: table sizes, upgrade options and throughput with services turned on.

Swipe sideways to see the whole diagram.

Animation

Live view: branch traffic crosses the WAN edge to the core, while campus users reach the core and the WAN.

Platforms and what forwards the packets

Role	Platform	Forwarding hardware	What I note
Small closets and branches	Catalyst 9200 and 9200L	UADP	Runs the lighter `cat9k_lite` image. Stacks of up to 8 with StackWise-160, or StackWise-80 on the 9200L.
Main access closets	Catalyst 9300 and 9300X	UADP	Stacks of up to 8 with StackWise-480, or StackWise-1T on the 9300X. Models with UPOE+ (Cisco Universal Power over Ethernet Plus) give 90 W per port.
Large access, one chassis	Catalyst 9400	UADP	Dual supervisors, so it can upgrade with ISSU (in service software upgrade).
Distribution and core	Catalyst 9500 and 9500X	UADP, Silicon One on the 9500X	Fixed switches, usually paired with StackWise Virtual or routed links.
Large campus core	Catalyst 9600 and 9600X	UADP, Silicon One on the 9600X	Modular, with redundant supervisors and StackWise Virtual.
Next generation switching	C9350, C9550 and C9610 smart switches	Silicon One	IOS XE again. Cisco positions them as the next step from the 9300, 9500 and 9600.
Small branch	ISR 1000	Multicore processor	Fixed router with built in switch ports.
Branch	Catalyst 8200 and 8300	Multicore processor, cores set aside for forwarding	Replaced the ISR 4000. Runs in autonomous mode or as a Catalyst SD-WAN edge.
WAN aggregation and internet edge	Catalyst 8500 and ASR 1000	QFP	Headends, carrier hand offs and high throughput crypto.
Cloud and virtual	Catalyst 8000V	x86 server processor	Runs on hypervisors and in public clouds. Replaced the older CSR (Cloud Services Router) 1000v.

End of life milestones I plan around

Product line	End of sale	Last date of support	Cisco's suggested move
Cisco 2800, including the 2811 in my lab	1 November 2011	31 October 2016	ISR G2, then ISR 4000, now Catalyst 8000
Catalyst 3850	30 October 2020	31 October 2025	Catalyst 9300
Catalyst 2960-X	31 October 2022	31 October 2027	Catalyst 9200L
ISR 4000, main models	7 November 2023	30 November 2028	Catalyst 8200 and 8300

Dates from Cisco end of life notices. A notice also sets end of software maintenance and end of security fixes, which arrive years before the last date of support.

Licensing

Base licence

Catalyst 9000 switches carry a perpetual Network Essentials or Network Advantage licence. Advantage adds the larger routing and fabric feature set on models that support it.

Subscription

A term subscription, sold as DNA (Digital Network Architecture) Essentials or DNA Advantage, sits alongside and covers Catalyst Center features. Cisco has repackaged these tiers more than once, so I check the current names and terms before I order.

Smart Licensing Using Policy

Since IOS XE 17.3 a device runs first and reports later. Usage goes to CSSM (Cisco Smart Software Manager) directly, through CSLU (Cisco Smart License Utility), through Catalyst Center or offline for closed sites. show license summary shows what is in use.

Choices I make

Stack, chassis or pair

I use 9300 stacks for closets of up to eight switches, a 9400 chassis where one closet needs dual supervisors, and 9500 or 9600 pairs at distribution and core. Stacks are cheaper; a chassis upgrades with less disruption.

Install mode, extended releases

Every Catalyst 9000 runs in install mode on an extended support release, one standard release per platform family, with the next one under test in the lab.

Autonomous or SD-WAN

Catalyst 8000 and ISR 1000 routers run either as classic IOS XE routers or as Catalyst SD-WAN edges, switched with controller-mode enable. I choose per estate, never per site. The fabric side is on the SD-WAN page.

Buy for the life of the box

Campus kit typically stays in service for seven years or more, so I size uplinks, PoE (Power over Ethernet) budgets and table space for the end of that life, not for day one. Spanning tree and campus design rules are on the switching page.

Boot

From power on to a working config

When a device will not come back after a reload, the fault sits at whichever boot stage failed. Knowing the stages, and the settings that steer them, turns a dead box at 2am into a short ROMMON (read only memory monitor) session. Step through a boot on a classic router and on a current switch.

Swipe sideways to see the whole diagram.

Step 1

The device tests its memory, processor and interfaces. Current hardware adds secure boot: each stage checks that the next is signed by Cisco before running it, so a tampered image will not load.
ROMMON is the small boot loader. On routers the config register decides what happens next: 0x2102 boots normally and loads the startup config, and 0x2142 skips the startup config for password recovery. Catalyst 9000 switches use ROMMON variables instead.
The boot variable names the image. In install mode it points at packages.conf, a list of the package files to load. In bundle mode it points at one .bin file. If the variable is empty or wrong, the device tries the first image it finds, or stops at ROMMON.
Classic IOS decompresses its image into RAM (random access memory) and runs it. IOS XE boots a Linux kernel and mounts its packages. Install mode boots faster because nothing has to be unpacked at each boot.
Processes start and the ASICs are programmed. In a stack, the members elect an active and a standby switch before ports come up, so a full stack takes longer to boot than a single switch.
The startup config loads from NVRAM (non volatile RAM). A new device with no config can join Catalyst Center through Plug and Play, or fetch a Python script through ZTP (zero touch provisioning), using DHCP (Dynamic Host Configuration Protocol) to find it.

Boot and recovery settings I check

Task	Command	What it does
Normal router boot	`config-register 0x2102`	Boots the image named by `boot system` and loads the startup config.
Router password recovery	`confreg 0x2142` then `reset` in ROMMON	Boots without the startup config. I copy it into the running config, set a new secret and put the register back to 0x2102.
Catalyst 9000 password recovery	`SWITCH_IGNORE_STARTUP_CFG=1` then `boot` in ROMMON	The same idea on switches. I reset the variable to 0 afterwards.
Stop recovery	`no service password-recovery`	Anyone with the console can then recover only by erasing the config. I use it where the console is not physically secure.
Check how it booted	`show version`, `show boot`	The image, the config register or boot variables, the uptime and the reason for the last reload.
Check install mode	`show install summary`	The packages and their states. A device in bundle mode shows a .bin file instead.

Recovery needs console access, which is why I treat console ports, terminal servers and out of band networks as privileged access. My lab has a terminal server for exactly this, a 2811 with a console cable to every device.

Upgrades

Upgrades that can undo themselves

Install mode is Cisco's recommended way to run IOS XE. It splits the image into packages, upgrades in clear stages and keeps a way back. Upgrade outages usually come from skipped checks rather than bad software, so I follow the same stages every time, and pick releases from Cisco's extended support trains.

Swipe sideways to see the whole diagram.

Step 1

I copy the new image into flash and check its hash with verify /md5 against the value Cisco publishes. Before copying, install remove inactive clears old packages, because one stack member short of space fails the whole upgrade.
install add file flash:cat9k_iosxe.17.18.04.SPA.bin expands the image into package files and checks them. The new packages sit inactive, state I. Nothing running has changed yet.
install activate rewrites packages.conf and reloads. An auto abort timer starts, 120 minutes by default. Every member of a stack upgrades at the same time.
The switch runs 17.18.4 in state U: activated, not yet committed. While the timer runs, I check the version, the stack, routing neighbours, PoE devices and a sample of user traffic against my before readings.
install commit makes 17.18.4 the committed release, state C, and stops the timer. Later, install remove inactive frees the space the old release used.
If tests fail, install abort reloads into 17.15.6. If I lose access, the timer does the same on its own. After a commit, show install rollback lists rollback points and install rollback to id returns to one.

IOS XE releases and how long they last

Release	Type	Status in October 2026
17.9	Extended support	Security fixes ended in September 2026. Upgrade.
17.12	Extended support	Maintenance releases ended in March 2026. Security fixes continue to September 2027.
17.15	Extended support	Recommended by Cisco for Catalyst 9000. Maintenance to March 2027.
17.18	Extended support	Recommended by Cisco for Catalyst 9000. Maintenance to February 2028.
17.10, 17.11, 17.13, 17.14, 17.16, 17.17	Standard support	12 months of fixes from first shipment, then move on.
26.1 onwards	Extended support, every release	Numbered by year. Two releases a year, each with 48 months of support.

Lifetimes from Cisco's IOS XE lifecycle statement and its recommended releases page for Catalyst 9000, checked in October 2026. Per release dates as tracked by endoflife.date from Cisco's notices. I confirm them against Cisco's own notice before I plan an upgrade.

Image and version decoder

Type an image file name or a release number, or pick an example. It tells me the platform, the release, its support type and what to check before I use it.

Image file or release

Platform: Catalyst 9300, 9400, 9500 and 9600
Software: IOS XE 17.15.6: release 17.15, rebuild 6
Support: Extended support. Cisco recommended it for Catalyst 9000 in September 2026, with 17.18. Maintenance to March 2027, security fixes to September 2028.
File: A bundle signed by Cisco (SPA). On Catalyst 9000 I install it with install add file, then activate and commit.
Before I use it: Read the release notes and open caveats, run the release through the Cisco Software Checker for security advisories, check ROMMON and firmware notes, and check flash space on every member.

17.15.6: an extended support release that Cisco recommends for Catalyst 9000. A sound standard for an estate.

Upgrade options by platform

Platform	How it upgrades	What users notice
Catalyst 9200	Install mode only, with a full reload	An outage while the switch or stack reloads, so I upgrade in a change window.
Catalyst 9300 stack	Install mode, with xFSU (extended fast software upgrade) where the model and release support it	A shorter outage than a full reload. There is no ISSU on a 9300.
Catalyst 9400 with two supervisors	ISSU	Supervisors upgrade one at a time. Line cards keep forwarding.
Catalyst 9500 or 9600 with StackWise Virtual	ISSU	One switch at a time. Devices dual homed to both keep forwarding.
Routers	Install mode, reload in a change window	Sites with two routers fail over to the second, so I upgrade one router per window.
A single fix	SMU (software maintenance upgrade): `install add file ... activate commit`	A hot SMU applies without a reload. A cold SMU needs one.

How I run an upgrade

Choose the release. Extended support, recommended for the platform, and clean in the Cisco Software Checker for the features I run.
Read the release notes. Open caveats, ROMMON or firmware updates that add reboots, and features that were removed or changed defaults.
Take before readings. Routing neighbours, route and address table counts, PoE devices, stack state, licences and a saved config, so I can compare after.
Lab, pilot, then waves. One lab device, one quiet site, then waves through change control, each with a tested rollback.
Upgrade with a safety net. Install mode with the auto abort timer, so a device I cannot reach rolls itself back.
Compare, then commit. Only when the after readings match the before readings do I commit and clean up flash.

Command line

The command line, and changes I can take back

IOS applies each command the moment I press Enter, whether I type it in the EXEC (the IOS command interpreter) or in a configuration mode. That makes it quick to work with, and easy to break things with. I protect myself three ways: TACACS+ (Terminal Access Controller Access Control System Plus) so every command is authorised and logged, an archive of every saved config, and changes that roll back on a timer unless I confirm them.

Swipe sideways to see the whole diagram.

Logging in through the console or SSH (Secure Shell) lands me in user EXEC, shown by the > prompt. I can run basic show commands and ping, but I cannot change anything or read the config.

enable moves me to privileged EXEC, the # prompt, at privilege level 15. Everything that changes state starts here: show running-config, copy, reload, debug and clear.

configure terminal enters global configuration. Each change takes effect when I press Enter, in the running config only. Nothing survives a reload until I save it with copy running-config startup-config.

Interfaces, routing protocols and lines each have their own mode. exit goes up one level, and end or Ctrl+Z goes straight back to privileged EXEC. do runs an EXEC command, such as do show ip interface brief, from any config mode.

ROMMON sits below IOS. I reach it by sending a break on the console while the device boots, or it appears when no image will load. Its short command set covers password recovery and booting a chosen image.

Animation

Swipe sideways to see the whole diagram.

Step 1

An archive block with a path and write-memory keeps a numbered copy each time I save. log config also records who typed which command, which helps after an incident.
configure terminal revert timer 10 starts a change I might regret. If I lock myself out, the device rolls back to the archived config after 10 minutes on its own.
configure confirm stops the timer and keeps the change. Then copy running-config startup-config saves it, which also adds a version to the archive.
configure replace flash:new.cfg list compares a complete config file with the running config and applies only the differences, adding and removing lines. list shows each line it applied. This is how I apply a config built from a template.
configure replace with an archived file returns to a known version. show archive config differences shows what changed between any two versions before I commit to one.

Access control and an audit trail

Device access through TACACS+ on Cisco ISE (Identity Services Engine), with a local account for when the servers cannot be reached. Names and addresses are examples.

aaa new-model
tacacs server ISE-1
 address ipv4 10.0.0.15
 key <shared secret>
aaa group server tacacs+ ISE
 server name ISE-1
 ip vrf forwarding Mgmt-vrf
 ip tacacs source-interface GigabitEthernet0/0
aaa authentication login default group ISE local
aaa authorization exec default group ISE local if-authenticated
aaa authorization commands 15 default group ISE local if-authenticated
aaa accounting commands 15 default start-stop group ISE
username breakglass privilege 15 algorithm-type scrypt secret <long random secret>
archive
 path flash:archive/cfg
 write-memory
 log config
  logging enable
  notify syslog

Privilege level	What it allows	How I use it
0	A handful of commands, such as `enable` and `exit`	Rarely used.
1	User EXEC: basic show commands and ping	The default after login, before `enable`.
2 to 14	Custom sets of commands	I avoid these. Per group command authorisation in TACACS+ is easier to audit.
15	Everything	Engineers, through TACACS+, with every command logged.

Filters I use every day

I want to see	Command
One part of the config	`show running-config \| section router ospf`
One interface	`show running-config interface GigabitEthernet1/0/1`
Lines that match	`show running-config \| include ^hostname\|^ip route`
Defaults as well as my settings	`show running-config all \| section line vty`
What changed since the last save	`show archive config differences`
Who changed what	`show archive log config all`

How sessions in Arista EOS (Extensible Operating System) and the Junos commit model compare with line by line changes in IOS is on the Arista and Juniper page.

Switching

Catalyst access switches that ride through failures

On Catalyst switches the work that matters most happens at the access layer: stacks that survive failures, power for phones and access points, and ports that know who is plugged in. Spanning tree, VLAN (virtual local area network) and EtherChannel basics are on the switching page, and network access control is on the security page.

Swipe sideways to see the whole diagram.

Stack cables join up to eight Catalyst 9300 switches into one logical switch, with one config and one management address. Traffic runs both ways round the ring, at up to 480 Gb/s with StackWise-480. The active switch runs the control plane, the standby keeps a synchronised copy, and every member forwards its own ports in hardware.

If one stack cable fails or is pulled, the ring becomes a line and traffic wraps back the other way. Stack bandwidth halves until it is fixed, but nothing drops. show switch stack-ports summary shows which port is down, and I treat it as urgent because a second failure splits the stack.

If a member fails, its own ports go down and nothing else does. The uplinks are one EtherChannel spread across switches 1 and 4, so the stack keeps its way out. Devices that matter, such as access points and door controllers, can be split across members.

If the active switch fails, the standby takes over the control plane with SSO (stateful switchover) and the stack elects a new standby. The other members keep forwarding through the change. Because the uplink bundle also lands on switch 4, the stack stays connected to distribution.

Animation

The active switch is chosen in this order: the current active, then the highest priority, then the lowest MAC (media access control) address. I set priority 15 and 14 on the two switches I want as active and standby, so a reload never hands control to a random member. The comparison with StackWise Virtual, VSS (Virtual Switching System) and vPC (virtual port channel) is on the switching page.

Power over Ethernet

Standard	Name	At the switch port	At the device	Typical devices
IEEE 802.3af	PoE (Power over Ethernet)	15.4 W	12.95 W	Phones, simple cameras
IEEE 802.3at	PoE+	30 W	25.5 W	Older access points, video phones
Cisco UPOE	Universal PoE, Cisco's own	60 W	About 51 W	Thin clients, early four pair devices
IEEE 802.3bt Type 3	Four pair PoE	60 W	51 W	Current Wi-Fi access points
IEEE 802.3bt Type 4	UPOE+ on Cisco	90 W	71.3 W	Large access points, displays, building systems

Power at the device is lower than at the port because of losses in up to 100 m of cable. The switch budget, set by its power supplies, caps the total across all ports.

I size the budget for every port in use at once, plus headroom, and set port priority so access points and phones keep power before a desk fan or a display does. Perpetual PoE on Catalyst 9000 keeps devices powered while the switch reloads, which turns a software upgrade into a short network outage rather than a reboot of every phone and access point.

Access security on Catalyst

Port authentication

Ports authenticate users with 802.1X, and printers or cameras with MAB (MAC authentication bypass), against ISE. Current IOS XE uses policy maps for this, often called IBNS (Identity Based Networking Services) 2.0; authentication display new-style converts older configs. show access-session interface Gi1/0/5 details shows the result.

MACsec

IEEE 802.1AE encrypts each frame on the wire. I use it on uplinks that leave a secure room, keyed with MKA (MACsec Key Agreement) and a pre shared key between switches, or through 802.1X to hosts. show macsec summary and show mka sessions confirm it.

Device tracking

IOS XE learns which IP (Internet Protocol) address sits behind which port with its device tracking feature. 802.1X, IP source guard and SD-Access all rely on it, so I check show device-tracking database when a client looks invisible.

Commands I use on access switches

Check	Command
Stack members, roles and versions	`show switch`
Stack cables and ring state	`show switch stack-ports summary`
Control plane redundancy	`show redundancy`
Power budget and each port	`show power inline`
One powered device in detail	`show power inline GigabitEthernet1/0/12 detail`
Who is on a port, and how they authenticated	`show access-session interface GigabitEthernet1/0/5 details`

Routing

How IOS forwards, and how it fails over

A router's first job is to move packets quickly. Cisco has done that three ways over the years, and the third, CEF (Cisco Express Forwarding), is the one every current device uses. Around it sit the features I rely on at branches and edges: routes tracked by IP SLA (IP service level agreement) probes, NAT (network address translation), VRF (virtual routing and forwarding) tables, flow records and event scripts. ARP (Address Resolution Protocol) supplies the next hop's hardware address in each case.

Swipe sideways to see the whole diagram.

Process switching sends every packet to the processor, which looks up the routing table and ARP for each one. It is the slowest path. Today only packets the forwarding path cannot handle take it, but the name survives in the counters of show interfaces switching.

Fast switching sent the first packet to a destination through the processor, then cached the result so later packets skipped the lookup. Caches went stale when routes changed, and a flood of new destinations, such as a worm scanning addresses, filled the cache and the processor at once.

CEF builds two tables in advance from the routing table and ARP: the FIB (forwarding information base) and the adjacency table, which holds the rewritten Layer 2 header for each next hop. Every packet is looked up in them, the first one included. Only exceptions reach the processor: packets for the router itself, packets whose TTL (time to live) has run out, and next hops with no ARP entry yet.

Animation

Swipe sideways to see the whole diagram.

Step 1

IP SLA sends an ICMP (Internet Control Message Protocol) echo to a target beyond the carrier every 10 seconds. A track object follows the result, and the default route through the primary line is tied to that track. A second default route with administrative distance 250 waits in reserve.
The carrier fails beyond my router, but the link to it stays up. A plain static route would stay in the table and send traffic into the failure. The probes start to fail instead.
Once the probes fail, the track goes down after its delay down time and the tracked route leaves the routing table. The floating route through the backup line is installed and traffic moves. An EEM (Embedded Event Manager) applet logs the change.
When probes succeed again, delay up 60 makes the track wait a minute before it comes up, so a line that flaps does not drag traffic back and forth. Then the primary route returns.

Tracked default route with a floating backup, and an EEM applet that logs the change. Addresses are documentation examples.

ip sla 10
 icmp-echo 203.0.113.10 source-interface GigabitEthernet0/0/0
 frequency 10
ip sla schedule 10 life forever start-time now
track 1 ip sla 10 reachability
 delay down 10 up 60
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1
ip route 0.0.0.0 0.0.0.0 198.51.100.1 250
event manager applet WAN-FAILOVER
 event track 1 state down
 action 1.0 syslog msg "Primary internet path down, using backup"

Router features I use most

Feature	What I use it for	How I check it
NAT	Internet edges, and partner networks that overlap with mine	`show ip nat translations`, `show ip nat statistics`
VRF lite	Separate routing tables for guest, building systems or management, without MPLS (Multiprotocol Label Switching)	`show vrf`, `show ip route vrf GUEST`
Zone based firewall	Stateful inspection between zones on a branch router	`show policy-map type inspect zone-pair sessions`
Flexible NetFlow	Who and what is using a link, exported to a collector	`show flow monitor WAN-MON cache`
IP SLA and tracking	Path checks, failover and delay and jitter measurement	`show ip sla statistics`, `show track`
EEM	Running commands or raising alerts when a log message or event appears	`show event manager policy registered`

Routing protocols themselves, from OSPF (Open Shortest Path First) to BGP (Border Gateway Protocol), are on the routing page. Tunnels, VPN (virtual private network) designs and DMVPN (Dynamic Multipoint VPN) are on the WAN and VPN page.

Control plane

Keep transit traffic in hardware, and the processor safe

A Catalyst switch forwards almost everything in its ASIC. The processor runs the control plane: routing protocols, spanning tree, logins and monitoring. Trouble starts when traffic that should stay in hardware is punted to the processor, or when a hardware table fills. CoPP (control plane policing) and TCAM (ternary content addressable memory) are where I look.

Swipe sideways to see the whole diagram.

Animation

Live view: user traffic crosses the ASIC, while routing and management traffic is punted to the processor within its rate and the flood is dropped at CoPP.

What reaches the processor

Packets addressed to the device, and routing and spanning tree control traffic. Also ARP for addresses it does not know yet, packets whose TTL has run out, IP options, and anything matched by an ACL (access control list) line with log. A loop or a scan can send a lot of these very quickly.

Catalyst 9000 defaults

Catalyst 9000 switches apply a built in policy, system-cpp-policy, with a class and rate for each kind of punted traffic. I tune its rates from a baseline rather than replace it. show policy-map control-plane shows each class and its drops.

Routers

On ISR, Catalyst 8000 and ASR routers I build my own policy on the control-plane: routing, management from known addresses, monitoring, then everything else at a low rate. I start in a report only mode with generous rates and tighten them once I know the normal levels.

Tables in the ASIC

Resource	What it holds	How I check it
FIB in TCAM	IPv4 and IPv6 routes the ASIC can forward	`show platform hardware fed switch active fwd-asic resource tcam utilization`
MAC table	Learned MAC addresses	`show mac address-table count`
ACL and QoS (quality of service) TCAM	Security ACLs, QoS classifiers and policy based routing	The same TCAM command, by region
Adjacencies	Next hops with their rewrite information	`show adjacency summary`, `show ip arp summary`
SDM (Switch Database Management) template	How the ASIC splits its tables between those uses	`show sdm prefer`, then `sdm prefer` and a reload to change it
Punt causes	Why traffic was sent to the processor	`show platform software fed switch active punt cause summary`

When a table fills, new entries either fail or fall back to the processor. Both look like random faults, so I watch utilisation as I would disk space.

A router CoPP policy in outline. Rates are indicative starting points, set from a baseline in practice.

ip access-list extended COPP-ROUTING
 permit ospf any any
 permit tcp any any eq bgp
ip access-list extended COPP-MGMT
 permit tcp 10.0.0.0 0.0.0.255 any eq 22
 permit udp 10.0.0.0 0.0.0.255 any eq snmp
class-map match-all COPP-ROUTING
 match access-group name COPP-ROUTING
class-map match-all COPP-MGMT
 match access-group name COPP-MGMT
policy-map COPP
 class COPP-ROUTING
  police 512000 conform-action transmit exceed-action transmit
 class COPP-MGMT
  police 256000 conform-action transmit exceed-action drop
 class class-default
  police 128000 conform-action transmit exceed-action drop
control-plane
 service-policy input COPP

Programmability

The same switch, driven by code

IOS XE describes its config and state in YANG (Yet Another Next Generation) data models. It exposes them through NETCONF (Network Configuration Protocol), RESTCONF (its web based equivalent) and gNMI (gRPC Network Management Interface), which runs over gRPC, Google's remote procedure call framework. That lets tools change devices without screen scraping, and lets devices stream their state instead of waiting for SNMP (Simple Network Management Protocol) polls. Catalyst Center builds on the same interfaces.

Swipe sideways to see the whole diagram.

Animation

Live view: NETCONF requests and replies, telemetry streaming on a schedule, Catalyst Center managing the switch, and logs going to a syslog server.

Interface	Carried over	What I use it for
NETCONF	SSH, port 830	Reading and changing config through YANG models from Ansible or Python, with a candidate datastore if I enable it.
RESTCONF	HTTPS (Hypertext Transfer Protocol Secure)	The same models over a REST (representational state transfer) API (application programming interface), for tools built around web requests.
gNMI	gRPC	Config and telemetry for collectors built around the OpenConfig tools.
Model driven telemetry	gRPC dial out, or dial in through NETCONF or gNMI	Interface counters, CPU (processor) use and routing state every few seconds, instead of SNMP polls every five minutes.
Guest shell	A Linux container on the device	Python on the box, ZTP scripts and small tools, kept apart from IOSd.
Catalyst Center	NETCONF, SSH and SNMP	Plug and Play, software image management, templates and Assurance across the estate.
SNMP and syslog	UDP (User Datagram Protocol)	Still the common ground for monitoring tools. I run SNMPv3 with authentication and encryption only.

Turning on the programmable interfaces, and one telemetry subscription that streams five second CPU use every 60 seconds. Addresses are examples.

netconf-yang
restconf
ip http secure-server
ip http authentication local
iox
telemetry ietf subscription 101
 encoding encode-kvgpb
 filter xpath /process-cpu-ios-xe-oper:cpu-usage/cpu-utilization/five-seconds
 source-address 10.0.0.21
 stream yang-push
 update-policy periodic 6000
 receiver ip address 10.0.0.50 57500 protocol grpc-tcp

update-policy periodic counts in hundredths of a second, so 6000 means every 60 seconds. Pipelines, testing and source of truth design are on the network automation page.

Hardening

A management plane that is hard to abuse

Most attacks on network devices go after the management plane: weak passwords, Telnet on the VTY (virtual terminal) lines, open web servers and default SNMP communities. None of these are hard to fix, but they reappear whenever a device is built by hand. I keep a baseline, apply it from a template and audit against it.

Config hardening check

Paste part of an IOS or IOS XE config, or edit the sample, then run the check. It looks for the weaknesses that turn up most often in audits. It runs entirely in this page, and nothing is sent anywhere.

Config to check

6 problems and 8 warnings, with 0 good controls found.

Enable password enable password stores the password weakly. Use enable algorithm-type scrypt secret.
Local account A username ... password line can be reversed. Use algorithm-type scrypt secret.
Telnet VTY lines accept Telnet, which sends passwords in clear text. Use transport input ssh.
No idle timeout exec-timeout 0 0 leaves sessions open for ever. Use exec-timeout 10 0.
Web server ip http server is on. Turn it off, and keep ip http secure-server only if RESTCONF or the web interface needs it.
SNMP community A well known community string such as public. Replace it with SNMPv3 using auth and priv.
Type 7 service password-encryption only hides passwords from a glance. Type 7 is reversible.
VTY access list No access-class on the VTY lines, so anyone who can reach the device can try to log in.
Other VTY lines Only line vty 0 4 is set. Check lines 5 to 15 have the same controls.
SSH version No ip ssh version 2. Set it, so no older release falls back to version 1.
Login protection No login block-for, so password guessing is not slowed down.
Central logins No aaa new-model: logins use local accounts only, with no central audit trail.
Remote logging No logging host, so logs are lost when the device reloads.
Time No ntp server, so log times cannot be trusted or lined up across devices.

The controls I apply to every device

Control	Commands	Why
Strong secrets	`enable algorithm-type scrypt secret`, `username ... algorithm-type scrypt secret`	Type 9 (scrypt) and type 8 (PBKDF2, Password Based Key Derivation Function 2) hashes resist cracking. Type 5 is an older MD5 (Message Digest 5) hash, and type 7 can be reversed.
SSH only	`ip ssh version 2`, `transport input ssh`, `crypto key generate rsa modulus 3072`	Telnet sends every keystroke in clear text.
Management from known places	`access-class MGMT-ONLY in vrf-also`, a management VRF	Only management networks can reach the login prompt at all.
Slow down guessing	`login block-for 120 attempts 5 within 60`, `exec-timeout 10 0`	Blocks logins for two minutes after five failures, and closes idle sessions.
Turn off what I do not use	`no ip http server`, `no cdp enable` on internet facing ports	Every service left on is something to attack, or something that leaks information.
Monitoring that cannot be read	`snmp-server group NMS v3 priv`	SNMPv3 with authentication and encryption, instead of community strings.
Logs and time	`logging host`, `service timestamps log datetime msec`, `ntp server`	Logs leave the device with timestamps I can line up across the estate.
Trusted software	`verify /md5`, `show software authenticity running`	The image is the one Cisco signed, and the hash matches the published value.

The core of my management plane baseline. Secrets and addresses are placeholders.

service timestamps log datetime msec localtime show-timezone
no ip http server
no ip http secure-server
ip ssh version 2
ip ssh time-out 60
ip ssh authentication-retries 3
login block-for 120 attempts 5 within 60
login on-failure log
enable algorithm-type scrypt secret <long random secret>
ip access-list standard MGMT-ONLY
 permit 10.0.0.0 0.0.0.255
line vty 0 15
 access-class MGMT-ONLY in vrf-also
 transport input ssh
 exec-timeout 10 0
snmp-server group NMS v3 priv
snmp-server user nms1 NMS v3 auth sha <auth key> priv aes 256 <privacy key>
logging host 10.0.0.60
ntp server 10.0.0.1

I build the baseline from Cisco's IOS XE hardening guide and the CIS (Center for Internet Security) benchmarks, then check every device against it on a schedule, so drift shows up as a report rather than an incident. Wider security design is on the security page.

Troubleshooting

Finding faults on Catalyst and IOS, layer by layer

Most faults on these platforms fall into a small set: a damaged link, a protection feature doing its job, a busy processor, a full hardware table, a half finished upgrade or an access problem. I work from the bottom up and read the counters before I touch the config. Choose a symptom to see where it sits and what I run first. My general method is on the troubleshooting page.

Swipe sideways to see the whole diagram.

Errors and slow transfers on one port. CRC (cyclic redundancy check) errors mean frames arrive damaged: a bad cable, a dirty or failing optic, or a duplex mismatch with an old device. I check counters on both ends, run test cable-diagnostics tdr interface Gi1/0/12 on copper, read optic levels with show interfaces transceiver detail, then replace the cheapest part first.

Phones or access points keep rebooting. The switch has run out of power budget, a device needs more than its port can give, or a power supply has failed. show power inline shows used and remaining power per switch and per port. I add power, move the device to a UPOE+ port, or set port priority so critical devices keep power first.

A port went dead after a device was plugged in. A protection feature shut it: BPDU (bridge protocol data unit) guard saw a switch on an access port, or port security saw too many MAC addresses. I find what was connected and remove the cause, then shutdown and no shutdown the port. Automatic recovery on a timer is possible, but I use it with care.

Slow logins and routing flaps. Something is being punted to the processor: an ARP storm, a loop, a routing loop expiring TTLs, or ACLs with log. show processes cpu platform sorted shows whether IOSd or FED is busy, show platform software fed switch active punt cause summary shows why, and an EPC (Embedded Packet Capture) on the control plane shows who. I fix the source, then check CoPP.

New routes or ACL lines do nothing. The hardware table for that feature is full, so new entries fail or fall back to the processor, and the log usually says so. I summarise routes, shorten ACLs with object groups, or move to an SDM template that gives that table more room, which needs a reload.

Switches missing from a stack after a reload. A member came back on different software and sits in version mismatch, or a stack cable is loose. show switch shows each member's state, and show switch stack-ports summary the cables. The software auto upgrade feature can copy the stack's release to the member; otherwise I upgrade it alone and rejoin it.

The switch went back to the old release. The upgrade was activated but never committed, so the auto abort timer rolled it back, as designed. show install summary shows the packages and their states. I upgrade again and run install commit once my checks pass, or stop the timer with install auto-abort-timer stop if testing needs longer.

One subnet cannot reach the internet. Its traffic is not being translated: the subnet is missing from the NAT ACL, or a new interface lacks ip nat inside. show ip nat translations and show ip nat statistics show what is translated and how often the ACL matched. show ip route on the way back checks the return path.

My directory login fails on one device. The device cannot reach TACACS+, or ISE does not know it. Usual causes are a wrong source interface, a VRF the server group does not use, or a shared key mismatch. show tacacs shows failed connections, test aaa group ISE tests one login, and the ISE live log shows whether the request arrived. The breakglass account gets me in meanwhile.

Animation

The commands I reach for

Question	Command	What it tells me
What is this device, and why did it reload?	`show version`	The release, the uptime and the last reload reason.
What happened recently?	`show logging`	Events with timestamps, if NTP (Network Time Protocol) and timestamps are set.
Which ports have errors?	`show interfaces counters errors`	CRC, runts, giants and discards on every port.
Where is the processor going?	`show processes cpu sorted`, `show processes cpu platform sorted`	Busy IOSd tasks, and busy Linux processes such as FED.
Is memory running out?	`show platform software status control-processor brief`	Processor load and memory per control processor, read over time.
What does the hardware hold?	`show platform hardware fed switch active ...`	ASIC tables, queues and drop counters.
What is the packet doing?	EPC with `monitor capture`, or packet trace on routers	Real packets captured on the box, or each feature a packet passed and why it dropped.
What does Cisco need?	`show tech-support`	The bundle Cisco TAC (Technical Assistance Center) asks for first.

Embedded Packet Capture on a Catalyst switch. The file opens in Wireshark.

monitor capture CAP interface GigabitEthernet1/0/12 both
monitor capture CAP match ipv4 host 10.20.0.15 any
monitor capture CAP buffer size 10
monitor capture CAP start
! reproduce the fault, then
monitor capture CAP stop
show monitor capture CAP buffer brief
monitor capture CAP export flash:cap.pcap

Packet trace on an ISR, Catalyst 8000 or ASR 1000 router: each feature a packet hit, and the drop reason if it died.

debug platform condition ipv4 10.20.0.15/32 both
debug platform packet-trace packet 64
debug platform condition start
! reproduce the fault, then
debug platform condition stop
show platform packet-trace summary
show platform packet-trace packet 0

I change one thing at a time, and repeat the same test after each change. Before I open a case with Cisco I collect show tech-support and the exact time of the fault, because the first question is always which release and what changed.

Engagement

How I run a Catalyst and IOS XE refresh, from audit to steady state

A campus or branch refresh usually starts with end of life dates, a security finding or a new need such as more power for access points. I follow the same path each time: agree the drivers, audit what is really installed, design standards and templates, prove them in a lab and a pilot, then hand over an estate that someone can keep current.

Swipe sideways to see the whole diagram.

Step 1

I agree the drivers first: end of life dates, audit findings, or new devices that need more power or speed. Then the sites, closets and routers in scope, and who decides, recorded in a RACI (responsible, accountable, consulted, informed) chart. Each site tells me which change windows it can live with.
I turn the drivers into requirements with numbers: ports and PoE per closet, uplink speeds, availability, and how long an upgrade may interrupt users. Security sets the hardening baseline, NAC (network access control) and logging needs. Each requirement has an owner who signs it off.
I audit the estate before designing anything: every device with its serial, release and end of life dates, how full each closet is, how much power is drawn, and how far configs have drifted from the standard. The account team confirms contract and end of life data by serial number.
The high level design picks platforms, stacking or pairing, licences and the software release standard. The low level design turns that into golden configs and templates, port maps, power budgets and a cutover plan for each closet, each with a tested way back.
I prove the templates and the upgrade method in the lab, then cut over one pilot closet out of hours. Every wave follows the same runbook: stage the new kit, load its config through Plug and Play, move the cables, then compare before and after checks.
Before handover every device is in monitoring, config backup and the CMDB (configuration management database). The NOC (network operations centre) gets runbooks for common faults, spares sit at the right sites and the RMA (return material authorisation) process is tested. Hypercare ends when agreed criteria are met.
I review the release standard twice a year against Cisco's recommended releases, assess each security advisory against the features in use, and track field notices. End of life notices feed a rolling refresh plan, so no platform reaches its last date of support by surprise.

Functional requirements

What it must do.

Give every user, phone, access point and building device a port with the right speed, VLAN and power.
Authenticate users and devices on every access port, with an ISE policy for each type.
Route between VLANs at distribution or access, and connect each site to the WAN and the data centres.
Keep each site working through the loss of one stack member, uplink, power supply or router.
Upgrade software with a way back, in agreed windows, without visiting the site.
Send logs, flow records and telemetry to central tools, and back up every config on change.

Non functional requirements

How well it must do it, with a number wherever one can be agreed.

Availability: 99.99% for core and distribution, 99.9% for a single access closet (typical).
Upgrade impact: No outage at core and distribution with ISSU, and one reload per access closet inside an agreed window (typical).
PoE headroom: At least 20% of each switch's power budget spare with every port in use (typical).
Uplinks: Access to distribution oversubscription of 20 to 1 or better (typical).
Security: Every device on the hardening baseline: SSH only, TACACS+ logins, SNMPv3 and logs sent off the device.
Software: One extended support release per platform family, and never more than two in the estate.
Recovery: Config backed up on every change, and a failed switch replaced and restored within one business day (typical).
Operability: Every device in monitoring, inventory and the CMDB before handover.

Numbers are typical starting points. Each organisation agrees its own.

Who I talk to

Network operations, who know the faults and the workarounds
Site and facilities teams, who know the closets, power and cabling
Security, for access control, logging and audit findings
Owners of phones, access points, cameras and building systems
The Cisco account team and reseller, for contracts and end of life data

How I find out

Inventory collected from the devices themselves, with serials, releases and modules
Port, PoE and uplink use over a busy week
Config audit against the standard, with drift listed per device
Closet visits with photos of racks, power, cooling and cabling
Contract and end of life review for every serial number

Discovery tools

Tool or source	What it tells me
`show inventory`, `show version` and `show license summary`	Hardware, serials, releases and licences
Catalyst Center inventory and Assurance	Estate wide inventory, health and software compliance
Cisco Software Checker and security advisories	Which advisories apply to each release in use
Ansible or Python with Genie parsers	Config and state collected from hundreds of devices at once
NetBox or another source of truth	The intended state to compare against
SNMP or telemetry history	Port, PoE and uplink use over time

After a refresh, most faults are configuration or cabling rather than hardware, so I check those first, once I know exactly what changed.

My approach

Scope. Which users, which closet or site, since when, and what changed: a cutover, an upgrade or a new type of device?
Physical. Link lights, errors, optics and PoE on the ports involved, and the stack and uplink state.
Layer 2. VLAN, trunk and spanning tree state, err-disabled ports and MAC learning along the path.
Layer 3. Gateways, routes, NAT and ACLs along the path, checked hop by hop.
Platform. Processor, memory, punt causes and hardware tables on each device the traffic crosses.
Fix and prove. One change through change control, the same test repeated, and the cause recorded for problem management.

Faults I see most

A port in the wrong VLAN after a cutover
Access points without enough PoE
A stack member on the wrong release
An upgrade rolled back by its own timer
TACACS+ unreachable from a new VRF
A full TCAM region after new ACLs

Layer by layer

Layer	What I check
1 · Physical	CRC errors, optic levels, cable tests with TDR (time domain reflectometry), and PoE budgets.
2 · Data link	VLANs and trunks, spanning tree, err-disabled ports and 802.1X sessions.
3 · Network	Routes, CEF entries, NAT translations, ACL hits and tracked routes.
4 · Transport	Session timeouts through the zone based firewall, and TCP (Transmission Control Protocol) resets in captures.
5 to 7 · Application	Logins through TACACS+, NETCONF sessions, and the services users report.

My general method, with worked faults, is on the troubleshooting page.

Projects

Where I have used it

Cisco IOS runs through my whole career, from telecoms networks in East Africa and South America to Catalyst campuses and SD-WAN across a global software company. My lab still runs classic IOS on 2811 routers and 3750 switches, which keeps the fundamentals close at hand.

Home lab · ongoing

Classic IOS in a 42U rack

Cisco 2811 routers form an MPLS core with two P (provider) and five PE (provider edge) routers, with Catalyst 3750 switches at each site and a 2811 terminal server giving console access to every device. I use it for proof of concept work, troubleshooting and certification study. See the topology.

BNY Mellon · 2018 to 2019

Refreshing a regional network

The EMEA (Europe, the Middle East and Africa) network refresh: audit, high and low level design and implementation of LAN (local area network), WAN, SD-WAN and data centre infrastructure. It replaced end of life equipment with minimal impact on global operations and no service outages.

AVEVA · 2020 to 2021

Catalyst at every site worldwide

Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center, and upgraded every global site remotely during the lockdowns through local enablement, with no delay.

XIDA · 2025 to now

From discovery to provisioning

Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access and the Amazon Web Services, Azure and Google Cloud environments.

JJ Food Wholesale · 2007 to 2011

A group wide network

LAN, WAN, VoIP (voice over IP), virtualisation and security across the group, including warehouse, cold chain and temperature monitoring systems at the food depots, linked to the group ERP (enterprise resource planning) system.

Passport Telecom · 2004 to 2007

Cisco networks on two continents

Cisco based telecoms infrastructure for over 4,000 clients across East Africa and South America.

Backed by CCNP (Cisco Certified Network Professional) Enterprise and Routing and Switching, CCNA (Cisco Certified Network Associate) Routing and Switching and CCDP (Cisco Certified Design Professional). I have also passed the CCIE (Cisco Certified Internetwork Expert) Routing and Switching written exam and the Cisco Certified Specialist exams in Enterprise Core and Enterprise Advanced Infrastructure Implementation. See all credentials

Explore

More areas I design and deliver

Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.

Cisco IOS and Catalyst, from first boot to rollback

One command line, several operating systems underneath

Cisco operating systems today

Read the version first

Same commands, different behaviour

Process faults show up differently

Catalyst switches and routers, matched to the job

Platforms and what forwards the packets

End of life milestones I plan around

Licensing

Base licence

Subscription

Smart Licensing Using Policy

Choices I make

Stack, chassis or pair

Install mode, extended releases

Autonomous or SD-WAN

Buy for the life of the box

From power on to a working config

Boot and recovery settings I check

Upgrades that can undo themselves

IOS XE releases and how long they last

Image and version decoder

Upgrade options by platform

How I run an upgrade

The command line, and changes I can take back

Access control and an audit trail

Filters I use every day

Catalyst access switches that ride through failures

Power over Ethernet

Access security on Catalyst

Port authentication

MACsec

Device tracking

Commands I use on access switches

How IOS forwards, and how it fails over

Router features I use most

Keep transit traffic in hardware, and the processor safe

What reaches the processor

Catalyst 9000 defaults

Routers

Tables in the ASIC

The same switch, driven by code

A management plane that is hard to abuse

Config hardening check

The controls I apply to every device

Finding faults on Catalyst and IOS, layer by layer

The commands I reach for

How I run a Catalyst and IOS XE refresh, from audit to steady state

Functional requirements

Non functional requirements

Who I talk to

How I find out

Discovery tools

My approach

Faults I see most

Layer by layer

Service model

Monitoring

Config backup

Runbooks

Spares

Reviews

Where I have used it

Classic IOS in a 42U rack

Refreshing a regional network

Catalyst at every site worldwide

From discovery to provisioning

A group wide network

Cisco networks on two continents

More areas I design and deliver

Networks and security (this area)

Foundations

Data centre and cloud

Industries

Governance and service