Cisco Nexus and NX-OS

Cisco Nexus, from first boot to the next upgrade

Cisco Nexus switches sit at the core of many of the data centres I have worked in. NX-OS (Nexus Operating System) is built to keep forwarding: each feature runs as its own process, changes can be checked and rolled back, and software can be upgraded while traffic flows. This page shows how NX-OS works, how I build vPC (virtual port channel) pairs that fail safely, and how I upgrade, automate and troubleshoot Nexus.

NX-OS architecture
Nexus families
vPC in depth
Checkpoints and rollback
ISSU and GIR
POAP and NX-API
Ethanalyzer and SPAN
CoPP and AAA

Start with the architecture Where I have used it

Live view · two switches, one logical partner for the server

NX-OS architecture

A switch operating system built to keep forwarding

NX-OS (Nexus Operating System) grew out of Cisco SAN-OS, the software for its storage switches, and runs on a Linux kernel. Its design goal differs from classic IOS (Internetwork Operating System): a fault in one part should not stop the switch forwarding. Choose a state to see how it handles a crash, a patch, a disabled feature and a supervisor fault.

Swipe sideways to see the whole diagram.

Each feature runs as its own process with protected memory, and starts only when I enable it. Processes talk through MTS (message and transaction service). The system manager starts and watches them. The kernel watches the system manager, and hardware watches the kernel.

OSPF (Open Shortest Path First) crashes. The system manager restarts it, and it reloads its state from PSS (persistent storage service), so it resumes instead of starting cold. Other processes and the forwarding ASICs (application specific integrated circuits) carry on. show processes log and show cores show what happened.

A hot SMU (software maintenance update) fixes one process without a reload. The process saves its state, stops, is replaced and restarts. Cisco quotes about 10 ms of recovery for a hot SMU, with no effect on traffic.

With no feature hsrp the HSRP (Hot Standby Router Protocol) process is not running at all. Nothing listens, nothing uses memory, and there is less to attack. NX-OS takes a system checkpoint when a feature is disabled, so the change can be rolled back.

A fault in the kernel is beyond a process restart. On a chassis with two supervisors, the standby takes over in a stateful switchover, because its state is already in sync. A fixed switch has one supervisor, so it reloads.

Animation

Habits that change when I move from IOS

Features start switched off

Nothing runs until I enable it: feature ospf, feature lacp, feature vpc, feature interface-vlan. show feature lists what is on. Fewer processes means less to patch and less to attack.

Management has its own routing table

mgmt0 sits in its own VRF (virtual routing and forwarding instance), called management, so I write ping 10.0.0.1 vrf management and point AAA (authentication, authorisation and accounting), syslog and NTP (Network Time Protocol) at it. Forgetting it is the most common first day fault.

Show commands work anywhere

I can run show commands inside configuration mode without do. Every interface is called Ethernet, whatever its speed, such as Ethernet1/49.

Saving and comparing

show running-config diff shows changes not yet saved, and copy running-config startup-config saves them. Checkpoints and rollback are built in.

Linux underneath

With feature bash-shell and the right role I can open a Bash shell on the switch. Access is controlled by RBAC (role based access control). Python runs on the switch too.

One image across families

The same NX-OS image runs on Nexus 9000 and Nexus 3000 switches. ACI (Application Centric Infrastructure) mode is a different image on the same Nexus 9000 hardware.

Features I enable most

Command	What it turns on	Where I use it
`feature lacp`, `feature vpc`	LACP (Link Aggregation Control Protocol) and virtual port channels	Every leaf pair
`feature interface-vlan`, `feature hsrp`	SVIs (switched virtual interfaces), one per VLAN (virtual LAN), and the HSRP gateway	Classic Layer 3 at the aggregation layer
`feature ospf`, `feature bgp`, `feature pim`	Routing and multicast processes	Underlay and border routing
`feature nv overlay`, `nv overlay evpn`	VXLAN (Virtual Extensible LAN) with an EVPN (Ethernet VPN) control plane	Fabrics, covered on the VXLAN EVPN page
`feature tacacs+`	Device logins and command authorisation through TACACS+ (Terminal Access Controller Access Control System Plus)	Every switch
`feature nxapi`, `feature telemetry`	The NX-API and streaming telemetry	Automation and monitoring
`feature ptp`	PTP (Precision Time Protocol)	Trading and media networks

NX-OS 10 reports licence use through Smart Licensing Using Policy, with Essentials and Advantage tiers. Before a design is signed off, I check which tier each feature in it needs, so the order includes the right licences.

Nexus families

Which Nexus, for which job

Nexus has changed a lot since the first switches shipped in 2008. The Nexus 9000 family now carries almost every new design, while estates I inherit often still run 7000, 5000 and 2000 hardware. This table is how I place each family, then how the same hardware can run in three different ways.

Family	Role	What to know	Status in October 2026
Nexus 9300	Leaf, top of rack, small spine	Fixed switches from 1G to 400G and beyond, in generations named by suffix: EX, FX, FX2, FX3, GX and GX2. Run NX-OS or ACI. The newest models support enhanced ISSU (in service software upgrade) out of the box.	Current, with the EX models near their end of support
Nexus 9500	Spine, end of row	Modular chassis with 4, 8 or 16 slots and redundant supervisors. NX-OS or ACI. No non disruptive ISSU, so I drain it with maintenance mode before an upgrade.	Current
Nexus 9800	High density spine	Modular chassis built on Cisco Silicon One for dense 400G. Both supervisors must run the same BIOS (basic input output system) before an upgrade.	Current
Nexus 3000 and 3500	Top of rack, low latency	Run the same NX-OS image as the 9000 family. The 3500 models were built for low latency.	Older models end of sale
Nexus 3550	Ultra low latency	A separate line from Cisco's purchase of Exablaze, for trading networks. See the low latency trading page.	Specialist range
Nexus 7000 and 7700	Core and aggregation	Modular chassis with VDCs (virtual device contexts), OTV (Overlay Transport Virtualisation) and FabricPath. Covered on the data centre page.	End of sale: migrate
Nexus 5000, 5500, 5600 and 6000	Access and aggregation	Unified ports for Ethernet and Fibre Channel, FCoE (Fibre Channel over Ethernet), and parents for fabric extenders.	End of sale: migrate
Nexus 2000	Fabric extender	A FEX (fabric extender) is a remote line card with no local switching, managed from its parent. Port names such as `Ethernet101/1/1` give it away.	Legacy: replace with leaf switches

Model ranges and status are a summary. Before any design or refresh, I check the exact model against Cisco's end of life notices and the release notes for the train I plan to run.

Swipe sideways to see the whole diagram.

Standalone NX-OS. Each switch holds its own configuration, which I write from templates with Ansible or by hand through change control. Every protocol is mine to design: an OSPF or BGP (Border Gateway Protocol) underlay, VXLAN EVPN, vPC. It is the most flexible model, and drift is my problem to find.

Still standalone NX-OS, but Nexus Dashboard holds the intent and generates each switch's configuration. It compares that with what is running and flags any drift (the red packet). The controller side is on the VXLAN EVPN page.

ACI mode is a different image on the same Nexus 9000 hardware. The APIC (Application Policy Infrastructure Controller) cluster owns the policy, and the switches take no CLI (command line interface) configuration. Moving a switch between modes means a new image and a wipe.

Animation

How I choose

Leaf pairs

Nexus 9300 in vPC pairs, sized for server port speed and count, with uplinks at four times the server speed or more.

Spines

Nexus 9300 for small fabrics, and 9500 or 9800 where I need port density or chassis redundancy.

Old estates

An inventory of every 7000, 5000 and 2000 first, with the features each uses, so nothing is lost in the move.

vPC in depth

Building a vPC pair that fails safely

A vPC (virtual port channel) lets a server or switch bundle links to two Nexus switches as if they were one. The data centre page covers its parts, the loop prevention rule and what each failure does. Here I go a level deeper: the order I build a pair in, the settings I always add, and the timers that decide how it recovers.

Swipe sideways to see the whole diagram.

Step 1

On both peers I enable feature vpc and feature lacp, and create the same vpc domain 10. Domain IDs must differ between pairs that connect to each other, because the ID is part of the LACP system ID.
The peer keepalive comes up first, over mgmt0 in the management VRF. It sends a heartbeat on UDP (User Datagram Protocol) port 3200 every second, with a five second timeout. It never carries data. It only answers one question: is my peer alive?
The peer link comes up and the peers start syncing over CFS (Cisco Fabric Services). They elect roles: lower role priority wins, the default is 32667, then the lower system MAC (media access control) address. I set priority 100 on the switch I want as primary. Roles do not preempt.
The peers compare their settings. Type 1 settings, such as the spanning tree mode, must match or vPCs are suspended. Type 2 settings only raise a warning. The checker below shows which is which.
Port channel 20 gets vpc 20 on both peers. Both send LACP with the same vPC system MAC, so the server sees a single partner and bundles both links. It runs plain LACP and knows nothing about vPC.
Both peers forward for the server, and spanning tree blocks nothing. Each peer sends traffic upstream on its own uplinks. The peer link carries almost nothing in normal running.
When Nexus B reloads, its vPC ports wait 30 seconds (delay restore) and its SVIs 10 seconds (delay restore interface-vlan), the defaults. That gives routing and ARP (Address Resolution Protocol) time to fill before traffic arrives. Meanwhile Nexus A carries everything.

Nexus A. Nexus B is the same, with its own keepalive addresses and the default role priority. Addresses and numbers are examples.

feature lacp
feature vpc
feature interface-vlan

vpc domain 10
  role priority 100
  peer-keepalive destination 192.168.0.12 source 192.168.0.11 vrf management
  peer-switch
  peer-gateway
  layer3 peer-router
  auto-recovery
  delay restore 150
  ip arp synchronize
  ipv6 nd synchronize

interface port-channel1
  description vPC peer link
  switchport
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel20
  description server01
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 110,120
  vpc 20

interface Ethernet1/20
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 110,120
  channel-group 20 mode active

Timers and the values I choose

Setting	Default	What I do
Keepalive interval and timeout	1 second, 5 seconds	Leave them. I alert the moment the keepalive drops, because the pair can no longer tell a dead peer from a broken link.
Keepalive hold timeout	3 seconds	Leave it. After the peer link fails, keepalives are ignored for this long, so a peer that is rebooting is not mistaken for a live one.
Role priority	32667, lower wins	Set 100 on the switch I want as primary, so I know which side suspends on a peer link failure.
Delay restore	30 seconds for vPC ports	Raise it on switches with large routing tables, so routes are in place before the ports come up.
Delay restore for SVIs	10 seconds	Raise it with many SVIs, and bring them up in batches.
Auto recovery reload delay	240 seconds	Configure `auto-recovery` on every pair, so one switch can bring up its vPCs alone after both lost power and only one came back.
Role preemption	Off	Use `vpc role preempt` in a change window if I need the primary role back on the preferred switch.

Defaults are from Cisco NX-OS documentation for the Nexus 9000. I check them against the release notes for the train in use.

Arista MLAG (multi chassis link aggregation) works the same way, and its failure cases are on the Arista and Juniper page.

vPC options and checks

The settings that make a vPC pair behave like one switch

A vPC pair has two control planes that have to agree. Five options close the gaps that catch people out, and the consistency checks decide what happens when the two peers disagree. Choose a state to see each option at work, then try the checker.

Swipe sideways to see the whole diagram.

Some storage arrays and load balancers reply to the real MAC address of the router that sent them a packet, not to the HSRP virtual MAC. The port channel hashes a frame for Nexus B's MAC onto the link to Nexus A, which bridges it across the peer link. B routes it towards the server, but a frame that came over the peer link may not leave on a vPC port. It is dropped.

With peer-gateway, each peer routes frames sent to its partner's router MAC as if they were its own. Nexus A forwards straight to the server and nothing crosses the peer link. I configure it on every pair.

With peer-switch, both peers send BPDUs (bridge protocol data units) with the same bridge ID, so downstream switches see one root bridge. If the primary fails, spanning tree has nothing to reconverge. Both peers need the same spanning tree priority for the VLANs.

A router attached by vPC forms OSPF neighbours with both peers. Some of its hellos hash to the wrong peer and cross the peer link, where peer gateway would decrement their TTL (time to live) to zero. layer3 peer-router fixes that. It needs peer gateway, and I still prefer separate routed links where I can.

With ip arp synchronize and ipv6 nd synchronize, each peer copies its ARP and IPv6 neighbour entries to the other. After a reload or a peer link failure, the returning switch already knows the hosts, so it forwards straight away.

Animation

vPC consistency checker

Pick a setting that differs between the two peers. The checker shows how NX-OS treats it, what I see, and the fix. Without scripts, the table below lists every case.

What differs between the peers?

Check type: Type 1, global
What I see: show vpc brief reports the consistency status as failed. show vpc consistency-parameters global shows the mismatch.
Fix: Set the same mode on both peers in one change. I generate both configurations from one template so this cannot drift.

With graceful consistency check, which I always leave on, the secondary suspends every one of its vPC member ports. The primary keeps forwarding, so servers run on half their links.

Every case in one table

What differs	Check type	What happens
Spanning tree mode, such as Rapid PVST+ (per VLAN spanning tree) on one peer and MST (Multiple Spanning Tree) on the other	Type 1, global	With graceful consistency check, which I always leave on, the secondary suspends every one of its vPC member ports. The primary keeps forwarding, so servers run on half their links.
MST region name, revision or VLAN to instance mapping	Type 1, global	The same as a spanning tree mode mismatch: the secondary suspends all its vPC member ports.
Spanning tree disabled for one VLAN on one peer	Type 1, per VLAN	Only that VLAN is suspended on the vPCs. Every other VLAN carries on, so the fault looks like one broken subnet.
LACP mode on the port channel, such as active on one peer and on (static) on the other	Type 1, interface	The secondary suspends its member of that vPC. The primary keeps its member up, so the server runs on one link and has no resilience.
Port speed or duplex on the member links	Type 1, interface	The secondary suspends its member of that vPC. Often this follows a new optic or a server NIC (network interface card) that negotiated differently.
Switchport mode: trunk on one peer, access on the other	Type 1, interface	The secondary suspends its member of that vPC.
Native VLAN on the trunk	Type 1, interface	The secondary suspends its member of that vPC.
MTU (maximum transmission unit) on the port channel	Type 1, interface	The secondary suspends its member of that vPC.
Spanning tree port type or guard on the port channel	Type 1, interface	The secondary suspends its member of that vPC.
Allowed VLAN list on the trunk	VLAN level	The vPC stays up, but only VLANs allowed on both peers come up on it. The rest are suspended on that vPC, so one or two subnets fail.
An SVI shut on one peer, or missing from it	Type 2	Nothing is suspended, but a host that sends to that peer loses its gateway for that VLAN. It shows up as intermittent loss for some hosts only.
HSRP settings for a VLAN, such as a different group or virtual address	Type 2	Nothing is suspended, but the gateway can split or flap between the peers.
An ACL (access control list) or QoS policy on the port channel	Type 2	Nothing is suspended, but traffic is filtered or queued differently depending on which link the server's hash picks. These are the hardest faults to find.

Safe changes

Every change with a tested way back

NX-OS has two tools that make changes safer than on classic IOS: checkpoints with rollback, and configuration sessions that apply as one. I use both on every production change. Step through a change that fails its checks and comes back out cleanly.

Swipe sideways to see the whole diagram.

Step 1

Before the change I run checkpoint pre_chg123. It saves the running configuration under that name. I also keep a copy on flash with checkpoint file bootflash:pre_chg123.cfg.
The change goes in: a new VLAN 300 and a port brought up for it. show running-config diff shows what is not yet saved to the startup configuration.
show diff rollback-patch checkpoint pre_chg123 running-config shows the exact commands a rollback would run. I paste this into the change record as the back out plan, so it is reviewed with the change.
My post change checks find a fault. Maybe the new VLAN clashes with one at a remote site. The change window allows for a rollback, so I do not try to fix forward.
rollback running-config checkpoint pre_chg123 runs the patch. The default is atomic: if any command fails, nothing is applied. best-effort and stop-at-first-failure exist, but I avoid them on production switches.
I compare again and expect no differences, then repeat the checks. show checkpoint summary lists the checkpoints, including any system checkpoints NX-OS took when a feature was disabled.

Configuration sessions

Command	What it does
`configure session CHG123`	Opens a named session. Commands are held in the session, not applied.
`show configuration session CHG123`	Lists the commands waiting in the session, for review.
`verify`	Checks the session against the current configuration and the hardware, without applying it.
`commit`	Applies the whole session. If a command fails, the session is rolled back, so the switch is never left half changed.
`abort`	Throws the session away.

Rules I keep

One checkpoint per change. Named after the change ticket, taken in the same window, and kept on flash until the change is closed.
The rollback patch is the back out plan. I attach show diff rollback-patch output to the change, so the reviewer sees the undo as well as the do.
Sessions for anything that must land together. An ACL (access control list) and the interface that uses it, or a VLAN and its SVI, go in one session so they apply as one.
Both vPC peers in one change. Type 1 settings change on both peers in the same window, with checks between, so the pair never runs mismatched for long.
Templates beat memory. Configuration comes from templates and a source of truth, as on the network automation page, so manual changes are the exception.

Upgrades and releases

Choosing a release, then upgrading without an outage

Upgrades are where many avoidable data centre outages start. The cause is rarely the software itself. It is a release chosen without reading its lifecycle, or an upgrade nobody checked would be disruptive. First the release model and a decoder, then what each kind of upgrade costs.

How NX-OS releases are numbered

A.B(C)x

NX-OS numbers each release as major train A.B, release C within the train, then a letter: F for a feature release, M for a maintenance release.

Feature phase

About the first 12 months of a train, typically three F releases with new features and new hardware.

Maintenance phase

The next 18 months, with M releases that fix bugs and security issues only. This is where I standardise.

Extended support

From 30 months after first shipment, security fixes only. Last date of support comes at 54 months.

Train	End of software maintenance	Last date of support	Phase in October 2026
10.2	30 November 2023	2025	Past support
10.3	30 November 2024	28 February 2027, extended for EX platforms	Security fixes only
10.4	28 February 2026	29 February 2028	Security fixes only
10.5	28 February 2027	28 February 2029	Maintenance releases
10.6	29 February 2028	28 February 2030	Maintenance releases

From Cisco's NX-OS software release guide, checked in October 2026. A new major train starts in the third quarter of each year.

NX-OS release decoder

Type a release, such as 10.6(4)M, or an image name such as nxos64-cs.10.4.8.M.bin. The decoder reads the train and the release type, and where the train is in Cisco's lifecycle today.

Release or image name

Train: 10.4
Release in the train: 8
Type: M: maintenance, bug and security fixes only
Train milestones: Software maintenance ended 28 February 2026. Last date of support 29 February 2028.

In October 2026 the 10.4 train gets security fixes only, until 29 February 2028. I would plan the move to 10.6.

Swipe sideways to see the whole diagram.

On Cloud Scale fixed switches, a non disruptive ISSU keeps the forwarding ASICs running while the supervisor software restarts on the new release. Cisco quotes 90 to 120 seconds without a control plane and no data plane loss. Routing neighbours hold my routes through graceful restart meanwhile.

Enhanced ISSU starts the new release as a second supervisor in a container, then switches to it, so the control plane is away for only 3 to 6 seconds. It is on by default on the GX2A and GX2B models, and on GX and FX3 models from 10.3(3)F. FX and FX2 models need boot mode lxc and a reload first. If the upgrade includes a kernel patch, it falls back to a standard ISSU.

A disruptive upgrade reloads the switch: about five minutes, depending on scale, with no control or data plane. In a vPC pair, hosts on vPCs carry on through the peer after a short blip. Single attached hosts lose service for the whole reload, one more reason I avoid orphan ports.

Modular chassis such as the 9500 and 9800 have no non disruptive ISSU, so I drain the switch with maintenance mode first, upgrade it, and bring it back. The chassis is down for the reload, but traffic has already moved to the other spines, so hosts see nothing.

Animation

The commands in the order I run them. show install all impact says whether the upgrade will be disruptive before anything changes. EPLD (electronic programmable logic device) upgrades reload the module, so they get their own window. File names are examples.

show version
show install all impact nxos bootflash:nxos64-cs.10.4.8.M.bin
show file bootflash:nxos64-cs.10.4.8.M.bin md5sum
install all nxos bootflash:nxos64-cs.10.4.8.M.bin non-disruptive
show install all status
install epld bootflash:n9000-epld.10.4.8.M.img module all

Maintenance mode

Take a switch out of service before anything breaks

GIR (graceful insertion and removal) is the NX-OS maintenance mode. It moves traffic off a switch by isolating it, works on it, then puts it back and proves nothing changed. I use it for upgrades on modular chassis, for hardware work, and on any switch where I want the change to be boring.

Swipe sideways to see the whole diagram.

Step 1

Both leaves carry traffic. The server is on a vPC, and each leaf has a path to both spines.
system mode maintenance on Leaf B starts GIR (graceful insertion and removal). NX-OS first takes a snapshot called before_maintenance: neighbours, routes, interfaces and vPC state.
The maintenance profile isolates the routing protocols, which steer traffic away while staying up, and shuts the vPC domain. Traffic moves to Leaf A with no loss, because nothing failed. It was drained.
Leaf B is upgraded and reloads, or gets whatever work it needed: an optic, a cable, a line card. Leaf A carries everything. Nothing on B is in the traffic path.
no system mode maintenance brings B back. Its vPCs and routing return, and traffic shares across both leaves again.
NX-OS takes an after_maintenance snapshot, and show snapshots compare before_maintenance after_maintenance shows any difference. Only then do I move to the next switch.

My upgrade runbook for a vPC pair

Read first. Release notes, open caveats and the upgrade path for the target release. Some jumps need an intermediate release.
Check the impact. show install all impact on each switch, and the image checksum against Cisco's published value.
Back up. A checkpoint, a copy of the configuration off the switch, and show tech-support saved in case I need TAC (Cisco's Technical Assistance Center).
Baseline. Neighbour counts, route counts, MAC and ARP counts, vPC state and interface errors, captured as a snapshot or with pyATS.
One peer at a time. Drain, upgrade and return one switch. Compare the snapshots and wait until the pair is stable before touching its peer.
Then EPLD. If the release note asks for an EPLD upgrade, it goes in its own window, also one peer at a time.
Close out. Record the new release in the source of truth, and update the lifecycle plan with the next milestone.

Programmability

Switches that build themselves and report on themselves

NX-OS has offered APIs, Python and a Linux shell for years, so a Nexus estate can be built and checked by code. The deeper method, with a source of truth, pipelines and testing, is on the network automation page. Here are the NX-OS pieces: zero touch provisioning, the APIs and streaming telemetry.

Swipe sideways to see the whole diagram.

Step 1

A new switch boots with no startup configuration, so POAP (Power On Auto Provisioning) starts by itself. Someone at the console can abort it, which is how I build a switch by hand in the lab.
The switch asks DHCP (Dynamic Host Configuration Protocol) for an address on mgmt0 or a front port. The reply also names the server holding the POAP script and the script's file name, through DHCP options 66 or 150 and 67.
The switch downloads the POAP script, a Python file I control, and runs it. All the logic lives in the script, so I can change how switches are built without touching DHCP.
The script picks this switch's configuration by serial number, MAC address or the neighbour it is cabled to, downloads it with the right NX-OS image, and checks each file's MD5 (Message Digest 5) hash before using it.
The switch reloads on the new release and applies its configuration. It comes up with its hostname, addresses, AAA and management settings in place.
From here the switch is managed like every other: Ansible or Nexus Dashboard pushes the rest of the configuration and checks it. Nobody needs to touch a console cable.

Swipe sideways to see the whole diagram.

Animation

Requests in orange, structured replies in teal, and streamed telemetry in purple. Nothing here scrapes CLI text.

The interfaces NX-OS offers

Interface	How I turn it on	What I use it for
NX-API CLI	`feature nxapi`	Send ordinary CLI commands over HTTPS (secure HTTP) and get JSON (JavaScript Object Notation) back. Quick wins for scripts.
NX-API REST	`feature nxapi`	Read and write the switch's object model directly over REST (representational state transfer), the same model the CLI drives.
Model driven telemetry	`feature telemetry`	Stream counters and state to a collector over gRPC (gRPC Remote Procedure Calls), instead of polling with SNMP (Simple Network Management Protocol).
NETCONF (Network Configuration Protocol), RESTCONF and gNMI	`feature netconf`, `feature restconf`, `feature grpc`	RESTCONF is the web form of NETCONF, and gNMI (gRPC Network Management Interface) runs over gRPC. All three use YANG (Yet Another Next Generation) data models, both Cisco native and OpenConfig (`feature openconfig`), for tools that speak standards.
On box Python	`python`, then `from cli import clid`	Small checks and fixes that run on the switch, reading command output as JSON.
Bash and guest shell	`feature bash-shell`, `guestshell enable`	Linux tools on the switch. The guest shell is a separate container for my own scripts and agents.
Ansible	The `cisco.nxos` collection	Resource modules such as `nxos_vlans` and `nxos_interfaces`, driven from a source of truth.
POAP	DHCP and a script server	Zero touch first boot, as above.

A small on box check. The field names come from the JSON output of show vpc brief | json on the release in use, so I read that first.

# on the switch: list vPCs that are not up
from cli import clid
import json
vpcs = json.loads(clid("show vpc brief"))
rows = vpcs["TABLE_vpc"]["ROW_vpc"]
rows = rows if isinstance(rows, list) else [rows]
for r in rows:
    if r["vpc-port-state"] != "1":
        print(r["vpc-id"], r["vpc-ifindex"], "is not up")

Visibility tools

Seeing the packets a switch never shows you

A modern Nexus forwards almost everything in hardware, so the CPU (central processing unit) never sees most traffic. Ethanalyzer, SPAN (Switched Port Analyzer), ERSPAN (encapsulated remote SPAN) and ELAM (embedded logic analyser module) each see different traffic, and picking the right one saves an hour of looking in the wrong place. Choose a tool to see what it can and cannot see.

Swipe sideways to see the whole diagram.

Ethanalyzer is a packet capture built into NX-OS. It sees traffic to and from the supervisor CPU, such as routing, ARP and management, through the inband or mgmt0 interface: ethanalyzer local interface inband capture-filter "tcp port 179" write bootflash:bgp.pcap. Server traffic switched in hardware never reaches it.

SPAN copies traffic from ports or VLANs to a port where an analyser is plugged in. On NX-OS a monitor session is shut by default, so it needs no shut. It is the first tool for traffic that never touches the CPU.

ERSPAN wraps the copies in GRE (Generic Routing Encapsulation) and sends them across the IP (Internet Protocol) network to an analyser anywhere. I check the extra load on the path, and the MTU, before I start one.

With no analyser nearby, a SPAN session can use the CPU as its destination (destination interface sup-eth0), then Ethanalyzer captures the copies. It is rate limited to protect the supervisor, so it suits small flows and short captures.

ELAM traces one chosen packet through the forwarding ASIC and shows each lookup and its result. It answers questions such as why this packet went out of that port. I use it with TAC, because its output needs care to read.

Animation

Commands I reach for first

Question	Command	What it tells me
What changed, and when?	`show logging logfile`, `show accounting log`	Events and every configuration command, with user and time
What did a protocol just do?	`show ip ospf event-history adjacency`, `show bgp event-history detail`	The internal history each process keeps, even with no debug running
Does hardware match software?	`show consistency-checker l2 module 1`, `show consistency-checker forwarding ipv4 vrf default`	Whether MAC and route tables in the ASIC match what the supervisor expects
Is a table full?	`show hardware access-list resource utilization`	TCAM (ternary content addressable memory) use per region, for ACLs and QoS
Are frames being damaged?	`show interface counters errors`	CRC (cyclic redundancy check) and other errors per port, the first sign of a bad optic or cable
Everything for TAC	`show tech-support details`, `show tech-support vpc`	A full snapshot for a support case, taken before anything is reloaded

Security

Protecting the switch as well as the traffic

A data centre switch is a target in its own right. If its CPU is swamped, routing drops and the whole fabric suffers. I protect the control plane with CoPP (control plane policing), keep the management plane apart, and give people only the access they need.

Swipe sideways to see the whole diagram.

Animation

Routing in teal and management in blue pass through their CoPP classes. The flood in red is policed, so only a trickle reaches the CPU.

How I harden a Nexus switch

Control	How	Why
CoPP	`copp profile strict`, checked with `show policy-map interface control-plane`	Strict is the default chosen at setup. Moderate, lenient and dense profiles exist, and I only move off strict with evidence from the drop counters.
Hardware rate limiters	`show hardware rate-limiter`	Separate limits for exception traffic, such as packets with an expired TTL, that would otherwise reach the CPU.
Fewer features	`show feature`	Every feature I do not enable is a process that is not running. Telnet is off unless someone turns it on with `feature telnet`.
AAA and roles	`feature tacacs+`, command authorisation and accounting	Every login and command is checked and recorded. Built in roles such as `network-admin` and `network-operator`, plus my own roles for read only and change teams.
Management plane	mgmt0 in VRF `management`, an ACL on the VTY (virtual terminal) lines	Management reachable only from jump hosts, never from production VLANs.
Protocols	SNMPv3, NTP with authentication, SSH (Secure Shell) only	No community strings or clear text logins, and clocks I can trust for logs.
Images	`show file bootflash:... md5sum` against Cisco's published value	Only images I have checked go on a switch, and only releases still inside their support dates.
Routing	Authentication on OSPF and BGP sessions, prefix limits and filters	A neighbour cannot inject routes or flap me into an outage. Wider routing security is on the security page.

Troubleshooting

Finding a Nexus fault layer by layer

Most Nexus faults fall into a handful of patterns, and nearly all of them leave a clear trace in one command's output. Choose a symptom to see where it usually sits and what I run first. My general method is on the troubleshooting page.

Swipe sideways to see the whole diagram.

vPCs down on one switch only. The peer link failed while the keepalive stayed up, so the secondary suspended its vPCs, as designed. Hosts on vPCs carry on through the primary. I check the peer link members with show port-channel summary and their optics, restore the link, and watch the vPCs return after delay restore.

One vPC suspended after a change. A type 1 setting differs between the peers, usually because a change went onto one peer only. The output shows the two values side by side. I make both sides match in one change, and use the consistency checker above to see how each setting behaves.

A single attached server went dark in a failure. It was an orphan port on the secondary, which suspended its vPC VLANs. I dual attach it, or add vpc orphan-port suspend to its port so the server fails over to its other network card.

Roles flipped when a switch rejoined. A switch that had been isolated came back with its sticky primary bit still set and took the primary role, causing about 40 seconds of disruption. Before rejoining an isolated switch, I check the bit is clear and that the live switch has the lower role priority.

High CPU and routing flaps. Something is sending traffic to the CPU: a loop, a scan, glean traffic for hosts that do not exist, or a misbehaving neighbour. show processes cpu sort shows which process is busy, and the CoPP counters show which class is dropping. I find the source rather than loosen CoPP.

A new ACL or QoS policy will not apply. The TCAM region it needs is full, or has no space carved for it. I trim the policy first. If the region really must grow, I change the carving with hardware access-list tcam region in a window, because it takes a reload.

Small packets pass, large ones drop. One interface or SVI on the path still has the default MTU. I test with a full size ping and the do not fragment bit, find the hop that fails, and set 9216 everywhere in the fabric from the template.

Errors on one link. CRC errors point at layer 1: a dirty or failing optic, a damaged cable or a bad port. show interface ethernet1/1 transceiver details shows the optic's light levels. I clean or replace the optic first, then the cable.

A protocol restarted by itself. A process crashed and the system manager restarted it. show cores lists the core files. I save them with show tech-support details, search Cisco's bug database for the release, and open a TAC case before anyone reloads the switch.

An upgrade turned out disruptive. The upgrade path, an EPLD change or the platform did not support a hitless upgrade, and nobody read the impact table first. I run the impact check on every switch before the window, and plan maintenance mode wherever the result says reset.

Animation

The commands I start with

Question	Command	What it tells me
Is the vPC pair healthy?	`show vpc brief`, `show vpc peer-keepalive`	Peer link, keepalive, roles, consistency and every vPC's state
Which ports are exposed?	`show vpc orphan-ports`	Ports in vPC VLANs on one peer only
Are the bundles up?	`show port-channel summary`, `show lacp neighbor`	Members in and out of each port channel, and what the far end says
Is the CPU busy, and why?	`show processes cpu sort`, `show system resources`	The busiest processes, and memory and load
Has anything crashed?	`show processes log`, `show cores`, `show system reset-reason`	Process exits, core files and why the switch last reloaded
Where is a MAC or route really going?	`show mac address-table address 0050.56aa.0001`, `show ip route 10.1.20.0`	The port or next hop the switch has chosen
What changed?	`show accounting log`, `show diff rollback-patch checkpoint pre_chg123 running-config`	Who changed what, and the difference from the last good checkpoint

I change one thing at a time and repeat the same check after each change. For faults in VXLAN EVPN fabrics, the checks on the VXLAN EVPN page follow on from these.

Engagement

How I run a Nexus programme, from end of life notice to steady state

Nexus work usually starts with an end of life notice, a capacity limit or a new data hall. I follow the same path each time. I agree why the estate must change, find out exactly what is there and design every role as a template. Then I prove failures in a lab, migrate in waves, and keep releases and hardware inside their support dates.

Swipe sideways to see the whole diagram.

Step 1

Nexus work usually starts with an end of life notice, a capacity limit or a new fabric. I agree why the estate must change and by when, which rooms and services are in scope, and when change windows are allowed. A RACI (responsible, accountable, consulted, informed) chart records who decides.
I turn the drivers into requirements: server ports and speeds with growth, uplink oversubscription, availability, upgrade windows, segmentation and monitoring. Each target gets a number and an owner. Hitless upgrades and no outage from a single switch failure are usually the targets that shape the design most.
I collect inventory, releases and configuration from every switch, check each against Cisco's end of life notices and release lifecycle, and audit vPC domains, orphan ports and fabric extenders. Interface counters, flow records and TCAM use show what the traffic really needs. Findings here often change the plan.
The high level design places each Nexus family by role and sets the vPC, routing and segmentation model. The low level design numbers every vPC domain and keepalive, picks the target release from the maintenance phase of a train, sets CoPP, AAA and MTU, and turns each role into a template.
I build the first pair in the lab with POAP and the templates, then fail each part on purpose: the peer link, the keepalive, a member link and a whole switch, and confirm the design behaves as written. Migration runs in waves through change control, with snapshots before and after each wave.
Before handover, monitoring alerts on peer link and keepalive state, consistency failures, interface errors, CoPP drops and TCAM use. Operations teams get runbooks for the faults on this page, the upgrade and maintenance mode procedures, and a route to Cisco TAC. Hypercare ends when the agreed criteria are met.
Each year I review the release plan against Cisco's lifecycle table and move the estate to a maintenance release in a supported train. Security advisories are checked as they appear. End of life notices go into a refresh plan, so no switch reaches its last date of support unnoticed.

Functional requirements

What it must do.

Give every server two links to two switches, with nothing blocked by spanning tree.
Route between VLANs at the leaf pair, with a gateway that survives either switch failing.
Keep every dual attached service up while one switch of a pair is upgraded or replaced.
Separate production, storage and management traffic, and keep management on its own network.
Build new switches from templates, with no hand configuration.
Report state, errors and capacity to the operations teams as it happens.

Non functional requirements

How well it must do it, with a number wherever one can be agreed.

Availability: No outage for dual attached hosts when any one switch, link or optic fails.
Upgrades: Each switch upgraded in its own window with no loss for dual attached hosts, using ISSU or maintenance mode.
Oversubscription: Leaf uplinks no worse than 3 to 1 against server ports at full speed (typical).
Convergence: Traffic moved to the surviving link within a second or two of a failure (typical).
MTU: 9216 bytes on every fabric link, port channel and SVI, proved with full size tests at acceptance.
Release: Every switch on a maintenance release, in a train inside its support dates.
Operability: vPC state, keepalive, errors, CoPP drops and TCAM use in monitoring, with thresholds.
Security: TACACS+ logins with command accounting, SSH only, SNMPv3 and CoPP on every switch.

Numbers are typical starting points. Each organisation agrees its own.

Who I talk to

Data centre and facilities owners, who know the rooms, racks and power
Server, storage and virtualisation teams, who own what plugs in
Application owners, for critical services and change windows
Network operations, who know the faults and the workarounds
The Cisco account team, for end of life dates and support contracts

How I find out

Inventory, release and configuration collected from every switch
A vPC, port channel and orphan port audit on every pair
Interface counters, flow records and TCAM use over a busy period
Releases and hardware checked against Cisco's lifecycle notices
A walk through past incidents and changes for each pair

Discovery tools

Tool or source	What it tells me
`show inventory`, `show version`, `show module`	Models, serial numbers, releases and modules
`show vpc brief`, `show vpc orphan-ports`	vPC health and exposed single attached ports
`show hardware access-list resource utilization`	How close each switch is to its TCAM limits
NetFlow or sFlow records	Who talks to whom, and how much
Nexus Dashboard	Inventory, anomalies and advisories across the estate
Ansible or pyATS collection runs	The same facts from every switch, saved as data
Cisco end of life notices and the NX-OS release guide	Which hardware and trains are still supported

Most Nexus faults show up first as one vPC, one link or one busy CPU. I work from the evidence the switch already keeps before I change anything.

My approach

Scope. Which hosts, which VLANs, which switch, and since when? What changed: a release, a template, a cable or a server?
Pair state. show vpc brief on both peers: peer link, keepalive, roles and consistency, before anything else.
Path. Where the MAC address or route really points, with show mac address-table and show ip route, on both peers.
Evidence. Logs, accounting, event history and the difference from the last checkpoint, to find what changed.
Capture. Ethanalyzer for control traffic, SPAN or ERSPAN for data, and ELAM with TAC for one packet.
Fix and prove. One change at a time through change control, the same checks repeated, and the cause recorded for problem management.

Faults to check first

A type 1 change made on one peer only
Orphan ports losing service in a peer link failure
A keepalive down for weeks without an alert
MTU left at 1500 on one SVI or link
TCAM regions too small for a new policy
An upgrade that was disruptive because nobody checked the impact

Layer by layer

Layer	What I check
1 · Physical	Optics and light levels, CRC errors, and the peer link and keepalive cabled on separate paths.
2 · Data link	Port channels, LACP, vPC consistency, spanning tree roles, orphan ports and the MAC table on both peers.
3 · Network	SVIs and HSRP or anycast gateways, routing adjacencies over vPC, ARP synchronisation and the route table.
4 · Transport	MTU along the path with full size tests, and TCP (Transmission Control Protocol) retransmissions for slow transfers.
5 to 7 · Application	Which service is affected, what it depends on, and whether the switch is really in its path.

My general method, with worked faults, is on the troubleshooting page.

Projects

Where I have used it

Nexus has run through my data centre work for more than ten years: managed infrastructure on ACI and Nexus at Manx Telecom, public sector modernisation at Civica, and Nexus estates in my current transformation work at XIDA.

XIDA · 2025 to now

Nexus in a network and security transformation

Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access and AWS (Amazon Web Services), Azure and Google Cloud environments.

Manx Telecom · 2016

ACI and Nexus for over 300 customer sites

Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI and Nexus. The fabric side of that work is on the VXLAN EVPN and ACI page.

Civica (Luton Borough Council) · 2017

Public sector modernisation on Nexus

Public sector network modernisation on Meraki, Nexus, FortiGate and Check Point, leading 7 engineers and third party suppliers.

iSystems Integration · three engagements, 2015 to 2019

Nexus in cloud and disaster recovery designs

Cloud, SDN (software defined networking), disaster recovery and security architecture on Nexus, Viptela, Palo Alto, F5 and Aruba for the firm's enterprise clients.

BNY Mellon · 2018 to 2019

A data centre refresh across Europe, the Middle East and Africa

Audit, high and low level design and implementation of LAN (local area network), WAN (wide area network), SD-WAN and data centre infrastructure, replacing end of life equipment with minimal impact on global operations. The estate was kept compliant with MiFID II throughout.

AVEVA · 2020 to 2021

Data centre transitions at 99.99% uptime

Safeguarded over £200M of transactional services through data centre transitions at 99.99% uptime, leading 7 engineers and architects and coordinating 5 vendors.

Backed by CCNP (Cisco Certified Network Professional) Enterprise, CCDP (Cisco Certified Design Professional), CCNP Routing and Switching, and a pass in the CCIE (Cisco Certified Internetwork Expert) Routing and Switching written exam. See all credentials

Explore

More areas I design and deliver

Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.

Cisco Nexus, from first boot to the next upgrade

A switch operating system built to keep forwarding

Habits that change when I move from IOS

Features start switched off

Management has its own routing table

Show commands work anywhere

Saving and comparing

Linux underneath

One image across families

Features I enable most

Which Nexus, for which job

How I choose

Leaf pairs

Spines

Old estates

Building a vPC pair that fails safely

Timers and the values I choose

The settings that make a vPC pair behave like one switch

vPC consistency checker

Every change with a tested way back

Configuration sessions

Rules I keep

Choosing a release, then upgrading without an outage

How NX-OS releases are numbered

A.B(C)x

Feature phase

Maintenance phase

Extended support

NX-OS release decoder

Take a switch out of service before anything breaks

My upgrade runbook for a vPC pair

Switches that build themselves and report on themselves

The interfaces NX-OS offers

Seeing the packets a switch never shows you

Commands I reach for first

Protecting the switch as well as the traffic

How I harden a Nexus switch

Finding a Nexus fault layer by layer

The commands I start with

How I run a Nexus programme, from end of life notice to steady state

Functional requirements

Non functional requirements

Who I talk to

How I find out

Discovery tools

My approach

Faults to check first

Layer by layer

Service model

Monitoring

Runbooks

Change

Upgrades

Contracts

Where I have used it

Nexus in a network and security transformation

ACI and Nexus for over 300 customer sites

Public sector modernisation on Nexus

Nexus in cloud and disaster recovery designs

A data centre refresh across Europe, the Middle East and Africa

Data centre transitions at 99.99% uptime

More areas I design and deliver

Data centre and cloud (this area)

Foundations

Networks and security

Industries

Governance and service