Troubleshooting

Finding the fault, one layer at a time

When something breaks I follow the same method every time: define the problem, gather facts, test one idea at a time and fix through change. This page shows that method, the tools I reach for, and a library of real faults with the checks and commands that find them.

OSI model
ping and traceroute
MTU
Packet capture
Wireshark
NetFlow
IP SLA
Escalation

See the method Where I use it

Live view · the flow stops at the firewall, and the capture proves it

Method

The same method, whatever has broken

When a network breaks, the pressure is to try things. I do the opposite. I define the problem, gather facts and test one idea at a time, because a fix I cannot explain is only a guess. The loop below is the method I use on a lab switch and on a priority one outage alike.

Swipe sideways to see the whole diagram.

Step 1

"The network is down" cannot be tested. I ask who is affected, what exactly fails, where they are, when it started, whether it is constant or intermittent, and the exact error. An "is and is not" list helps: if A fails and B works, I look at what differs.
One user, one floor, one site or every site? One application or all of them? Scope sets the priority, the SLA (service level agreement) clock, and whether I call a major incident.
I read before I touch: monitoring graphs, central logs, alarms, the ticket history and read only show commands. Time synchronised with NTP (Network Time Protocol) lets me put events from many devices in order.
I check the change calendar and the configuration backups for anything in the last day or two. A diff of the running configuration against last night's backup often finds the cause in a minute.
I pick the simplest cause that explains every symptom, including the things that still work. If the idea does not explain the working cases, it is the wrong idea.
I say what I expect before I test, change one thing at a time and record it. If the result does not change, I undo it before trying the next idea. Simulation tools, such as a firewall's packet trace, test without touching traffic.
Even at 2am the fix goes through change control. An emergency change can take five minutes, but it records what I did and how to undo it.
The user confirms, monitoring is green and the counters stay clean. I compare the post checks with the pre checks line by line. I write the cause, the fix and the evidence in the ticket. A repeat fault becomes a problem record. If the fault is still there, the loop starts again.

Read before I touch

Most outages that get worse do so because someone changed something while guessing. I gather facts with read only commands first, and I check the redundancy state before I shut anything.

Explain every symptom

A root cause has to explain what fails and what still works, fit the timeline, and stop the fault when it is removed. If it does not, I keep looking.

Leave a trail

Every command, change and result goes in the ticket with a time. The next engineer, the problem manager and the post incident review all depend on it.

Approaches

Six ways into a fault, and when I use each

The OSI model (Open Systems Interconnection) splits networking into seven layers. It gives me a map. The approach decides where on that map I start, and the symptom decides the approach.

Swipe sideways to see the whole diagram.

Bottom up. I start at Layer 1 and climb: link, errors, then VLANs (Virtual Local Area Network) and MAC (Media Access Control) tables, then routing. It is slow for most faults but right for physical symptoms, such as rising CRC (cyclic redundancy check) errors after a patching change. show interfaces Gi1/0/12 and show interfaces transceiver detail come first.

Top down. I start at the application. If one application fails and everything else works, the cable is almost certainly fine. I check DNS (Domain Name System), the certificate, the port and the server first. nslookup, curl -v and the application log often answer it.

Divide and conquer. My default for "cannot reach" faults. I start at Layer 3 with a sourced ping and a traceroute. If they work, Layers 1 to 3 are fine and I look up. If they fail, I look down. ping 10.20.1.10 source Vlan20 repeat 200 also shows loss that five pings would miss.

Follow the path. I walk the flow device by device: MAC and ARP (Address Resolution Protocol) tables, routes, firewall sessions, until I find the last device that saw it. Captures on both sides of a firewall prove whether it was dropped there or never arrived.

Compare. When one site or one switch fails and another identical one works, I compare them line by line: configuration, software version, routes, trunks and MTU (maximum transmission unit). Configuration backups make this fast.

Swap. Replacing a suspect cable, optic or port with a known good one is quick, and sometimes the fastest way to restore service. It proves which part failed, not why, so I still send the part for analysis if faults repeat.

Animation

Layer by layer

What I check at each layer

When I work bottom up, this is my checklist. The commands are Cisco IOS XE unless I say otherwise. Each step shows the questions on the left and the commands below.

Swipe sideways to see the whole diagram.

Step 1

show interfaces status, show interfaces Gi1/0/1 | include error|CRC|duplex, show interfaces transceiver detail. Counters are totals since the last clear, so I clear them and measure a rate. CRC errors usually mean a cable, patch lead or dirty fibre. A duplex mismatch shows late collisions on one side and CRC errors on the other.
show mac address-table address 0011.2233.4455, show interfaces trunk, show spanning-tree vlan 20, show interfaces status err-disabled. If the MAC address is not learned where I expect, the problem is at or below this layer.
show ip route 10.1.1.10, show ip arp 10.1.1.10, show ip cef 10.1.1.10 detail, a sourced ping and a traceroute. The routing table shows what should happen; CEF (Cisco Express Forwarding) shows what the hardware will really do.
On Windows Test-NetConnection 10.1.1.10 -Port 443, on Linux nc -zv 10.1.1.10 443, from a Cisco device telnet 10.1.1.10 443. Then the firewall session table and a capture on both sides. A SYN (synchronise) with no reply points at the path or the server; a reset points at a firewall or a closed port.
This layer covers login and session set up for SSH (Secure Shell), RDP (Remote Desktop Protocol) and SMB (Server Message Block). Session faults look like random drops. If long running sessions die at the same age, a firewall or proxy idle timeout is the usual cause. I compare the timeout with the application's keepalive.
openssl s_client -connect app.example.local:443 shows the certificate chain, dates and the TLS (Transport Layer Security) version agreed. A device whose clock is wrong rejects every certificate, so NTP comes before PKI (public key infrastructure).
nslookup app.example.local, curl -v https://app.example.local/health and the application logs. If the network passes every lower test, I hand the evidence to the application team rather than keep searching the network.

First five minutes

A priority one lands. The first five minutes.

A priority one incident means a critical service is down for many people. What I do in the first five minutes decides how the next hour goes. I do not start typing commands on the first device I think of.

Swipe sideways to see the whole diagram.

Step 1

I accept the incident so the SLA response clock is met, and tell the service desk who has it. From now on everything goes on a timeline with times, because the post incident review will need it.
I read the ticket and the alarms properly. The first alarm in time is usually nearest the cause; the rest are often the knock on effects.
I check the change calendar and recent patches, and use the monitoring map to see how wide the fault is. If the scope is bigger than the ticket says, I raise the priority or ask for a major incident.
A read only look at the device at the centre of the service: show logging, show interfaces status, routing neighbours, show standby brief or show vpc brief and show processes cpu sorted. I learn the redundancy state before I touch anything, so I never shut the only working path.
If it is big, I ask the incident manager to open a bridge. The first update goes out in plain English with what we know, what we are doing and the time of the next update.

Toolkit

The tools I reach for, and how I read them

Most faults are found with a handful of tools used well. These are the ones I use every time, with the detail that turns their output into evidence.

Swipe sideways to see the whole diagram.

Step 1

Every IP (Internet Protocol) packet carries a TTL (time to live) that each router lowers by one. Traceroute sends its first probes with a TTL of 1, so the first router drops them and sends back an ICMP (Internet Control Message Protocol) time exceeded message. That reply tells me the first hop and the round trip time.
The next probes start with a TTL of 2. They pass R1 and expire at R2, which replies in the same way. Three probes per hop give three times, so I can see jitter as well as delay.
R3 does not reply. Many routers rate limit or block ICMP replies to protect their CPU (Central Processing Unit). The stars do not mean traffic stops there, because the next hop still answers.
The probes reach the server. Cisco and Linux traceroute send UDP (User Datagram Protocol) probes to high ports, so the server answers "port unreachable". Windows tracert sends ICMP echo requests and gets an echo reply.
Stars at one hop with later hops answering are fine. Stars from one hop to the end show where the path stops. A jump in delay that carries on to the end is real; a jump at one hop only is that router being slow to reply. I trace from both ends, because the return path can differ. On Cisco I use traceroute 203.0.113.10 source Loopback0 numeric.

Swipe sideways to see the whole diagram.

It fits. I test the path MTU with the DF (do not fragment) bit set. On Windows ping -f -l 1372 192.0.2.10 sends 1372 bytes of data plus 28 bytes of IP and ICMP headers: 1400 in total, which fits the tunnel.

Too big. ping -f -l 1472 192.0.2.10 makes a 1500 byte packet. Router A cannot fit it into the tunnel and cannot fragment it, so it drops it and sends ICMP type 3 code 4, "fragmentation needed", with the MTU it can carry. The host lowers its packet size. That is path MTU discovery working.

Black hole. If a firewall drops that ICMP message, the host never learns. Pings and small requests work, but large transfers, file copies and some web pages hang. This is one of the most common "the network is slow" faults after a VPN (Virtual Private Network) or overlay is added.

The fix. Allow ICMP type 3 code 4, and clamp the TCP (Transmission Control Protocol) MSS (maximum segment size) on the tunnel: 1400 minus 40 bytes of IP and TCP headers gives ip tcp adjust-mss 1360. On Linux the test is ping -M do -s 1372 192.0.2.10; on Cisco IOS ping 192.0.2.10 size 1400 df-bit, where the size includes the headers.

Animation

Ping, properly

I source the ping from the user's gateway or a loopback, send enough to see loss (repeat 200), and sweep sizes with the DF bit for MTU. A ping that works proves Layers 1 to 3 for that path only, in that direction.

Show before debug

Targeted show commands answer most questions. When I must debug, I check CPU headroom, turn off console logging, log to a buffer, narrow it with a condition or an access list, and have undebug all ready before I start.

Logs in one place

One device's log tells one story. Central syslog and the SIEM (security information and event management) tell the estate's. With NTP on every device, I put link, spanning tree, routing and HA (high availability) events in order and the first one usually points at the cause.

Flow records

SNMP (Simple Network Management Protocol) shows that a link is full. NetFlow or IPFIX (IP Flow Information Export) shows who filled it: the top talkers, applications and conversations. NetFlow v5 has a fixed format, v9 and Flexible NetFlow use templates, and IPFIX is the open standard. sFlow samples packets instead. Sampling cuts the load on the device, and also the accuracy.

Monitoring tells me where to look

A monitoring platform such as SolarWinds shows where and when: availability, errors and utilisation against a baseline. A configuration tool shows what changed, with diffs against the approved baseline. A capture then shows why. I use them in that order.

IP SLA probes

Cisco IP SLA sends test traffic on a schedule and measures loss, delay and jitter. Tied to tracking, it can also move a default route when a circuit fails in a way the interface does not show.

Simulate before changing

Vendors give tools that test a flow against the policy without touching traffic. I use them to prove a hypothesis before any change.

Tools that prove where a packet was dropped

Platform	Simulate the flow	See the real packets
Cisco ASA	`packet-tracer input inside tcp 10.1.1.10 51000 192.0.2.10 443 detailed`	`capture CAP interface inside match ip host 10.1.1.10 host 192.0.2.10`, `show asp drop`
Cisco FTD (Firepower Threat Defense)	`packet-tracer` from the diagnostic command line	`capture` with trace, and `system support trace`
FortiGate	`diagnose debug flow filter addr 10.1.1.10`, `diagnose debug flow trace start 20`, `diagnose debug enable`	`diagnose sniffer packet any "host 10.1.1.10 and port 443" 4`
Palo Alto PAN-OS	`test security-policy-match from trust to untrust source 10.1.1.10 destination 192.0.2.10 protocol 6 destination-port 443`	`show session all filter source 10.1.1.10`, `show counter global filter delta yes packet-filter yes`
Juniper SRX	`show security match-policies from-zone trust to-zone untrust source-ip 10.1.1.10 destination-ip 192.0.2.10 source-port 51000 destination-port 443 protocol tcp`	`show security flow session source-prefix 10.1.1.10`, flow traceoptions
VMware NSX	Traceflow injects a marked packet and reports every hop	Port mirroring and `pktcap-uw` on the ESXi host

Packet capture

When I need proof, I capture the packets

Monitoring shows where and when. A packet capture shows why. The skill is capturing at the right point, usually both sides of the device I suspect, and filtering on the one flow that matters.

Swipe sideways to see the whole diagram.

SPAN (Switched Port Analyzer). The switch copies traffic from a source port or VLAN to a destination port with an analyser. monitor session 1 source interface Gi1/0/1 both then monitor session 1 destination interface Gi1/0/24. Copies can be dropped if the destination port is busier than it can send, so a capture with gaps is not proof of loss.

Remote SPAN. Copies cross trunks in a dedicated RSPAN (remote SPAN) VLAN to an analyser on another switch. Useful in a campus where the analyser cannot sit next to every switch. The RSPAN VLAN must be allowed on every trunk between them.

ERSPAN (Encapsulated Remote SPAN). Copies are wrapped in GRE (Generic Routing Encapsulation) and routed to an analyser anywhere: another site, or a capture appliance in the data centre. It adds headers, so I check the MTU on the path.

On the device. Cisco IOS XE Embedded Packet Capture stores packets in a buffer: monitor capture CAP interface Gi1/0/1 both, monitor capture CAP match ipv4 host 10.1.1.10 any, monitor capture CAP start, then show monitor capture CAP buffer brief. Firewalls have their own captures. On a Nexus, Ethanalyzer sees only traffic to and from the supervisor, such as routing hellos, not traffic switched in hardware.

Animation

Wireshark filters I use most

Display filter	What it shows me
`ip.addr == 10.1.1.10 && tcp.port == 443`	One host and one service, both directions.
`tcp.flags.syn == 1 && tcp.flags.ack == 0`	Connection attempts. Many with no reply point at a drop or a dead server.
`tcp.analysis.retransmission`	Lost or delayed segments: loss on the path, or an MTU black hole.
`tcp.flags.reset == 1`	Resets: a closed port, a firewall or a load balancer ending sessions.
`tcp.analysis.zero_window`	The receiver is full. The application or server is slow, not the network.
`icmp.type == 3`	Unreachable messages. Code 4 is "fragmentation needed".
`dns`, `dhcp`, `arp`	The basic services. Slow or missing answers here break everything above.
`tls.handshake.type == 1`	TLS client hellos, with the server name each client asked for.

Capture filters, written in BPF (Berkeley Packet Filter) form such as host 10.1.1.10 and port 443, decide what is recorded. Display filters decide what is shown. On a busy link I always use a capture filter, and I capture at both ends with synchronised clocks so I can compare the two traces.

Three Wireshark features save the most time. Follow TCP Stream shows one conversation as the application saw it. Expert Information lists retransmissions, resets and zero windows in one place. IO (input and output) graphs show when the trouble started. On a server with no screen, tshark does the same from the command line, and Wireshark decodes VXLAN (Virtual Extensible LAN) and Geneve, so overlay traffic is readable too.

People and pressure

The fault is technical. The incident is about people.

Half of fixing a serious incident is not technical. People need to know what is happening, the right experts need to be on the call, and decisions need to be made calmly with the evidence to hand.

Swipe sideways to see the whole diagram.

Step 1

The service desk logs one ticket, links duplicates and captures the symptom and the exact error. A known error record or a knowledge article may restore service here.
If second line cannot fix it, they escalate by skill. A good handover says what is affected, since when, what was tried, what was ruled out and why, and attaches the evidence.
I accept it, update the ticket and the user, and run the method. I set a time box: on a priority one with no clear idea after 30 to 60 minutes, I bring in more help rather than carry on alone.
If I suspect a hardware fault or a software bug, I open a vendor case at a severity that matches the real impact and send the evidence pack. For the most severe Cisco cases I also phone, because a call gets an engineer faster.
When the impact grows or the SLA is at risk, I tell the incident manager early, not at the deadline. They can call a major incident, bring in duty managers and make decisions that need authority. Escalating late is the real failure.

The evidence pack I send

A one line problem statement and the business impact
Model, serial number and software version (show version, show inventory)
A timeline with times and the time zone
What changed before it started
What I tried, and what I ruled out with evidence
Logs, captures and command output from the time of the fault
The vendor's support bundle, such as show tech-support
How to reproduce it, if I can

How I keep people informed

One bridge, one voice. The incident manager runs the call; the technical lead runs the diagnosis. Nobody changes a device off the bridge.
Parallel lines of enquiry. One person per area, such as links, routing and the firewall, with one pair of hands on each device.
Updates on a fixed rhythm. What we know, what we are doing, what users should do, and the time of the next update, in plain English.
Calm decisions. Every change is announced with its rollback and agreed before it is made. If a senior person suggests a risky fix, it goes through the incident manager and change authority like any other.
A clean handover. On a long incident I plan a handover with the timeline, so a tired engineer does not make the call that matters most.

Fault library

206 faults, with the checks that find them

These are faults that turn up in campus, data centre, WAN (wide area network), firewall, NSX, wireless, SD-WAN, OT (operational technology) and cloud networks, some I have seen and some I have rehearsed. Each one has the symptom, the likely causes, the checks in order with the commands, the fix, and how to stop it coming back. Pick an area or search, then open a fault and step through the checks one at a time.

FundamentalsThe seven layer walk for "the device doesn't work"

What I check, in order

Power (PoE (Power over Ethernet) budget, brown out)
Link and radio (speed and duplex, RSSI (received signal strength indicator), association)
Identity (802.1X or MAB (MAC Authentication Bypass), ISE (Identity Services Engine) result, VLAN and SGT (Security Group Tag) returned)
IP (DHCP (Dynamic Host Configuration Protocol), device tracking, LISP (Locator/ID Separation Protocol) registration, gateway)
Path (SGACL (security group ACL) counters, border, fusion firewall, MTU)
Transport and TLS (time, certificate, cipher, SNI (Server Name Indication))
Protocol and application (MQTT (Message Queuing Telemetry Transport) return code, topic ACL (Access Control List), keep alive, payload)

Worth knowing

Device side principle. Isolate the layer first: power, radio, IP, TLS, protocol, policy, application.

More on this: Networking fundamentals, SD-Access campus fabric, Security

FundamentalsDebugging a misbehaving device or gateway (method)

Worth knowing

Capture at the gateway or a SPAN port. Wireshark dissectors for MQTT, CoAP, Modbus, BACnet, OPC UA (Open Platform Communications Unified Architecture). CONNECT/CONNACK return codes, keep alive gaps, TLS alerts, retransmissions, MTU. USB (Universal Serial Bus) 802.15.4 sniffer with the network key for Thread/Zigbee.
Broker side. Logs and $SYS topics. mosquitto_sub -t '#' -v or MQTT Explorer. Duplicate client ID loops.
Hardware. Multimeter for supply under load (brown outs cause random reboots), logic analyser for I2C (Inter Integrated Circuit)/SPI (stuck lines, wrong address), oscilloscope for noisy analogue, JTAG (Joint Test Action Group)/SWD with OpenOCD.
Network. The fabric checks above. Cloud: connect failure and rule error metrics, certificate status, a policy deny that looks like a silent drop on the device.
Serial console first (boot logs show radio association, DHCP, time sync, TLS; most "network" faults show as a TLS error here).

More on this: Networking fundamentals, IoT (Internet of Things) and edge

FundamentalsA user cannot reach a server

Symptom. A host cannot connect to a server or application. One user or many, one app or all, recent change?

Likely causes

Link down or erroring.
Wrong VLAN or MAC not learned.
Missing route or wrong default gateway.
ARP failure.
Firewall not permitting the port or no session.
DNS failure.
Service not listening.
MTU or MSS problem.
Certificate problem.

What I check, in order

Scope: one user or many, one application or all, any recent change.
Layers 1 and 2: show interface status, show interface for errors, CRC and drops, the access VLAN, and show mac address-table to see the MAC learned.
Layer 3: ping and traceroute from the right source, show ip route, the ARP table and the host default gateway.
Layer 4 and the firewall: is the port permitted, firewall logs and deny counters, show conn and packet-tracer on an ASA, diagnose debug flow on a FortiGate, test security-policy-match on a Palo Alto.
The application: DNS, the server listening on the port, MTU and MSS, certificate validity.
Capture with Wireshark if it is still unclear: retransmissions, RTT (round trip time), window size, resets.

Fix

Correct the layer that failed (cable or optic, VLAN, route, rule, DNS record, service).

Stop it coming back

Standard configs, monitoring on each layer, and turning a recurring cause into a design standard.

More on this: Networking fundamentals

FundamentalsIntermittent packet loss on a link

Symptom. Loss comes and goes on one path.

Likely causes

Physical faults (bad cable or optic), duplex or speed mismatch, microbursts filling buffers, QoS (Quality of Service) drops, congestion upstream.

What I check, in order

show interface for input errors, CRC, late collisions and drops.
Duplex and speed on both ends.
Optic light levels through DOM (digital optical monitoring).
Buffer and QoS drop counters.
The upstream device.
Correlate with time and load. Rising CRC points to the physical layer.
Output drops point to congestion.

Fix

Replace the cable or optic, correct duplex and speed, adjust QoS or add capacity.

Stop it coming back

Baseline interface counters and DOM levels in monitoring. Alert on CRC growth.

More on this: Networking fundamentals

Fundamentals"The application is slow": is it the network?

Symptom. Users report slowness.

Likely causes

Path latency, jitter or loss.
Congested interfaces.
QoS misclassification.
Server, application or database slowness.

What I check, in order

Quantify latency, jitter, loss and throughput against a baseline.
traceroute and per hop latency.
Interface utilisation and drops.
QoS counters.
A packet capture showing retransmissions, zero windows (a slow server), duplicate ACKs (acknowledge) and RTT.

Worth knowing

Rule. Rule out application, database and server before blaming the network, and prove it with data.

Fix

Depends on where the evidence points. Rule out the application, database and server before changing the network.

Stop it coming back

Baselines and application performance monitoring so "slow" can be measured, not argued.

More on this: Networking fundamentals

FundamentalsMTU black hole: small things work, large transfers hang

Symptom. Ping, TCP handshake and SSH login work. show running-config over SSH, file copies and some web pages hang.

Likely causes

Tunnel or overlay overhead (GRE 24, IPsec 50 to 70+, VXLAN 50, Geneve 50+)
ICMP type 3 code 4 or ICMPv6 (Internet Control Message Protocol version 6) Packet Too Big blocked.
Jumbo mismatch.
OSPF (Open Shortest Path First) stuck in ExStart on the same link.

What I check, in order

ping <dest> size 1500 df-bit and step down.
Windows ping -f -l 1472
Linux ping -M do -s 1472
Largest passing size + 28 = path MTU.
Check firewall ICMP policy.
show interfaces for MTU each hop.
Capture for retransmitted full size segments.
Check MTU on tunnel interfaces.

Fix

Correct MTU end to end. Allow ICMP 3/4. On tunnels ip mtu 1400 and ip tcp adjust-mss 1360.

Stop it coming back

MTU in the design standard (NSX underlay 1600 minimum, 1700 recommended, 9000 typical in data centres), DF ping in acceptance tests.

More on this: Networking fundamentals

FundamentalsLink works but is slow: duplex mismatch

Symptom. Link up, slow, worse on big transfers.

Likely causes

One side hard set, other auto (falls to half duplex at 10/100)

What I check, in order

show interfaces Gi1/0/5 | include duplex|CRC|collision|runts on both ends.
show interfaces status (full on the hard set end, a-half on the auto end)
%CDP-4-DUPLEX_MISMATCH

Fix

Speed auto and duplex auto on both ends, or hard set both.

Stop it coming back

Standard: autonegotiate on both ends. Gigabit copper requires autoneg.

More on this: Networking fundamentals

FundamentalsOutput drops while average utilisation is only 30%

Symptom. Output drops climbing, utilisation graphs look fine.

Likely causes

Microbursts hidden by 5 minute averages.
10G feeding 1G.
Small buffers on access switches such as the 3750.
QoS queue limits.

What I check, in order

show interfaces | include drops|rate
show policy-map interface for per class drops.
Shorter load interval.

Fix

QoS queue and buffer tuning, more uplink capacity, move bursty sources.

Stop it coming back

Baseline with short intervals, size uplinks for bursts.

More on this: Networking fundamentals

FundamentalsNew fibre link does not come up

Symptom. Link down on a new run.

Likely causes

Optics not the same type.
Multimode patch on single mode optic or the reverse.
Polarity (Tx to Tx)
Dirty connectors.
Fixed speed optic in a faster port.

What I check, in order

show interfaces transceiver detail / show inventory both ends.
Inspect and clean.
Swap polarity.
Read Rx power against DOM thresholds.
NX-OS show interface ethernet1/49 transceiver details

Fix

Matching optics and fibre, correct polarity, clean, set speed 1000 for a 1G optic in a 10G port.

Stop it coming back

One fibre type and polarity method per estate, approved optics list, loss test every permanent link.

More on this: Networking fundamentals, Data centre networks

FundamentalsIntermittent slowness with a rising topology change count

Symptom. Count of topology changes rising every minute, pointing at an uplink.

What I check, in order

show spanning-tree detail | include ieee|occurr|from|is exec on each switch in turn, following the "from" port via show cdp neighbors <port> detail
show logging | include SPANTREE|LINK-3-UPDOWN

Fix

PortFast on the access port that is flapping, or replace the cable or optic on the flapping link.

Stop it coming back

PortFast and BPDU (Bridge Protocol Data Unit) Guard defaults on access ports.

More on this: Networking fundamentals

FundamentalsDuplicate IP address conflict

Likely causes

Static address inside a DHCP scope, cloned VM (virtual machine), device restored from an old image.

What I check, in order

show logging | include DUPADDR
show ip arp <ip> (flipping MACs)
show mac address-table address <mac>
show cdp neighbors <port> detail hop by hop
show ip dhcp snooping binding | include <ip>
Wireshark arp.duplicate-address-detected
IPAM (IP Address Management), DHCP leases and CMDB (Configuration Management Database) for the legitimate owner.

Fix

Through the device owner (server team change if two servers). Unknown device means tell security before doing anything disruptive.

Stop it coming back

Exclude static ranges from DHCP. DHCP snooping, DAI (ip arp inspection vlan 140) and IP Source Guard (ip verify source)

More on this: Networking fundamentals

FundamentalsGrey failure: a link drops some packets but stays up

Symptom. Intermittent slowness or retransmits. Interface up.

Likely causes

Dirty or failing optic, cable or connector.
FEC (forward error correction) errors climbing.
One way fault behind optical transport.

What I check, in order

Interface error and CRC counters.
Corrected and uncorrectable FEC trends.
Telemetry for micro bursts.
BFD (Bidirectional Forwarding Detection) state.

Fix

Drain the link or switch with maintenance mode rather than waiting for hard failure. Clean and inspect connectors (IEC 61300-3-35). Replace optic or cable.

Stop it coming back

Inspect and clean every connector before mating. Loss budget design. Burn in thresholds (zero uncorrectable FEC, stable corrected FEC, zero CRC over 24 h)

More on this: Networking fundamentals

Layer 2Native VLAN mismatch and VLAN hopping

Symptom. Traffic leaking between VLANs on a trunk, or CDP (Cisco Discovery Protocol) native VLAN mismatch messages.

Likely causes

Different native VLANs on each end of a trunk.
Untagged native VLAN exploited for double tagging.
Auto trunking.

What I check, in order

Native VLAN on both ends.
Allowed VLAN lists.
Trunk negotiation state.

Fix

Match the native VLAN on both ends and use an unused one. Prune the allowed list. Turn trunk negotiation off. Shut unused ports.

Stop it coming back

Trunk template standard.

More on this: Switching and campus LAN (local area network)

Layer 2Whole VLAN gets 169.254 addresses

Symptom. Every PC (personal computer) in one VLAN self assigns 169.254.x.x.

Likely causes

SVI (switched virtual interface) down or helper address removed.
Relay cannot reach the DHCP server.
Firewall between relay and server blocks UDP 67/68.
Scope missing or exhausted.
DHCP snooping uplink not trusted or access ports err-disabled by the rate limit.
Option 82 with giaddr 0.0.0.0 dropped upstream.
A recent change.

What I check, in order

show interfaces Vlan20
show ip interface Vlan20 | include Helper|line protocol
ping 10.0.0.5 source Vlan20
Firewall logs for UDP 67/68 from the relay address.
Scope and free leases on the server.
show ip dhcp snooping
show ip dhcp snooping statistics
show interfaces status err-disabled
show logging | include DHCP
Capture at the server to see if the Discover arrives.

Worth knowing

Single user variant. Wrong access VLAN (show interfaces Gi1/0/10 switchport), port security, cable, NIC (network interface card), static address. If Discover arrives and no Offer, it is the server or scope.

Fix

Restore helper or SVI, open UDP 67/68, extend or create the scope, trust the uplink, fix option 82 (no ip dhcp snooping information option or ip dhcp relay information trust-all), recover err-disabled ports.

Stop it coming back

Change templates that include helpers, monitoring of scope utilisation, persisted snooping database, record cause in the ticket.

More on this: Switching and campus LAN, Networking fundamentals

Layer 2Unicast flooding and slowness in one VLAN (asymmetric routing)

Symptom. Users slow. High output utilisation on many access ports in one VLAN. A capture on an idle port shows unicast for another host.

Likely causes

HSRP (Hot Standby Router Protocol) active on switch A, return traffic routed via switch B.
B's ARP entry (4 hours) outlives its MAC entry (300 s), so B floods.
ARP timer longer than MAC ageing so the switch keeps sending to a MAC it has forgotten.

What I check, in order

show ip arp <host ip> on B (present)
show mac address-table address <mac> on B (absent)
show standby brief
show spanning-tree root
Routing towards the subnet on both switches.
ARP and MAC ageing timers (NX-OS defaults ARP 1500 s, MAC 1800 s)

Fix

Align HSRP active, STP (Spanning Tree Protocol) root and routing per VLAN. Or arp timeout 270 on the SVI. Or raise MAC aging.

Stop it coming back

Design standard that ties gateway, root and routing together. On vPC (virtual Port Channel) use ip arp synchronize. Platform defaults kept in the standard.

More on this: Switching and campus LAN, Data centre networks, Networking fundamentals

Layer 2Adding one VLAN to an uplink takes the site down

Symptom. After a trunk change the whole site, and maybe management, drops.

Likely causes

switchport trunk allowed vlan 40 without add replaced the list

What I check, in order

Console or out of band.
show interfaces trunk
Compare with pre change output.

Fix

switchport trunk allowed vlan add <list> to restore, or reload to the saved config if unsaved.

Stop it coming back

Always use add or remove. Reload in or configure terminal revert timer before remote uplink changes. Peer check. On vPC also add the VLAN to the peer link and both peers. Record as major incident plus problem record (human error in change, template fix)

More on this: Switching and campus LAN, Data centre networks

Layer 2MAC flapping messages

Symptom. %SW_MATM-4-MACFLAP_NOTIF in the log.

Likely causes

One MAC on two server ports means NIC teaming without matching channel.
VM MAC moves after vMotion (normal if occasional)
Many MACs including the gateway on two uplinks means layer 2 loop.
Gateway MAC flapping means HSRP dual active or a vPC fault.
A dual homed device without a proper port channel or vPC.
Miscabling.
A broken vPC.

What I check, in order

show logging | include MACFLAP
show mac address-table address <mac>
show interfaces <port> | include rate|broadcast
show spanning-tree detail | include ieee|occurr|from|is exec
CDP to follow ports.
vSwitch teaming policy.
show mac address-table and show logging to find the ports.
show spanning-tree for topology changes.
Look for missing BPDU Guard and miscabled links.
show vpc and show port-channel summary on Nexus.
show mac address-table, logs, show spanning-tree

Worth knowing

Meaning. The same MAC learned on two ports rapidly: usually an L2 (Layer 2) loop or a dual homed device without a proper port channel or vPC.

Fix

Align teaming with the switch (IP hash means static channel; LACP (Link Aggregation Control Protocol) means LAG (link aggregation group) on a VDS (vSphere Distributed Switch); originating port means no channel). For a loop, follow the broadcast storm fault in this area.

Stop it coming back

Server connection standard agreed with the virtualisation team. BPDU Guard and PortFast on edge ports, Root Guard and Loop Guard where they belong, vPC consistency checks.

More on this: Switching and campus LAN, Data centre networks

Layer 2Broadcast storm or layer 2 loop

Symptom. Site very slow, high CPU on every switch in the VLAN, links near 100% broadcast and multicast, MAC flaps, HSRP and routing neighbours flapping, management sessions dropping.

Likely causes

Interface BPDU Filter.
Unmanaged switch or hub bridging two ports.
VM bridging two NICs.
One way fibre.
STP disabled on a VLAN (or over the 3750 128 instance limit)
PortFast on an inter switch link.

What I check, in order

Console or OOB (out of band) first.
show processes cpu sorted | exclude 0.00
show interfaces | include line protocol|input rate|broadcasts
show logging | include MACFLAP|SPANTREE|HSRP|ADJCHG
show spanning-tree detail | include ieee|occurr|from|is exec
show spanning-tree root
show spanning-tree inconsistentports
show running-config | include bpdufilter|no spanning-tree
show interfaces status err-disabled

Fix

Contain first: shut the redundant link or access port the evidence points to, starting at the edge, log the time. If no access at all, shut downlinks one at a time from the core console while watching CPU and broadcast rate.

Stop it coming back

BPDU Guard on every access port, Loop Guard or UDLD (Unidirectional Link Detection) on uplinks, Root Guard on downlinks, storm control at the edge, no interface BPDU Filter, deliberate root placement. Major incident timeline then a problem record.

More on this: Switching and campus LAN, AI (artificial intelligence) infrastructure

Layer 2Port channel member suspended or stand alone

Symptom. One member flagged s or I in show etherchannel summary.

Likely causes

Stand alone (I): no LACP partner (far end not configured or cabled elsewhere)
Suspended (s): member config differs (VLAN mask, speed, mode)
Different partner system ID means cable to another switch.
err-disabled channel-misconfig means mode on against a plain port.

What I check, in order

show etherchannel summary
show lacp neighbor (same partner ID on all members)
show interfaces Gi1/0/50 switchport
show running-config interface
show logging | include EC-5|ERR_DISABLE (%EC-5-CANNOT_BUNDLE2... Vlan mask is different)

Fix

Default interface, re add channel-group 1 mode active, match far end. shutdown / no shutdown after misconfig guard.

Stop it coming back

Configure the port channel interface and let members inherit. LACP active both ends.

More on this: Switching and campus LAN

Layer 2A user's port keeps going err-disabled

Symptom. Port err-disabled, reason bpduguard (or psecure-violation).

Likely causes

Small unmanaged switch, docking station or VM bridging NICs sending BPDUs.
A desk move with sticky MACs.

What I check, in order

show interfaces status err-disabled
show logging (%SPANTREE-2-BLOCK_BPDUGUARD, %PM-4-ERR_DISABLE)
Go and look at what is plugged in.
show port-security interface

Fix

Remove the device, then shutdown / no shutdown. clear port-security sticky interface <old port> for desk moves.

Stop it coming back

Do not auto recover BPDU Guard. Desk move process clears sticky entries.

More on this: Switching and campus LAN, AI infrastructure

Layer 2VLANs vanish across a site (VTP)

Symptom. VLANs disappear, access ports go inactive.

Likely causes

A switch with the same VTP (VLAN Trunking Protocol) domain and a higher revision joined a trunk.

What I check, in order

show vtp status on several switches (revision jump, last modified by)

Fix

Restore VLANs on the real server. Set the rogue switch to transparent.

Stop it coming back

VTP transparent or off (or VTPv3 with a primary server). Reset revision before connecting any switch.

More on this: Switching and campus LAN

Layer 2Two VLANs merge or are blocked on a trunk (native VLAN mismatch)

Symptom. Odd reachability, CDP NATIVE_VLAN_MISMATCH, VLANs blocked as port VLAN ID inconsistent.

What I check, in order

show interfaces trunk both ends.
show spanning-tree inconsistentports
show logging | include NATIVE_VLAN_MISMATCH

Fix

Same unused native VLAN both ends.

Stop it coming back

Trunk template with native VLAN 999 and vlan dot1q tag native.

More on this: Switching and campus LAN

Layer 2Half a floor cannot get out after a switch reload (DAI)

Symptom. After a reload many hosts lose connectivity until their lease renews.

Likely causes

DHCP snooping bindings not persisted, so DAI drops ARP from valid hosts.

What I check, in order

show ip arp inspection statistics vlan <n> for drops.
show ip dhcp snooping binding count

Fix

ip dhcp snooping database flash:dhcp-snoop.db. Renew leases. Static hosts get ARP ACLs.

Stop it coming back

Persist bindings. Rollout order snooping first, then DAI and IPSG (IP Source Guard).

More on this: Switching and campus LAN, Networking fundamentals

Layer 2A user loses connectivity after DAI is enabled

What I check, in order

show ip arp inspection statistics vlan 20 and logs for denied ARP from the host.
show ip dhcp snooping binding for that host.
show interfaces status err-disabled (rate limit)

Fix

Static host gets an ARP ACL or static binding through change. Recover err-disabled port.

Stop it coming back

Rollout order snooping, then DAI, then IPSG. Persist bindings.

More on this: Switching and campus LAN, Networking fundamentals

Layer 2Users on one floor cannot reach anything

Likely causes

Access stack down or reloaded, member down, uplink err-disabled (UDLD, EtherChannel misconfig). STP blocking the VLAN or root or loop inconsistent after something was plugged in, DHCP snooping uplink no longer trusted after a change.

What I check, in order

Monitoring for the stack and uplinks.
Narrow by symptom (no IP means DHCP, IP but no gateway means VLAN, trunk or HSRP, gateway but nothing beyond means routing or firewall)
From distribution show etherchannel summary, show interfaces Po10, show spanning-tree interface Po10 detail, show cdp neighbors detail
On the stack via console or OOB show switch, show version | include uptime|reason, show logging, show interfaces status err-disabled. show errdisable recovery, show spanning-tree summary, show spanning-tree inconsistentports, show mac address-table dynamic vlan 120, show ip dhcp snooping

Fix

Emergency change. UDLD err-disabled uplink: check and clean or replace fibre and optics before bringing it back, never just bounce it.

Stop it coming back

Write it up. err-disable reasons tracked.

More on this: Switching and campus LAN, Networking fundamentals

Layer 2DHCP failing for one VLAN that uses a relay

Likely causes

Helper address missing or wrong after a rebuild.
Scope exhausted or not matching the SVI subnet (giaddr selects the scope)
Snooping dropping offers on an untrusted uplink or option 82 rejected.
Firewall not allowing UDP 67 from the new VLAN's subnet.
No return route from the server to the SVI.

What I check, in order

show running-config interface Vlan140
show ip dhcp snooping, statistics, binding vlan 140.
show ip route <server>
ping <server> source Vlan140
NX-OS show ip dhcp relay
Server logs and scope usage.
Capture.

Worth knowing

Note. APIPA (Automatic Private IP Addressing) 169.254 means no offer at all, so focus on the relay path and snooping.

Fix

ip helper-address 10.50.1.10 (NX-OS ip dhcp relay address), ip dhcp snooping trust on the uplink, scope or firewall fix.

More on this: Switching and campus LAN, Data centre networks, Networking fundamentals

Layer 2MAC address flapping on a switch

Symptom. Logs show the same MAC learned on two ports in quick succession.

Likely causes

A layer 2 loop.
A dual homed server or device without a proper port channel or vPC.
Miscabling.
Missing BPDU Guard.

What I check, in order

show mac address-table for the MAC.
show logging
show spanning-tree for unexpected topology changes.
Check port channel or vPC config on the dual homed device.

Fix

Break the loop or correct the bundling.

Stop it coming back

BPDU Guard on access ports, Loop Guard and UDLD on uplinks, standard dual homing configs.

More on this: Switching and campus LAN, Data centre networks

Layer 2A 4 x 10G port channel only moves 10G for one transfer

Symptom. A single large transfer never exceeds one member's speed.

Likely causes

Per flow hashing by design.

What I check, in order

show etherchannel summary
Check the load balance hash.

Fix

Use a hash with more entropy (src-dst-ip or L4 ports) for many flows. Accept that one flow uses one link, or move to faster members.

Stop it coming back

Set expectations in the design.

More on this: Switching and campus LAN, Load balancing

Layer 2Loop or black hole after building a port channel

Symptom. Loop or lost traffic when a bundle is brought up.

Likely causes

One side in mode on, the other not bundling.
Member mismatch in speed, duplex, VLANs, trunk mode or native VLAN.

What I check, in order

show etherchannel summary (P bundled, s suspended, D down)
Compare member configs.

Fix

Use LACP active on at least one side, align member settings.

Stop it coming back

LACP everywhere, min-links where needed.

More on this: Switching and campus LAN, Networking fundamentals

Layer 2MST region splits after a VLAN change

Symptom. Traffic suddenly follows the CIST (Common and Internal Spanning Tree), unexpected blocking or tromboning.

Likely causes

A VLAN added to the instance map on one switch but not the others, so name, revision or mapping no longer match.

What I check, in order

show spanning-tree mst configuration on each switch and compare digests.
show spanning-tree mst per instance

Fix

Make the region config identical everywhere.

Stop it coming back

Push MST (Multiple Spanning Tree) config from a template, change it as one change.

More on this: Switching and campus LAN

Layer 2MAC flapping or false duplicate detection

Symptom. A MAC is frozen and not advertised. Duplicate address alarms.

Likely causes

Edge loop.
Misconfigured active active NIC team.
Cloned VM with the same MAC.
MAC moving 5 times in 180 s.

What I check, in order

MAC mobility sequence numbers in type 2 routes.
Host mobility manager.
Edge STP and BPDU guard events.

Fix

Break the loop, fix teaming, re MAC the clone, clear the duplicate.

Stop it coming back

BPDU guard on host ports, storm control, consistent teaming standards, duplicate detection settings reviewed.

More on this: Switching and campus LAN

Layer 2Broadcast storm in one data centre threatens the other

Symptom. BUM (broadcast, unknown unicast and multicast) traffic climbing on DCI (Data Centre Interconnect) links.

Likely causes

Loop or storm in a stretched segment.

What I check, in order

BUM rates at BGWs (border gateway).
Which VNIs (VXLAN Network Identifier) are extended.

Fix

Storm control at BGWs. Break the loop.

Stop it coming back

Extend only named VNIs. ARP suppression. No STP across DCI. L2 extension only as time bound exception.

More on this: Switching and campus LAN, Networking fundamentals

Campus platformsAn Extended Node shows in Catalyst Centre but its ports never get the right VLAN

What I check, in order

Edge uplink in the INFRA_VN pool and set for extended node onboarding.
Node in the correct site and finished provisioning (PnP status, image)
Port assignments applied to the Extended Node's ports, not the parent's.
Parent sees it (show cdp neighbors, port channel state if dual homed)

Worth knowing

If policy is not enforced. It is a plain Extended Node, enforcement is on the parent edge, so check SGACL counters there.
Rings. REP (Resilient Ethernet Protocol) segment status (show rep topology) and the edge's REP configuration.

More on this: Switching and campus LAN

Campus platformsA new or replacement stack member will not join

Symptom. Member shows Version Mismatch, Provisioned, Waiting or Progressing.

Likely causes

IOS not compatible with the master.
Auto upgrade not possible (flash too small, different feature set)
Stack cables not seated.
Wrong member number.

What I check, in order

show switch
show switch stack-ports
show boot (auto upgrade)
show logging | include STACKMGR

Fix

Let auto upgrade copy the master image, or archive download-sw /overwrite /reload tftp://.../<image>.tar for the whole stack. Reseat cables. Renumber the switch while standalone.

Stop it coming back

Pre stage spares on the same IOS with the right number and a low priority. Hot insert only under change.

More on this: Switching and campus LAN

Campus platformsStack running as a half ring, or split

Symptom. Reduced cross stack performance. Or duplicate address messages and odd routing after two failures.

What I check, in order

show switch stack-ports
show switch stack-ring speed (Configuration: Half)
show switch neighbors
show logging | include STACKMGR|DUPADDR

Fix

Reseat at both ends, known good cable, test the port on another member. Expect the losing half to reload on merge.

Stop it coming back

Treat one down stack port as urgent. Log in the PPM (planned preventive maintenance) record.

More on this: Switching and campus LAN

Campus platformsVSS dual active

Symptom. VSL (virtual switch link) lost, both chassis active with same IPs and router IDs.

What I check, in order

show switch virtual dual-active summary
show switch virtual link
Log.

Fix

Detection puts the old active in recovery mode (all ports down except VSL and excluded). Repair VSL (optics, cables, module). Recovery chassis reloads and rejoins as standby. Check MECs (multichassis EtherChannel) rejoin on both chassis.

Stop it coming back

Two detection methods (enhanced PAgP (Port Aggregation Protocol) and fast hello), VSL on at least two links on different modules, exclude the OOB port.

More on this: Switching and campus LAN

Campus platformsVSS standby in RPR, not SSO

Symptom. show redundancy shows RPR (route processor redundancy) or the peer not in STANDBY HOT.

Likely causes

Software or configuration mismatch between supervisors.

Fix

Find the mismatch in the log, align versions or config in a window.

Stop it coming back

Include show redundancy in the PPM check.

More on this: Switching and campus LAN

Campus platformsHigh CPU on a core switch or 3850 stack

Symptom. High CPU, slow management, protocol flaps.

Likely causes

Punted traffic (loop ARP floods, TTL expiry, IP options, ACL logging, unresolved next hops)
A busy process (ARP Input, IP Input, routing churn, SNMP ENGINE, SSH brute force)

What I check, in order

show processes cpu sorted (total versus interrupt)
show processes cpu history
show interfaces | include line protocol|input rate|broadcasts
6800: show processes cpu sorted 5sec, debug netdr capture rx, show netdr captured-packets, undebug all
3850: show processes cpu platform sorted, show platform software fed switch active punt cpuq all
show policy-map control-plane

Fix

Stop the source (break the loop, fix the next hop, tune the SNMP poller, ACL on SNMP or SSH).

Stop it coming back

CoPP (control plane policing), SNMP views and ACLs, monitoring that walks big tables less often.

More on this: Switching and campus LAN, AI infrastructure, Networking fundamentals

Campus platformsStack member failed at 2am

Symptom. One member down, possibly the active (standby took over with SSO (stateful switchover)).

What I check, in order

show switch
show switch stack-ports summary
show logging | include STACKMGR|SWITCH_REMOVED
What was on the member (users, phones, APs (access point), uplinks)

Fix

Decide replace now (emergency change) or stabilise. Spare of same model and licence on the same IOS XE in install mode. On the spare: switch 1 priority 1, switch 1 renumber 3, reload. Power off failed unit, note cables, fit spare, reconnect stack and StackPower cables, power on. Inherits provisioned config. Verify show switch (Ready, not V-Mismatch), show interfaces status, show power inline, show etherchannel summary, show version.

Stop it coming back

Incident timeline, emergency change with evidence, RMA (return material authorisation), CMDB serial update, config backup. Never add a spare with higher priority or an old config. No 3850 and 9300 mixed stacks, so fallback is a spare from another site, moving users, or a temporary standalone switch recorded as a known risk.

More on this: Switching and campus LAN

Campus platformsRoutes flapping and high CPU on a core switch

Likely causes

Unstable link causing churn.
Redistribution loop.
Punted traffic (TTL expiry from a routing loop, ARP floods, unresolved adjacencies, ACL logging)

What I check, in order

show processes cpu sorted 5sec (process versus interrupt: "85%/60%")
show processes cpu history
show logging | include ADJCHG|BGP-5
show ip ospf neighbor (uptimes)
show ip ospf statistics (SPF (shortest path first) runs)
show ip route summary
show ip traffic
show policy-map control-plane
Punt capture on Sup2T and Sup6T: debug netdr capture rx, show netdr captured-packets, undebug all

Fix

Shut the flapping link under emergency change. CoPP protecting the CPU. Long term route tags and filters at redistribution points and the link repair. Dampening only as a recorded mitigation.

More on this: Switching and campus LAN, AI infrastructure, Networking fundamentals

Campus platformsA 3850 stack member failed and the model is past last date of support

What I check, in order

Everything in the stack member fault in this area, then show switch detail, show environment all, show power inline, dir crashinfo: (crash evidence for vendor or problem record)
Spare from site stock because no RMA after LDoS (last date of support).
switch 3 provision ws-c3850-48p before insertion.
Flag on the obsolescence register.
Failed unit's flash may need to stay on site or be destroyed.
All uplinks on the failed member means design finding (spread uplinks across members)

More on this: Switching and campus LAN

Nexus and vPCvPC pair inconsistent after automation changed one member only

Symptom. VLANs suspended on vPC member ports.

Likely causes

Job inventory limit or a failed task on the second switch left the pair asymmetric.
Type 1 consistency mismatch.

What I check, in order

show vpc
show vpc consistency-parameters interface port-channel N
Job log for the second host.
Controller compliance for both peers.

Fix

Complete the change on the peer (idempotent rerun) or roll both back.

Stop it coming back

Target vPC pairs together (inventory groups per pair, serial: 0 for pair operations, or controller vPC policies that apply to both). "pair consistent" post check.

More on this: Data centre networks, Network automation

Nexus and vPCvPC failure scenarios

What I check, in order

show vpc, show port-channel summary
Consistency check status.

Worth knowing

Keepalive down, peer link up. No traffic impact. Restore the keepalive.
Peer link down, keepalive up. The secondary suspends its vPC member ports to avoid loops. Orphan ports on the secondary can be stranded.
Both down. Dual active split brain.

Fix

Restore the peer link. Resolve Type 1 consistency errors.

Stop it coming back

Keepalive never over the peer link. Dual home rather than orphan devices. Peer gateway where needed.

More on this: Data centre networks

Nexus and vPCvPC peer link fails, keepalive still up

Symptom. Half the port channel members on every dual homed device go down. Orphan devices on the secondary unreachable.

Likely causes

Peer link fault.
Peer link on a single ASIC (application specific integrated circuit) or line card.

What I check, in order

show vpc (peer link is down, peer is alive)
show vpc peer-keepalive
show port-channel summary
show interface port-channel 1
show logging logfile | include VPC
Keepalive status (separate path over OOB)
vPC role.
Which ports are suspended on the secondary.
Consistency check status.

Fix

Restore peer link members (optics, cables, line card). Secondary restores vPCs after consistency and delay restore.

Stop it coming back

Peer link members on different line cards and optics, monitored. Minimise orphans. orphan-port suspend. Dual active exclude interface-vlan for SVIs that must stay up. Keepalive never over the peer link. Consider ESI (Ethernet segment identifier) multihoming or vPC fabric peering.

More on this: Data centre networks

Nexus and vPCOnly the vPC keepalive fails

Symptom. No traffic impact. show vpc peer-keepalive shows Suspended (Destination IP not reachable).

What I check, in order

ping <peer> vrf management source <local>
show interface mgmt0
show running-config vpc | include keepalive
Management network, VRF (Virtual Routing and Forwarding), addresses, ACLs, OOB switch.

Fix

Repair the keepalive path the same day.

Stop it coming back

Monitor keepalive state. Raise as an incident with a short target because the next failure becomes dual active. Duty manager decides out of hours.

More on this: Data centre networks

Nexus and vPCvPC dual active (peer link and keepalive both down)

Symptom. Intermittent failures. Both peers show role primary, both HSRP active, MAC flaps, duplicate address warnings, two STP roots.

Fix

Restore the keepalive first, then the peer link. Expect a short disruption while one side re syncs. Problem record on why both failed together.

Stop it coming back

Keepalive and peer link on physically separate paths and modules, both monitored.

More on this: Data centre networks

Nexus and vPCvPC primary reloads, short black hole on return

Symptom. Traffic lost for seconds when the old primary comes back.

Likely causes

delay restore too short for the routing table.
ARP sync off.

What I check, in order

show vpc role (secondary, operational primary)
show vpc
show port-channel summary
show logging logfile | egrep "VPC|ETHPORT"

Fix

Raise delay restore, enable ip arp synchronize.

Stop it coming back

Tune timers to table size. Peer switch avoids STP root change.

More on this: Data centre networks, Networking fundamentals

Nexus and vPCOne vPC member link fails

Symptom. Device reachable but peer link load rises.

What I check, in order

show vpc brief
show port-channel summary
show interface ethernet1/1
show interface port-channel 1 | include rate

Fix

Replace cable or optic under change, check P flag on both peers. RCA (root cause analysis) if it repeats.

Stop it coming back

Size the peer link for line card failure cases.

More on this: Data centre networks

Nexus and vPCBoth vPC peers lose power and only one returns

Symptom. Surviving switch keeps all vPCs down.

What I check, in order

show vpc (peer is not reachable; auto recovery timer running)
show logging logfile | egrep "AUTO_RECOVERY|VPC"

Fix

Auto recovery brings vPCs up after 240 s. Without it, configure it or manually intervene.

Stop it coming back

Auto recovery as a standard setting.

More on this: Data centre networks

Nexus and vPCvPC down on one peer after last night's change

Symptom. show vpc brief shows consistency failed, for example "Compatibility check failed for MTU".

Likely causes

Type 1 mismatch (MTU, LACP mode, port mode; or global STP mode or MST region)

What I check, in order

show vpc consistency-parameters interface port-channel 10
show vpc consistency-parameters global
show accounting log | include port-channel10
show running-config interface port-channel10

Fix

Align the wrong peer (for example mtu 9216). Global fixes on both peers together.

Stop it coming back

Change templates name both peers. Post check show vpc brief on both. Documented standard values. Problem record for process gaps.

More on this: Data centre networks, AI infrastructure, Networking fundamentals

Nexus and vPCNew VLAN works for some hosts behind a vPC but not others

Symptom. Intermittent reachability in one VLAN. Orphan devices worst hit.

Likely causes

VLAN created on one peer only.
Missing from the peer link.
Missing from the member trunk on one peer.
Per VLAN consistency (STP disabled for the VLAN on one peer)

What I check, in order

show vlan id 50
show vpc brief (peer link active VLANs versus vPC active VLANs)
show vpc consistency-parameters vlans
show interface trunk
show spanning-tree vlan 50
Local suspended VLANs line.

Fix

Create the VLAN and add it with "add" on both peers and the peer link.

Stop it coming back

VLAN change template covering both peers and the peer link.

More on this: Data centre networks

Nexus and vPCvPC peer link running hot with drops

Likely causes

Orphan devices.
vPC members down on one side.
Layer 3 uplinks or routes on only one peer.
Routing adjacencies over vPC.

What I check, in order

show interface port-channel 1 | egrep "rate|drops"
show vpc orphan-ports
show port-channel summary
show ip route 0.0.0.0/0 on both.
show ip ospf neighbors
show hsrp brief

Fix

Dual attach, symmetric uplinks, dedicated L3 (Layer 3) links, separate link for non vPC VLANs. Quick relief: add peer link members, move orphans to the peer where their traffic arrives.

Stop it coming back

Problem record with a design CR (change request).

More on this: Data centre networks

Nexus and vPCFEX offline or stuck

Symptom. All ports on a FEX (Fabric Extender) down, or FEX in discovery or image download.

What I check, in order

show fex
show fex 101 detail
show interface fex-fabric
Parent version.

Fix

Repair fabric uplinks. Wait for the image push before reloading anything.

More on this: Data centre networks

Nexus and vPCvPC peer link fails

Symptom. Half the vPC member ports go down.

Likely causes

Peer link down while keepalive is up, so the secondary suspends its vPC ports by design.
Both down gives dual active split brain.

What I check, in order

show vpc (peer status, keepalive status, consistency)
Confirm keepalive path is separate from the peer link.

Fix

Restore the peer link. Never run keepalive over the peer link.

Stop it coming back

Diverse peer link members, keepalive over the management network.

More on this: Data centre networks

OSPFOSPF adjacency stuck at ExStart or Exchange

Likely causes

MTU mismatch.
Also check timers, area, authentication, subnet.
2-Way between DROTHERs is normal.

What I check, in order

show ip ospf neighbor
show interface MTU both ends.

Fix

Match MTU (or ip ospf mtu-ignore as a last resort). Prevention: MTU assertions in templates.

More on this: Routing protocols, Networking fundamentals

OSPFRoute redistribution loop between OSPF and BGP

Symptom. Route flaps, loops or suboptimal paths after mutual redistribution.

Likely causes

Routes re injected into the protocol they came from.
Metric or AD (administrative distance) clashes.

What I check, in order

Route source and tags.
Administrative distance on each router at the boundary.

Fix

Route maps with tags and filters, a single clear redistribution boundary.

Stop it coming back

Never redistribute ad hoc. Define boundaries in the design.

More on this: Routing protocols

OSPFOSPF neighbour will not come up

Symptom. Neighbour missing from show ip ospf neighbor, or stuck in a state.

Likely causes

Area.
Subnet or mask.
Hello and dead.
Network type.
Stub or NSSA (not so stubby area) flag.
Authentication mismatch.
Duplicate router ID.
Passive interface.
ACL or firewall blocking 224.0.0.5/6 or protocol 89.
Interface not in OSPF (wrong wildcard, missing ip ospf or ip router ospf)
Layer 2 (VLAN not allowed on trunk, STP blocking, one way link)
MTU (ExStart/Exchange)

What I check, in order

show ip ospf neighbor
show ip ospf interface brief
show ip ospf interface Gi0/0/1 on both ends side by side.
show ip protocols
show interfaces Gi0/0/1 | include MTU|errors|CRC
Ping the far interface.
show ip access-lists hit counts
Firewall logs or a capture.
show logging | include OSPF
debug ip ospf hello ("Mismatched hello parameters", mismatched area)
debug ip ospf adj (auth, "Nbr has smaller interface MTU")
undebug all
Hellos: area, timers, subnet and mask, authentication, stub flags, passive interface, ACL blocking 224.0.0.5 and 224.0.0.6.
Init means one way hellos.
2-Way normal between DROthers.

Fix

Correct the mismatch on the side that drifted, under change.

Stop it coming back

Record the cause. A mismatch usually means one side was changed without the other, so add both ends to the change template.

More on this: Routing protocols, Networking fundamentals

OSPFNeighbour Full but a route is missing

Likely causes

Likely causes, not in LSDB (link state database). Interface not in OSPF at the source, ABR (Area Border Router) area range or filter-list, area type blocking externals, ASBR (Autonomous System Boundary Router) redistribution not matching (route map, missing subnets keyword on old IOS). In LSDB not in RIB (routing information base): distribute-list in, lower AD from another source, network type mismatch breaking SPF, type 5 forwarding address unreachable, ASBR unreachable (no type 4). Or the route is summarised on purpose.

What I check, in order

show ip ospf database | include <prefix>
show ip ospf database summary/external <prefix>
show ip route <host address> (longest match)
show ip route ospf
show ip protocols (distribute lists, AD, redistribution)
show ip ospf border-routers

Fix

Correct the origin, filter, area type or AD.

Stop it coming back

Redistribution via tagged route maps. Documented summaries.

More on this: Routing protocols

OSPFOSPF takes a suboptimal path

Likely causes

Default reference bandwidth (1G and 10G both cost 1)
Different reference bandwidth between routers.
Leftover ip ospf cost
Route type preference (intra beats inter area)
E2 ignoring internal cost.
Summarisation hiding the better path.
Inter area transit through area 0.
Asymmetric return path through stateful firewalls.

What I check, in order

traceroute <dest> source Loopback0
show ip route <dest> on each hop.
show ip ospf interface brief on each router.
show ip ospf database router <id>
show ip cef exact-route <src> <dst>

Fix

Align reference bandwidth, set explicit costs, move a link to the right area, under change with before and after paths in the record.

More on this: Routing protocols

OSPFOSPF adjacency keeps flapping

Likely causes

Physical faults or carrier circuit.
Lost hellos from high CPU.
CoPP.
Congestion without QoS for control traffic.
One way link.
STP changes.
MTU, duplicate router ID, BFD too aggressive.

What I check, in order

show logging | include ADJCHG|BFD|LINK (%OSPF-5-ADJCHG... Dead timer expired versus interface down)
show interfaces (errors, CRC, flaps)
show interfaces transceiver
show processes cpu sorted
show policy-map control-plane (CoPP drops)
show spanning-tree detail | include changes|from
show ip ospf statistics (SPF runs)
show bfd neighbors details
Time pattern (backup jobs)

Fix

Physical repair or carrier ticket with evidence. Protect control traffic. Relax BFD.

Stop it coming back

Monitoring of SPF runs and adjacency changes. QoS for control traffic.

More on this: Routing protocols, AI infrastructure, Networking fundamentals

OSPFOSPF stuck in ExStart after a circuit migration

Symptom. Interface up, neighbour cycling EXSTART/EXCHANGE, traffic on the backup path.

Likely causes

MTU mismatch from the new handoff or subinterface.
If MTU matches on both routers, something in between (carrier switch, firewall)
Then duplicate router ID.
Then unicast DBD (database description) blocked.

What I check, in order

Confirm backup is carrying traffic (impact)
show ip ospf neighbor
show interfaces Gi0/0/1 | include MTU on both ends.
show ip ospf interface
ping <far end> size 1500 df-bit, then 1400.
debug ip ospf adj ("Nbr... Has smaller interface MTU")
undebug all

Fix

Align mtu and ip mtu to what the circuit carries under emergency or normal change. Confirm Full, routes back, traffic back on primary.

Stop it coming back

Do not use ip ospf mtu-ignore except as a short documented workaround. Problem record so the migration runbook gains an MTU step (CSI)

More on this: Routing protocols, Networking fundamentals

OSPFOSPF adjacency will not reach Full

Symptom. Neighbours stuck in ExStart or Exchange, or stuck in 2-Way, or not forming.

Likely causes

ExStart or Exchange: MTU mismatch.
2-Way between DROthers is normal.
No adjacency at all: mismatched hello or dead timers, area ID, authentication or subnet.

What I check, in order

show ip ospf neighbor
Compare interface MTU, timers, area, authentication and subnet on both sides.

Fix

Align MTU or the mismatched parameter.

Stop it coming back

Interface templates.

More on this: Routing protocols, Networking fundamentals

BGPBGP neighbour stuck in Active or Idle

Symptom. The session never reaches Established.

Likely causes

No TCP 179 reachability.
Wrong neighbour IP or remote AS (Autonomous System).
An ACL or firewall blocking 179.
Wrong source interface (needs update-source loopback)
Authentication mismatch (MD5 (Message Digest 5) or TCP-AO)
eBGP (External BGP) multihop not set when peering between loopbacks.
Maximum prefix limit reached.
Admin shutdown (Idle (Admin))
Max prefix exceeded (Idle (PfxCt), MAXPFXEXCEED)
No route to the neighbour (loopback peering with missing IGP (Interior Gateway Protocol) route; eBGP TTL 1 to a non connected peer)
Back off after an error (NOTIFICATION 2/2 bad peer AS)

What I check, in order

show ip bgp summary, show ip bgp neighbors
Reachability to the peer address.
ACLs and firewalls on the path.
Configured AS, source and password on both ends.
show ip bgp neighbors x | include BGP state|Last reset|prefix
show ip route <neighbour>
show logging | include BGP|NOTIFICATION|MAXPFX
ping x source Loopback0
telnet x 179 /source-interface Loopback0 (open means BGP (Border Gateway Protocol) config, refused means far end not configured, timeout means dropped)
show tcp brief | include 179
show ip bgp neighbors x | include BGP state|Local host|Foreign host
show logging | include BADAUTH
show access-lists

Fix

Correct the failing item.

Stop it coming back

Peer templates, session state monitoring, max prefix alerts before the limit.

More on this: Routing protocols

BGPBGP Established but no routes exchanged

Likely causes

Routes not in my BGP table (network mask not matching the RIB exactly, redistribution route map)
Outbound or inbound filters (implicit deny)
Address family not activated (no bgp default ipv4-unicast, IPv6 (Internet Protocol version 6), VPNv4 (VPN IPv4 address family))
iBGP (Internal BGP) split horizon.
NO_ADVERTISE or NO_EXPORT community.
Next hop unreachable.

What I check, in order

show ip bgp neighbors x advertised-routes / received-routes / routes
show ip bgp <prefix>
show ip route <prefix>
show route-map
show ip prefix-list

Fix

Correct network statement, filters, activation, next-hop-self.

More on this: Routing protocols

BGPRoutes in the BGP table but not in the routing table

Likely causes

Not best (no >, next hop unreachable)
RIB failure (r code: static at 1 or OSPF at 110 beats iBGP at 200, maybe a forgotten static)
A more specific route from elsewhere wins.
Wrong VRF.

What I check, in order

show ip bgp | include ^r
show ip bgp rib-failure ("Higher admin distance")
show ip bgp <prefix>
show ip route <prefix>
show ip route vrf RED <prefix>

Fix

Remove the overriding route or design so a prefix is not learned two ways.

More on this: Routing protocols

BGPBGP session flapping

Likely causes

Hold timer expiry with interface up (congestion without QoS, high CPU, CoPP drops, MTU: small OPEN and keepalives pass, first big UPDATE dropped, PMTUD (path MTU discovery) broken by blocked ICMP)
BFD too aggressive.
NOTIFICATION code.
Max prefix restart cycle.
Physical or carrier flaps.

What I check, in order

show logging | include BGP-5-ADJCHANGE|NOTIFICATION|BFD
show ip bgp neighbors x | include Last reset|hold time|max data segment
show bfd neighbors details
show interfaces | include drops|errors
ping x size 1500 df-bit
show policy-map control-plane
show processes cpu sorted

Fix

QoS for control traffic, MTU alignment or allow ICMP, relax BFD, fix the prefix source.

More on this: Routing protocols, AI infrastructure, Networking fundamentals

BGPBGP to the WAN provider stuck Active after a firewall change

Symptom. Site on backup path after last night's firewall change.

Likely causes

Rule missing, NAT (Network Address Translation) changing the source, ASA stripping TCP option 19 (MD5) or randomising sequence numbers, asymmetric path through the stateful firewall.

What I check, in order

Confirm impact and backup.
Read the change record.
show ip bgp summary
ping 192.0.2.1 source Gi0/0/0
telnet 192.0.2.1 179 /source-interface Gi0/0/0
Firewall rule base for TCP 179 both ways.
New NAT translating the source.
Firewall logs or capture.
On an ASA: packet-tracer input inside tcp 192.0.2.2 33000 192.0.2.1 179 detailed, capture capbgp interface inside match tcp host 192.0.2.2 host 192.0.2.1 eq 179, show capture capbgp, show conn address 192.0.2.1, show asp drop
show logging | include BGP|BADAUTH

Fix

Emergency change or roll back. ASA TCP map allowing option 19 and no sequence randomisation for the flow. Confirm Established, prefix count and traffic off backup.

Stop it coming back

Pre change list of routing peers crossing the firewall. Post change BGP test in the plan. Alert on BGP neighbour down, not only interface down. Problem record.

More on this: Routing protocols

BGPBGP between NSX Tier-0 and physical switches down

What I check, in order

Scope (one Edge or all; one switch or both: all at once points at the switch or uplink VLANs)
Switch: show bgp ipv4 unicast summary vrf all, show bgp ipv4 unicast neighbors <edge>, show ip arp <edge uplink>, show interface vlan 2001, show bfd neighbors
Uplink VLAN trunked to the ESXi host carrying the Edge and on the VDS uplink port group.
SVI up.
ACL or CoPP on TCP 179.
AS, IPs, password, BFD, multihop.
MTU on the uplink.
Edge (admin): get logical-routers (newer get gateways), vrf <id>, get interfaces, get bgp neighbor summary, get route, get bgp neighbor x advertised-routes
Edge not in maintenance mode.
Gateway firewall rule on the Tier-0 uplink.

Fix

Correct the layer 2 path, BGP parameters or firewall rule.

More on this: Routing protocols, Networking fundamentals

BGPBGP up but switches do not learn VM segment subnets

Likely causes

Tier-1 route advertisement for connected segments not enabled.
Segment not attached to that Tier-1 or Tier-1 not linked to the Tier-0.
Outbound route map or prefix list on the Tier-0 neighbour.
Inbound filter on the switch.
NAT and LB (load balancer) addresses need their own options.
No default route sent back to NSX.

What I check, in order

get route on the Tier-0 SR (service router).
get bgp neighbor x advertised-routes
Switch show ip bgp neighbors <edge> routes, show bgp ipv4 unicast <prefix>, show ip route <prefix>

Fix

Tick advertisement and redistribution options, fix filters, send a default back.

More on this: Routing protocols

BGPBGP reaches OpenSent then drops

Likely causes

TCP fine, BGP parameters wrong: AS number, authentication password, capabilities.
Neighbour detail shows the NOTIFICATION.
eBGP TTL issues if a device in the path decrements TTL (multihop or TTL security)

Worth knowing

BGP MD5 through an ASA. The ASA strips TCP option 19 and randomises sequence numbers unless a TCP map says otherwise:

tcp-map BGP-MD5
 tcp-options range 19 19 allow
class-map BGP
 match port tcp eq bgp
policy-map global_policy
 class BGP
  set connection random-sequence-number disable
  set connection advanced-options BGP-MD5

More on this: Routing protocols

Gateways and VRFsAfter a gateway move some hosts cannot reach anything for about ten minutes, then recover

Symptom. Partial, self healing outage after the gateway move.

Likely causes

Host ARP caches still point at the legacy HSRP virtual MAC.
The anycast gateway uses a different MAC and no gratuitous ARP reached the hosts.
Recovery when ARP ages out.

What I check, in order

show ip arp (or arp -a) on an affected host.
Leaf ARP suppression cache.
Confirm legacy SVI was shut before the fabric SVI came up (duplicate IP gives a different symptom)
Storm control counters on the bridge trunk.

Fix

Send GARP (Gratuitous ARP) from the fabric SVI or clear host ARP.

Stop it coming back

Decide once to reuse the legacy virtual MAC as the fabric anycast gateway MAC, or always send GARP, or shorten host ARP timers. Post check "probe hosts ARP again within 30 seconds".

More on this: Routing protocols, Networking fundamentals

Gateways and VRFsUsers fail over to the standby gateway but return traffic is lost (asymmetric routing through a stateful firewall)

Symptom. ICMP works, TCP sessions fail or reset after an HSRP failover.

Likely causes

HSRP active and routing preference on different switches.
ECMP (Equal Cost Multi Path) across two firewalls without shared state.
PBR (policy based routing) or a static sending return traffic another way.
uRPF (Unicast Reverse Path Forwarding) strict on the return interface.

What I check, in order

show standby brief
show ip route <user subnet> on the upstream router or firewall.
traceroute both ways
ASA show asp drop | include not SYN, capture asp type asp-drop tcp-not-syn, show capture asp
FortiGate diagnose debug flow filter addr x, diagnose debug flow trace start 100, diagnose debug enable ("no session matched")

Fix

Align FHRP (First Hop Redundancy Protocol) and routing (same track objects), return route follows the active gateway, state sharing firewall clusters.

Stop it coming back

Design rule for symmetric paths. Never disable state checking on a secure network.

More on this: Routing protocols

Gateways and VRFsHSRP flapping between active and standby

Likely causes

Lost hellos (VLAN missing from the inter switch trunk, STP changes, one way link, storm, high CPU or CoPP)
Both active (layer 2 path broken, auth or version mismatch)
Tracked object flapping with preempt.
Duplicate group number on the VLAN.

What I check, in order

show logging | include HSRP|SPANTREE|LINK|TRACK (%HSRP-5-STATECHANGE timestamps)
show standby Vlan10
show standby brief
show interfaces trunk
show spanning-tree vlan 10 detail | include occurr|from|executing
show track brief
show processes cpu sorted
debug standby events briefly to buffer

Fix

Repair the layer 2 path or tracked link, add preempt and track delays, align versions and auth.

More on this: Routing protocols, AI infrastructure

Gateways and VRFsRoute leaking between VRFs not working

Likely causes

Route target typo.
Prefix not in BGP for the VRF (missing redistribute connected)
No return route.
Overlapping local prefix wins.
Static next hop invalid in the other table.
Firewall policy if inter VRF traffic is meant to cross one.

What I check, in order

show vrf detail RED (RD (route distinguisher), import and export RTs (Route Target))
show ip route vrf SHARED <prefix>
show ip route vrf RED <prefix> (return)
show ip bgp vpnv4 all | include <prefix>
show ip bgp vpnv4 vrf RED <prefix>
ping vrf RED x source Vlan110

Fix

Correct RTs character by character, redistribute, add return route, NAT for overlaps.

More on this: Routing protocols, Data centre networks

Gateways and VRFsRedistribution loop

Symptom. Routes changing source every few seconds, traceroute bouncing between two routers, TTL exceeded, high CPU from SPF or DUAL (Diffusing Update Algorithm).

What I check, in order

show ip route <prefix> (source, Route tag)
show ip ospf database external | include Tag|Link State ID
show ip protocols

Fix

Tag and deny route maps on every redistributing router. Summarise at the boundary. Adjust AD deliberately.

More on this: Routing protocols

Gateways and VRFsAfter overnight maintenance one VLAN has no working gateway

Likely causes

VLAN deleted, missing from a trunk (allowed vlan without add), SVI left shut, VLAN missing from the vPC peer link or a consistency problem.
VLAN not carried between cores so both become HSRP active (intermittent gateway)
HSRP flapping: timers, group numbers, version (v1 224.0.0.2, v2 224.0.0.102, UDP 1985), authentication, ACL or CoPP dropping hellos.

What I check, in order

Read the CR.
On both core or vPC peers show vlan id 130, show interfaces Vlan130 (or show interface vlan 130), show standby brief or show hsrp brief. show interfaces trunk, show spanning-tree vlan 130, show vpc brief, show vpc consistency-parameters vlans

Fix

Back out the change or correct under emergency change (switchport trunk allowed vlan add 130). Test from a host in the VLAN.

Stop it coming back

Incident linked to the CR (change success rate, PIR (post incident review))

More on this: Routing protocols, Data centre networks

WAN and VPNSite to site IPsec tunnel down

Symptom. No traffic over a site to site VPN.

Likely causes

Phase 1 failure (peer unreachable, UDP 500 or 4500 blocked, PSK (Pre Shared Key) or parameter mismatch, NAT in the path)
Phase 2 failure (crypto ACL or proxy ID not mirrored, the most common cause, transform set mismatch)
Routing or a firewall dropping ESP (Encapsulating Security Payload) or IKE (Internet Key Exchange).
MTU or MSS black holing.
Evidence to cause. No SA (security association) means no interesting traffic, crypto map not applied or no route to peer. MM_NO_STATE with "retransmitting phase 1" means peer unreachable, UDP 500 blocked, wrong peer, peer not configured. "atts are not acceptable" means phase 1 proposal mismatch. Stuck MM_KEY_EXCH with %CRYPTO-4-IKMP_BAD_MESSAGE means pre shared key mismatch. IKEv2 (Internet Key Exchange version 2) NO_PROPOSAL_CHOSEN means proposal mismatch. AUTHENTICATION_FAILED means key, identity or certificate. Works until NAT appears means UDP 4500 blocked or NAT-T off.

What I check, in order

show crypto isakmp sa or show crypto ikev2 sa for phase 1.
If phase 1 is down, check reachability, UDP 500 and 4500, the PSK, phase 1 parameters and NAT.
show crypto ipsec sa for phase 2, watching the encaps and decaps counters: encaps rising with no decaps means a return problem at the far side (ACL, routing or firewall)
Confirm the crypto ACLs mirror exactly.
Check routing in both directions and any firewall on the path.
Test large packets for MTU problems.
Underlay ping x source Gi0/0/0
UDP 500, 4500 and ESP allowed.
Generate interesting traffic with a sourced ping
show crypto map
show crypto session detail
debug crypto condition peer ipv4 x then debug crypto isakmp or ikev2.
Ask the far end for their logs.

Fix

Correct the mismatch, mirror the selectors, open the ports, clamp MSS.

Stop it coming back

IKEv2 with DPD (Dead Peer Detection), tunnels monitored like any other link.

More on this: WAN, MPLS (Multiprotocol Label Switching) and VPN, Networking fundamentals

WAN and VPNA tunnel works without NAT but fails through it

Symptom. VPN comes up on a direct path, fails behind NAT.

Likely causes

ESP (IP protocol 50) has no ports, so NAT cannot track it.
NAT-T needs UDP 4500 and it is blocked.

What I check, in order

Confirm a NAT device in the path.
Check that UDP 4500 is allowed.
Confirm NAT-T is negotiated (built into IKEv2)

Fix

Allow UDP 4500, enable NAT-T or move to IKEv2.

Stop it coming back

IKEv2 as the default.

More on this: WAN, MPLS and VPN

WAN and VPNA site has lost its MPLS routes

Likely causes

Physical or PE (provider edge) fault.
BGP down (Idle or Active)
Provider not sending.
Inbound filter or max prefix.
AS path loop rejection because every site uses the same AS (needs allowas-in or as-override)
Not advertising our own routes so others cannot return.

What I check, in order

Scope (one site or many: many means provider, raise at once)
Carrier maintenance notices and own change calendar.
show interfaces Gi0/0/0 | include line protocol|errors|drops
Ping the PE.
show ip bgp summary
show ip bgp neighbors x | include state|Last reset|prefixes|loop
show ip bgp neighbors x routes
advertised-routes
show ip route bgp

Fix

Per cause. Carrier ticket with circuit ID, timestamps, counters, BGP state and expected routes.

Stop it coming back

Keep service desk and users updated in the ticket while the carrier works.

More on this: WAN, MPLS and VPN

WAN and VPNPhase 1 up, phase 2 fails

Likely causes

Proxy IDs not mirrored ("proxy identities not supported", TS_UNACCEPTABLE)
Transform set mismatch.
PFS (Perfect Forward Secrecy) on one side only or different group.
Route based side proposing any to any to a policy based peer.

What I check, in order

show crypto ipsec sa peer x (local and remote ident)
show access-lists VPN-TO-B
show crypto ipsec transform-set
debug crypto ipsec
show crypto ikev2 sa detailed
Compare both configs on a shared call.

Fix

Mirror the selectors, set specific proxy IDs on the route based side, align transforms and PFS.

More on this: WAN, MPLS and VPN

WAN and VPNVPN users can ping but large transfers and some pages hang

Likely causes

Tunnel overhead (GRE 24, ESP 50 to 70+, NAT-T 8 more) with ICMP 3/4 blocked (PMTUD black hole)

What I check, in order

ping x source Vlan10 size 1500 / 1400 / 1360 df-bit
Windows ping -f -l 1372 (1372 + 28 = 1400)
show interfaces Tunnel10 | include MTU
show ip interface Tunnel10 | include MTU|MSS

Fix

ip mtu 1400 and ip tcp adjust-mss 1360 (1400 minus 40). Allow ICMP 3/4 through firewalls. Note MSS clamping only helps TCP.

More on this: WAN, MPLS and VPN, Networking fundamentals

WAN and VPNApplication slow over the WAN for a remote site

Likely causes

On backup path.
QoS misclassification competing with bulk jobs.
MTU.
TCP window and latency limit on long links.
Asymmetric return through a firewall.
Server or app problem.

What I check, in order

Pin down which app, users, since when, busy times, other apps.
Monitoring for utilisation, latency, loss, errors.
Change calendar.
traceroute x source Vlan10 (silent failover to a smaller backup)
show ip route
show interfaces | include rate|drops|errors|duplex
show policy-map interface (class drops, app in class-default)
show ip sla statistics
ping repeat 200 size 1400
Compare from a site near the server.
Packet capture for retransmissions and window sizes.

Fix

Restore primary, correct classification, reschedule bulk traffic, capacity request backed by data.

More on this: WAN, MPLS and VPN, Networking fundamentals

WAN and VPNGRE tunnel keeps going down with %TUN-5-RECURDOWN

Likely causes

Route to the tunnel destination learned through the tunnel itself

What I check, in order

show ip route <tunnel destination>
show logging | include TUN

Fix

Static or filtered route so the destination is reachable only via the underlay.

More on this: WAN, MPLS and VPN

WAN and VPNA whole site reports intermittent slowness

Likely causes

Backups, updates or large transfers saturating the WAN.
Bad optic or duplex mismatch.
Undersized QoS class.
Topology change flushes.
MTU.
Server side.

What I check, in order

Monitoring history against complaint times.
WAN utilisation peaks with NetFlow top talkers (show flow monitor <name> cache sort highest counter bytes top 10)
Errors and drops (show interfaces Gi0/0/0 | include rate|drops|errors|duplex, show interfaces counters errors)
QoS class drops (show policy-map interface)
CPU history.
STP topology changes.
MTU with ping size 1500 df-bit repeat 100
Is it one application or the server.
DNS response times and the proxy.

Fix

Per cause via change, with before and after measurements. If monitoring shows nothing, capture at the site edge during a slow period.

More on this: WAN, MPLS and VPN, Networking fundamentals

WAN and VPNA remote site has lost its WAN connection

What I check, in order

Monitoring.
Reach the ISR (Integrated Services Router) via backup link or OOB.
show interfaces Gi0/0/0 (down/down physical or carrier; up/down layer 2; up/up with routing down means routing, provider or filter)
show controllers
show logging | include LINK|LINEPROTO|BGP|OSPF
show ip bgp summary
show ip route 0.0.0.0
show track
show ip sla statistics (did failover work)

Fix

Carrier fault with circuit ID, timestamps and evidence. Local staff check NTE (network terminating equipment) power and lights with hands and eyes instructions. Dead router means site visit with a spare and the backed up config. Regular updates via the service desk. Carrier says no fault found: request intrusive tests in a window, share our error counters and timestamps, check our cable to the NTE, port and optic.

More on this: WAN, MPLS and VPN

WAN and VPNReplication breaks only after failover

Symptom. Replication or backups fail when traffic moves to the backup path.

Likely causes

Backup carrier path with 1500 byte MTU.
Untested backup path.

What I check, in order

Full size DF pings on every path including backup.

Fix

Raise MTU or adjust host MTU or MSS.

Stop it coming back

Test backup paths in acceptance. MTU in contracts.

More on this: WAN, MPLS and VPN, Networking fundamentals

NSXSame segment VMs talk on one host but not across hosts

Likely causes

Tunnel or underlay: TEP (tunnel endpoint) VLAN not trunked.
Wrong transport VLAN or double tagging.
No route between TEP subnets.
Duplicate TEP IP.
Firewall blocking UDP 6081 or BFD.
MTU.
Control plane (MAC behind wrong TEP, host disconnected from controllers)
A recent switch port, host replacement or switch upgrade.

What I check, in order

Tunnel status in the UI (User Interface).
nsxdp-cli bfd sessions list
esxcfg-vmknic -l
vmkping ++netstack=vxlan -I vmk10 <TEP> then -d -s 1572.
Switch show interface trunk, show vlan id <TEP VLAN>, show mac address-table vlan <TEP VLAN>, show interface <port> | include MTU
nsxcli get segment <vni> mac-table and vtep-table
get controllers

Worth knowing

One host only. Compare its switch ports, uplink profile vmnic mapping (swapped vmnics after hardware change), TEP IPs and controller connection with a working host.

Fix

Correct VLAN, trunk, route, rule or MTU in the underlay.

More on this: VMware NSX, Networking fundamentals

NSXSmall pings work, large transfers stall and SSH hangs (overlay MTU)

What I check, in order

vmkping ++netstack=vxlan -I vmk10 -d -s 1572 <TEP> fails while small ping works
VDS MTU in vCenter.
Host ports, port channels, vPC peer link, SVIs and routed links, spine links, any new device.
For north south only: Edge uplink MTU versus ToR (Top of Rack) SVI and beyond (WAN or firewall with smaller MTU and ICMP blocked)

Fix

Consistent MTU under change (9216 on fabric), retest with 1572 or 8972. Check guest MTU has not been raised. MSS clamping is a workaround only.

More on this: VMware NSX, Data centre networks, Networking fundamentals

NSXA VM cannot reach its default gateway

What I check, in order

VM IP, mask, gateway, vNIC (virtual Network Interface Card) connected to the right segment or port group.
Segment gateway address and connection to a Tier-1.
Traceflow to the gateway IP (DFW (Distributed Firewall) rule ID)
SpoofGuard and IP discovery bindings after an IP change or duplicate.
get segment <vni> mac-table
get firewall packetlog last 20
Compare with a working VM on the same segment.

Fix

Connect the segment, fix the VM, approve the binding, or correct the rule.

More on this: VMware NSX

NSXNew segment unreachable from the physical network

What I check, in order

Tier-1 route advertisement.
Edge vrf <T0 SR> get route <prefix> (t1c)
get bgp neighbor <ToR> advertised-routes
Tier-0 redistribution and route maps.
Nexus show bgp ipv4 unicast <prefix>, show ip route <prefix>, show ip prefix-list NSX-IN
Onward propagation and firewalls.
NAT case: advertise All NAT IPs and redistribute NAT IPs

Fix

Tick advertisement or redistribution, update filters via change.

Stop it coming back

Add a prefix list and redistribution check to the segment build runbook.

More on this: VMware NSX, Data centre networks

NSXTier-0 BGP neighbour down after a ToR change

What I check, in order

Change record before and after.
Edge get bgp neighbor summary, get interfaces, ping ToR from the SR VRF.
Uplink VLAN on the trunk to the Edge's host, SVI state, IP and mask, ARP.
Neighbour IP, AS, password, hop limit, update source, ACL or CoPP on TCP 179.
BFD timers and MTU both sides.
Nexus show bgp ipv4 unicast summary, neighbors, show bfd neighbors, show logging logfile | include BGP
Confirm ECMP keeps traffic flowing (impact is resilience)

Fix

Roll back via the original change's back out plan if clearly the cause and impact grows, else a correcting change. Record root cause.

More on this: VMware NSX, Data centre networks, Networking fundamentals

NSXApplication broke straight after a DFW change

What I check, in order

Traceflow client to server on the app port (rule ID)
Rule hit counts.
DFW packet log.
Category and rule order (new deny above an allow or in an earlier category)
Group effective members (tag removed, VM renamed or rebuilt with a new IP)
Applied To too narrow.
Services and ports incomplete.
Host vsipioctl getrules and getaddrsets.

Fix

Fix first: back out under emergency change using the change's plan (saved DFW config versions help).

Stop it coming back

Peer review of DFW changes, Traceflow test in the change plan, logging on new denies, problem record.

More on this: VMware NSX

NSXHost transport node Partial Success or preparation failed

Likely causes

Software (maintenance mode or reboot needed, space, old bundles, vLCM image misaligned)
Connectivity (DNS, routing, firewall, certificates)
Profile (vmnic mapping, VDS not 7+, VDS MTU, transport zones)
TEP pool exhausted.

What I check, in order

Node status detail (names the failing step)
esxcli software vib list | grep -i nsx (or component list)
esxcfg-vmknic -l
tail -f /var/log/nsx-syslog.log
nsxcli get managers, get controllers

Fix

Correct cause then resolve or retry with the host in maintenance mode. Only return to service once tunnels are up. Never improvise NSX removal.

More on this: VMware NSX, Networking fundamentals

NSXOne of three NSX Managers down

What I check, in order

get cluster status on a healthy node.
Alarms.
VM power, datastore health and latency, disk, certificates.

Worth knowing

Protect. Freeze non urgent NSX changes, confirm last good backup. Two Managers down is a priority 1, vendor case, avoid host reboots and vMotion.

Fix

Restart or storage fix under emergency change. Documented replace procedure with vendor support while quorum holds.

More on this: VMware NSX, AI infrastructure

NSXNorth south slow with packet loss, Edge VM looks busy

What I check, in order

get dataplane cpu stats (DPDK (Data Plane Development Kit) polling makes vCenter CPU look high)
get physical-ports, get physical-port fp-eth0 stats
get logical-routers (how many SRs on this Edge)
get edge-cluster status
ToR show ip route <range> longer-prefixes (both Edges used?)
Host contention.
Elephant flows pinning a core.

Fix

Fix now: move Tier-1 SRs, fail over a standby. Later via change: bigger form factor or bare metal, more Edges (ECMP up to 8), separate service Edge cluster. Problem record.

More on this: VMware NSX

NSXTraffic dropping for one vSphere cluster after an NSX upgrade

What I check, in order

Per host upgrade state (stuck, failed, in maintenance), node status, tunnels, controller connection, DFW realisation.
TEP checks (vmk ports, VLAN, DF ping)

Fix

Contain: stop the upgrade, move workloads to healthy hosts, keep other clusters on the old version. Vendor case with upgrade logs, problem review.

Stop it coming back

Release notes and interoperability matrix (ESXi, VDS, vCenter), verified backup, coordinator pre checks, Edges then a pilot cluster then the rest then managers, Traceflow and BGP checks per stage, defined stop points.

More on this: VMware NSX

NSXIntermittent north south drops with active/active Tier-0 and a stateful firewall above the ToRs

Likely causes

ECMP sends outbound via one Edge and return via another, firewall sees one direction.
uRPF on ToR or firewall.

What I check, in order

Start capture interface on both Edges.
Firewall logs for out of state or no session drops.
ToR routing table showing both Edges as next hops.

Fix

State aware firewall design, BGP steering (prepend or local pref) for that traffic, active/standby Tier-0 or stateful active/active where supported. Agree with the design authority rather than disabling ECMP.

More on this: VMware NSX

NSXSilent MTU fault: small packets work, large transfers hang

Symptom. Ping and SSH succeed. File copies, backups, TLS handshakes, vMotion or replication stall or hang, sometimes only after failover to a backup path.

Likely causes

One link (fabric, SVI, L3VNI (Layer 3 VNI) SVI, DCI, carrier backup at 1500) below the needed MTU.
VTEPs (VXLAN Tunnel Endpoint) must not fragment so oversize VXLAN drops silently.
NSX Geneve inside VXLAN without headroom.
1500 byte external handoff without MSS adjust.

What I check, in order

DF bit pings at increasing sizes between VTEP loopbacks and host to host in every VNI (ping <vtep> source <vtep> df-bit packet-size 9000)
Check MTU on physical ports, SVIs and the L3VNI SVI.
Test the backup DCI path too.
For ACI (Application Centric Infrastructure) Multi-Pod check IPN (Inter Pod Network) MTU 9150.

Fix

Set 9216 on all fabric, super spine and DCI links and SVIs, 9000 or 1500 on hosts. TCP MSS adjustment at 1500 byte borders. Carrier MTU in contract. Lower host MTU where only 1500 exists.

Stop it coming back

MTU standard in the pod design. Full size DF ping tests in factory acceptance and after DCI changes.

More on this: VMware NSX, VXLAN EVPN (Ethernet Virtual Private Network) and ACI, Networking fundamentals

FirewallsAn application broke after moving behind a new firewall

Symptom. The application fails after a migration although the rule looks right.

Likely causes

Flow not actually permitted.
Asymmetric routing so the return path skips the firewall and the stateful check drops it.
NAT mismatch (the application sees an unexpected source IP)
MTU or MSS over a new tunnel.
Dynamic or secondary ports (FTP (File Transfer Protocol), SIP (Session Initiation Protocol), RPC (Remote Procedure Call)) needing an ALG (Application Layer Gateway) or inspection.

What I check, in order

Confirm src, dst and port against the rule and deny counters.
Routing both ways.
NAT translation.
Large payload test.
ALG settings.
packet-tracer or diagnose debug flow
Captures both sides.
show conn

Fix

Correct the rule, make the path symmetric, fix NAT, clamp MSS, enable the right ALG.

Stop it coming back

Discover application flows before migrating, stage rules in log only mode, cut over in waves with rollback.

More on this: Firewall platforms, Networking fundamentals

FirewallsUsers lose connectivity for minutes after a firewall or gateway failover

Symptom. Some users cannot reach anything for a few minutes after failover.

Likely causes

Gratuitous ARP from the new active unit not received or ignored.
Upstream router holds the old MAC.

What I check, in order

show ip arp <firewall address> on the upstream router.
Compare MAC with the new active unit.
MAC table on the switches.

Fix

clear ip arp <address>. Ensure virtual MAC is used where the platform supports it.

Stop it coming back

Use virtual MACs on clusters, test failover in a window, keep ARP timers sensible.

More on this: Firewall platforms, Networking fundamentals

FirewallsZone based firewall applied and management broke

Symptom. SSH or routing adjacency lost after zoning an interface.

Likely causes

One interface zoned and another unzoned (traffic between them dropped)
Self zone policy blocking SSH or routing protocols.

What I check, in order

show zone security
show zone-pair security
show policy-map type inspect zone-pair sessions

Fix

Zone every interface and write self zone rules.

Stop it coming back

Plan all zones and self zone rules before applying. Use a revert timer.

More on this: Firewall platforms

FirewallsA proxy firewall rule change has broken an application

What I check, in order

Scope, timing against the change record.
Compare rule before and after (central manager history)
Live audit for the client while the user retries (acat -k | grep <client>, acat_acls -d)
Captures on both interfaces.
Reading. Policy deny means rule match problem (zone, application, missing object, order). Allowed then reset means proxy problem (application defence rejecting a header or command, size limit, timeout, TLS decryption, server now seeing a different source address).

Fix

Backout plan with the change owner if impact is high. Otherwise a narrow fix with security approval, tested and verified with the user.

Stop it coming back

Record in the service desk, known error if a workaround is used, lesson into the CR test plan. Never switch to packet filter as a 2am workaround.

More on this: Firewall platforms

FirewallsServer team sees the firewall address instead of the client

Worth knowing

Expected for many proxy rules. Options X-Forwarded-For or keep source setting, via a security design change.

More on this: Firewall platforms

FirewallsForcepoint engine lost contact with the SMC

What I check, in order

Network path engine to Management Server, certificates, time.

Worth knowing

Behaviour. Keeps enforcing the last installed policy.

Fix

sg-contact-mgmt to re establish trust.

More on this: Firewall platforms

FirewallsUsers cannot reach an application after a firewall change

Symptom. The client's SYN leaves but the connection times out.

Likely causes

Rule order.
Wrong zone pair.
Edited shared object.
NAT change altering which address the policy needs.
Stale sessions hiding the problem.
Rule allows the outbound SYN but the return SYN ACK is blocked or arrives on a different firewall.
Asymmetric routing.

What I check, in order

Link to the change, compare before and after configs.
Scope (who, what, since when, all or some)
Trace: arrives, rule, NAT, leaves, returns.
ASA packet-tracer input inside tcp <src> <sport> <dst> <dport> detailed, show access-list INSIDE_IN | include <dest>, show conn address
Palo Alto test security-policy-match, show session all filter source
FortiGate debug flow
Capture both sides of the firewall.
Check session table for a half open entry.
Check routing in both directions.

Fix

Early rollback via backout plan if impact high. Else fix under change. Verify with users. Incident linked to the change. Problem record if it may recur. Owner refuses rollback means escalate with impact and evidence to the incident manager or service owner.

Stop it coming back

Validate bidirectional flows, not only outbound rules, during every migration.

More on this: Firewall platforms

FirewallsVPN tunnel up, no traffic passes

What I check, in order

ASA show crypto ipsec sa peer x | include local ident|remote ident|#pkts, show nat detail, packet-tracer input inside icmp
FortiGate diagnose vpn tunnel list name, get router info routing-table all
Palo Alto show vpn ipsec-sa tunnel, show vpn flow name
SRX show security ipsec security-associations, show security ipsec statistics
show crypto ipsec sa peer x | include ident|encaps|decaps|errors
ping inside host source Vlan10 (not from the outside interface)
show ip route <remote>
show ip nat translations | include <remote>
Reading. Encaps 0 means routing into tunnel, crypto ACL, or NAT translating before encryption (needs NAT exemption: deny VPN traffic in the NAT ACL). Encaps rising, decaps 0 means far end routing, NAT or policy, or ESP blocked. Both rising means internal routing, inside firewall, host firewall or MTU. Errors or replay drops means reordering by QoS or key issues after rekey.

Worth knowing

Counter reading. Mine flat both ways means routing, NAT exemption, crypto ACL or selectors, inside policy. My encrypt rising, their decrypt flat means ESP or UDP 4500 blocked, NAT between peers. Both rising, my decrypt flat means far side policy, route or NAT. Small works, large fails means MTU.
Proxy IDs. Route based side (Palo Alto) must match a policy based peer's ACL exactly.

More on this: Firewall platforms, Networking fundamentals

FirewallsTraffic works one way only

Likely causes

Return route via another firewall (asymmetric)
NAT not routed back or not reversed.
Callbacks needing their own rule (FTP active)
ARP, host firewall, dual NIC server replying out of the wrong NIC.
Reverse path checks (uRPF, FortiGate RPF (reverse path forwarding), ASA ip verify reverse-path)
HSRP active on one side and active firewall on the other.
One way ping often ICMP policy or host firewall.

What I check, in order

Session with bytes out, zero back.
ASA show conn address x detail, show asp drop | include tcp-not-syn|rpf-violated, capture
SRX show security flow session
FortiGate sniffer.
Palo Alto show session all
IOS show ip route and show ip cef x detail both ways

More on this: Firewall platforms, Networking fundamentals

FirewallsIntermittent drops through a firewall HA pair after a failover

What I check, in order

Timeline and failover reason.
State sync (FortiGate session pickup off by default)
Layer 2: virtual MAC ownership and gratuitous ARP, stale ARP or MAC, MAC flaps (show logging | include MACFLAP, show mac address-table address)
Stability: flapping monitored interface or path monitor, split brain (both active, duplicate IP or MAC flap)
Config, software, routes, licences, upstream links identical.
Group ID clashes (FortiGate, Palo Alto)
ASA MACs change if the primary was replaced without virtual MACs.
ASA show failover, show failover history, show failover | include Stateful|conn|xlate
FortiGate get system ha status, diagnose sys ha checksum cluster
Palo Alto show high-availability state / state-synchronization
SRX show chassis cluster status / statistics

Fix

Fix the cause before failing back. Plan failback in a window.

More on this: Firewall platforms, Networking fundamentals

FirewallsA new rule was added but traffic still hits the default deny

Likely causes

Deny above or rule at the bottom (SRX), wrong Panorama or FortiManager section.
Wrong zone pair because routing decides the egress zone.
Wrong address for that vendor's NAT order.
Object typos, masks, missing group members, wrong port or protocol.
App-ID differs or dependencies missing.
Rule disabled, schedule expired, wrong VDOM (Virtual Domain) or vsys.
Commit, push or install not finished.
ASA ACL not bound with access-group

What I check, in order

Palo Alto test security-policy-match, show jobs all.
SRX show security match-policies, hit-count
ASA packet-tracer, show running-config access-group
FortiGate debug flow
Test matches but logs still deny means real traffic differs (upstream NAT, proxy, port, interface): capture and test with real values.

More on this: Firewall platforms

FirewallsLarge transfers fail through the firewall but small requests work

Likely causes

MTU on a tunnel or overlay with ICMP 3/4 blocked.
IPS (Intrusion Prevention System) or inspection dropping large payloads.
Proxy size limit.
Idle timeout on long transfers.

What I check, in order

Windows ping -f -l 1472 then 1372.
Linux ping -M do -s 1472
IOS ping size 1500 df-bit
ASA show asp drop

Fix

Allow ICMP fragmentation needed. Clamp MSS: IOS ip mtu 1400 and ip tcp adjust-mss 1360. ASA sysopt connection tcpmss 1380 (default 1380). FortiGate per policy set tcp-mss-sender/receiver 1360. SRX set security flow tcp-mss ipsec-vpn mss 1350. Or raise underlay MTU (NSX needs 1600+).

More on this: Firewall platforms, Networking fundamentals

FirewallsUpgrading an HA firewall pair with no outage

What I check, in order

Confirm healthy and in sync.
Upgrade standby, rejoin, verify.
Fail over, test services.
Upgrade the other.
Fail back if required.
Post checks (sessions, VPNs, routing, logs, monitoring)
Update CMDB, version register, service desk.
ASA show failover, verify /sha-512 disk0:/IMAGE.SPA, failover reload-standby, no failover active.
FortiGate get system ha status (secondaries first by default)
Palo Alto request high-availability state suspend, request system software install version, request high-availability state functional
SRX show chassis cluster status then ISSU (In Service Software Upgrade) or node by node per model.

Worth knowing

Prepare. Release notes and upgrade path (intermediate versions), advisories fixed, known issues, resources, licence, off box backups, image hashes, staged images. Change with test plan, rollback, comms. Warn owners about long lived sessions.
Rollback. Fail back to the unit still on the old version, then re image the first.

More on this: Firewall platforms

FirewallsASAv dropping packets under load

What I check, in order

Licence tier (show version, show license all), ASAv (Adaptive Security Appliance Virtual) CPU (show cpu usage), interface drops and overruns, host CPU ready and contention (esxtop), pNIC drops, vNIC on the right port group and VLAN.

More on this: Firewall platforms

FirewallsFortiGate HA checksums do not match

Likely causes

Config drift, often a change made on one unit while the cluster was split.

What I check, in order

diagnose sys ha checksum show on each unit to find the table.

Fix

Find the difference, decide which is correct, resync via change, find how it happened.

More on this: Firewall platforms

FirewallsPalo Alto change made in Panorama but the firewall behaves the old way

What I check, in order

Push job finished without errors (show jobs all), firewall in the edited device group, local rule above the post rules catching the traffic.

More on this: Firewall platforms

FirewallsSRX: match policies shows the permit but there is no session

What I check, in order

Actual arrival interface and zone, screens (show security screen statistics zone untrust), route lookup, flow traceoptions with packet filters, then delete them.

More on this: Firewall platforms

FirewallsSRX inbound NAT address does not answer ARP

Likely causes

NAT address in the interface subnet without security nat proxy-arp

Fix

set security nat proxy-arp interface <if> address <ip>/32.

More on this: Firewall platforms, Networking fundamentals

FirewallsFTP listing hangs after login, or one way audio on SIP

Likely causes

ALG or inspection not applied.
Encrypted control channel hiding ports.
Passive FTP announcing an unreachable address.
Active FTP blocked inbound.
SIP ALG fighting another NAT device.

What I check, in order

ASA show service-policy, show running-config policy-map global_policy
SRX show security alg status
FortiGate show system session-helper

Fix

Tune inspection for that flow, passive mode with fixed port range, correct announced address.

More on this: Firewall platforms

FirewallsApplication broke after last night's firewall change: roll back or fix forward

Likely causes

Rule order, edited shared object group used by other rules, NAT change, undocumented application port.

What I check, in order

Prove the link (deny logs since the change, hit counts on changed rules. packet-tracer or policy match for the app flows; Palo Alto traffic log filter (action eq deny) and (addr.src in x); SRX show security policies hit-count and show | compare rollback 1; FortiGate diagnose debug flow and the policy ID)

Fix

Decision: roll back if the backout is clean and quick, the cause is unclear or impact is high. Fix forward only if the cause is certain, the fix small and approved as emergency. Incident manager and change authority decide. If rollback reopens a security gap the security owner joins (targeted fix or temporary narrow rule with expiry), record who decided.

Stop it coming back

CR recorded as causing an incident, review, application owner test in every firewall change plan.

More on this: Firewall platforms

FirewallsEdge performance drops after enabling inspection

Symptom. Throughput falls or CPU climbs after turning on IPS, URL (Uniform Resource Locator) filtering or TLS proxy.

Likely causes

UTD (Unified Threat Defense) container exceeds the platform's memory and CPU envelope.
TLS proxy overhead.

What I check, in order

show platform hardware qfp active feature utd stats
Platform model.
Traffic mix capture.

Fix

Scale back features, move inspection to SIG (secure internet gateway) or SSE (security service edge), or upgrade the platform.

Stop it coming back

Size UTD per platform before enabling.

More on this: Firewall platforms

FirewallsCold start fails: circular dependencies

Symptom. After a full power loss nothing comes back. Encrypted VMs or arrays will not start.

Likely causes

KMS (Key Management Service) running on the encrypted cluster it serves.
Domain controllers on a cluster whose management needs domain login.
DNS behind a firewall whose management needs DNS.

What I check, in order

Documented start order.
Which foundation services are hosted where.

Fix

Bring up power, OOB, time, DNS, identity, keys, fabric, platforms, workloads in order. Use local break glass and local name resolution.

Stop it coming back

Foundations in a separate management pod ("pod zero"). Cold start test in factory acceptance and periodically in production, timed as evidence.

More on this: Firewall platforms, Networking fundamentals

FirewallsAsymmetric routing through stateful firewalls

Symptom. Sessions fail one way. Firewall drops return traffic.

Likely causes

ECMP or inter DC (data centre) paths send the return through a different firewall (DC1 out, DC2 back)
Firewall cluster without symmetric hashing.

What I check, in order

Trace both directions.
Which firewall saw the SYN and which saw the reply.
PBR hashing configuration.

Fix

Symmetric PBR hashing. Keep the firewall decision local to the pod or site owning the flow. Anycast gateways in both sites. Host route advertisement or GSLB (Global Server Load Balancing) for inbound.

Stop it coming back

Design review question: does the firewall see both directions.

More on this: Firewall platforms

Security and accessISE was down for twenty minutes: what happened, and what should have happened?

Worth knowing

What happens. Existing sessions stay up until reauth. Any port that reauthenticates and any new device fails AAA (Authentication, Authorisation and Accounting). Without critical auth, IoT ports go dark one by one.
What should happen. Two PSNs (Policy Service Node) per region with RADIUS (Remote Authentication Dial In User Service) dead server detection. Critical VLAN or critical auth on edge templates so ports fail into a known limited state (ideally the normal VLAN with a static critical SGT). Reauth timers long enough that a short outage is invisible. Inactivity timers tuned for sleepy devices. PSN capacity for the reconnect storm when ISE returns. Client side backoff.

Stop it coming back

An "ISE unavailable" procedure and a quarterly failover test.

More on this: Security, SD-Access campus fabric, IoT and edge

Security and accessAdmins cannot log in to a device (TACACS+ or ISE)

What I check, in order

ISE live logs (did the request arrive, which policy, why it failed)
show tacacs
debug aaa authentication and debug tacacs briefly

Fix

Correct device group, shell profile or command set. If servers unreachable, local break glass account per procedure.

Stop it coming back

Test AAA changes from a second session before saving. if-authenticated on authorisation.

More on this: Security, AI infrastructure

Security and accessAuthentication fails after a policy node fails, or CoA does not land

Symptom. Wireless or wired auth outage when one PSN dies. Posture or quarantine changes never apply.

Likely causes

RADIUS server group without a working secondary or dead time.
Failover never tested.
PSNs behind a load balancer without source IP persistence.
CoA (Change of Authorisation) from ISE not reaching the right NAD (network access device) through the load balancer path.

What I check, in order

NAD RADIUS server group and dead time.
ISE live logs per PSN.
CoA counters on the NAD.
Load balancer persistence settings.

Fix

Local PSN first, remote second, dead time set. Fix persistence and the CoA return path.

Stop it coming back

Scheduled RADIUS failover tests.

More on this: Security

Security and accessKey manager outage becomes a platform outage

Symptom. After host reboots, encrypted VMs and arrays cannot mount data.

Likely causes

KMS unreachable or lost.
Single site key service.

What I check, in order

KMIP (Key Management Interoperability Protocol) connectivity on TCP 5696.
HSM (Hardware Security Module) cluster state.
Key replication.

Fix

Restore key service from replicated cluster or backup.

Stop it coming back

HSM cluster across sites. Never host KMS on the platforms it serves. Key backups and a vault key store. Tested key service recovery.

More on this: Security

Security and accessClock skew breaks authentication and certificates

Symptom. Kerberos logins fail, certificates appear expired or not yet valid, logs out of order.

Likely causes

NTP source failure or a bad source.
GNSS (Global Navigation Satellite System) jamming or spoofing.
PTP (Precision Time Protocol) path through non PTP aware hops.

What I check, in order

Clock offset and source monitoring.
Number of sources per client.
Kerberos 5 minute skew.

Fix

Correct source, resync.

Stop it coming back

Three or four independent sources, GNSS plus terrestrial UTC (Coordinated Universal Time), holdover oscillators, NTS (Network Time Security) or NTP keys, drift alerts, local time servers per pod.

More on this: Security

Security and accessMass certificate expiry

Symptom. Many management interfaces or APIs (Application Programming Interface) fail on the same day.

Likely causes

Certificates issued together expiring together.
Manual renewal.

What I check, in order

Certificate inventory with expiry dates.

Fix

Emergency reissue from the issuing CA (certificate authority).

Stop it coming back

Automated issuance and renewal (ACME (Automatic Certificate Management Environment), SCEP (Simple Certificate Enrollment Protocol), EST (Enrollment over Secure Transport)), inventory with alerts. Public TLS certificates now last at most 200 days, falling to 100 days in March 2027 and 47 days in March 2029, so renewal has to be automatic.

More on this: Security

Security and accessRansomware encrypts production and replicas

Symptom. Production and replicated copies encrypted. Backup admin accounts compromised.

Likely causes

Replication copies corruption.
Shared credentials between production and backup.

What I check, in order

Last clean immutable copy.
Vault integrity.
Anomaly detection alerts.

Fix

Clean room recovery: identity and core services (DNS, NTP, PKI) first, restore from immutable or vault copy, scan, forensics for root cause and last clean point, promote clean systems.

Stop it coming back

Immutable compliance mode copies, vault with separate identity and keys pulling one way, multi person approval for retention changes, protected time source, regular timed recovery tests.

More on this: Security, Networking fundamentals

Security and accessA log source goes quiet

Symptom. No logs from a device in the SIEM.

Likely causes

Collector failure, UDP syslog loss, misconfiguration after change, time skew making logs look missing.

What I check, in order

Log source coverage report.
Collector health.

Fix

Restore forwarding to redundant collectors.

Stop it coming back

Alert when a source goes quiet (missing logs are an audit finding). Syslog over TLS 6514. Monitoring onboarding as acceptance criterion.

More on this: Security

WirelessWireless client cannot connect, or Wi-Fi is slow

Symptom. One client, one area, one SSID (service set identifier) or everyone? One area: RF (AP down: show ap summary; coverage hole; DFS (dynamic frequency selection) event). One client: follow its path.

Likely causes

Area: AP down, coverage hole, DFS event.
Client timeline: association (band support, minimum data rates), authentication (expired certificate, wrong policy set matched. MAB unknown endpoint), DHCP (VLAN or helper wrong on the policy profile), DNS or traffic (dACL (downloadable ACL) or redirect ACL too tight)

What I check, in order

Four gates. Association (band support, minimum data rates). Authentication (WLC (wireless LAN controller) client detail plus ISE or ClearPass live logs: expired certificate, wrong policy set, unknown MAB endpoint). DHCP (client stuck in IP learn: VLAN or helper wrong on the policy profile). DNS and traffic (dACL too tight, or redirect ACL catching traffic it should not).
Tools. 9800 show wireless client mac-address <mac> detail and RadioActive traces. Aruba show user, show ap debug client-table, Central client timeline.
Scope first.
For an area, show ap summary and RRM (radio resource management) and DFS events.
For a client, show wireless client mac-address <mac> detail and a RadioActive trace on the 9800, ISE or ClearPass live logs for the failing policy.

Worth knowing

Slow. Channel utilisation, retries, same channel interference from over powered APs, a legacy client dragging the cell, 2.4 GHz steered client, Sidekick spectrum for non Wi-Fi interference.

Fix

Correct the failing gate.

Stop it coming back

Fix, verify with the client device, update the Ekahau project if RF changed. Monitor auth failures. Keep the Ekahau project and NAC (Network Access Control) policy in step.

More on this: Wireless and mobile, Security, Networking fundamentals

WirelessWi-Fi is slow

Symptom. Connected but poor throughput.

Likely causes

High channel utilisation.
Retries.
Same channel interference from over powered APs.
One legacy client dragging the cell down.
A client on 2.4 GHz.
Non Wi-Fi interference.

What I check, in order

Channel utilisation and retry rates.
CCI (co channel interference) map.
Client band and data rate.
Spectrum view with Sidekick or CleanAir.

Fix

Reduce power, narrow channels, steer clients, disable low rates, remove interferers.

Stop it coming back

Survey led design, RRM tuned, resurvey after change. Update the Ekahau project after any RF fix.

More on this: Routing protocols

MulticastPIM neighbour missing on a routed link

Symptom. Receivers behind one router get no multicast, and show ip pim neighbor is empty on its uplink.

Likely causes

PIM not enabled on one end.
An ACL or control plane policy dropping PIM hellos to 224.0.0.13.
Sparse mode on one side and dense mode on the other.
The link sits in a different VRF on one side.

What I check, in order

show ip pim interface on both ends.
show ip pim neighbor
show ip mroute
ACLs and CoPP for 224.0.0.13.
The VRF on each end of the link.

Fix

ip pim sparse-mode on both ends of the link, and permit PIM in the ACLs and control plane policy.

Stop it coming back

PIM in the routed link template. Alert when the PIM neighbour count drops.

More on this: Routing protocols

MulticastMulticast floods every port, or stops after a few minutes

Symptom. In a layer 2 only VLAN, a multicast stream floods every port, or works for a few minutes and then stops.

Likely causes

IGMP (Internet Group Management Protocol) snooping is on, but no router or querier sends IGMP queries in the VLAN, so membership entries age out.
No PIM enabled gateway on the VLAN.

What I check, in order

show ip igmp snooping querier
show ip igmp snooping groups vlan <id>
show ip igmp snooping mrouter

Fix

Turn on a snooping querier on one switch in the VLAN with ip igmp snooping querier, or add a PIM enabled SVI as the querier.

Stop it coming back

A querier or a PIM enabled SVI in every multicast VLAN by design.

More on this: Routing protocols

IPv6Rogue IPv6 router advertisements send hosts the wrong way

Symptom. Some hosts lose connectivity or send traffic to an unexpected device, and show an IPv6 default gateway or prefix nobody configured.

Likely causes

A device sending router advertisements, such as a laptop with connection sharing on, a misconfigured router or a consumer device.
No RA (router advertisement) Guard on the access ports.

What I check, in order

The IPv6 default gateway on an affected host (ipconfig /all or ip -6 route)
A capture with the display filter icmpv6.type == 134 to find the sender.
show ipv6 neighbors for its MAC.
show mac address-table address <mac> for its port.

Fix

Remove the device, then apply RA Guard on access ports so only the uplinks may send router advertisements.

Stop it coming back

IPv6 first hop security (RA Guard, DHCPv6 (DHCP for IPv6) Guard and ND (Neighbor Discovery) inspection) in the access template, even where IPv6 is not used yet.

More on this: IP addressing and subnetting, Networking fundamentals

IPv6Some hosts cannot reach a new IPv6 gateway after a router swap

Symptom. After a router is replaced, IPv4 (Internet Protocol version 4) works but some hosts cannot reach the IPv6 gateway for minutes or longer.

Likely causes

Hosts keep a stale neighbour cache entry, or the old link local address stays their default router until its lifetime runs out.
The new router uses a different link local address.
Device tracking on the access switch still binds the address to the old MAC.

What I check, in order

show ipv6 routers and show ipv6 neighbors on the router.
The neighbour cache on a host (netsh interface ipv6 show neighbors or ip -6 neigh)
show device-tracking database on the access switch.

Fix

Give the replacement the same link local gateway address (ipv6 address fe80::1 link-local), or clear the stale host and device tracking entries.

Stop it coming back

A fixed link local gateway address such as fe80::1 in the standard, and an IPv6 test in every router swap plan.

More on this: IP addressing and subnetting

IPv6Dual stack applications start slowly because IPv6 is broken

Symptom. Web pages and applications take seconds to start and then work, while IPv4 tests look normal.

Likely causes

DNS returns AAAA (IPv6 address) records so hosts try IPv6 first, but the IPv6 path is broken: a missing route, a firewall dropping IPv6, or an MTU black hole because ICMPv6 Packet Too Big is blocked. Each connection waits before falling back to IPv4.

What I check, in order

nslookup -type=AAAA <name>
ping -6 and tracert -6 to the server.
curl -6 -v https://<name> against curl -4 -v https://<name>
The firewall IPv6 policy and logs.
ICMPv6 type 2 allowed on the path.

Fix

Repair the IPv6 path, or stop offering IPv6 to those hosts until it works end to end. Never block ICMPv6 Packet Too Big.

Stop it coming back

Monitor IPv6 reachability as well as IPv4, and treat IPv6 as production wherever hosts have it turned on.

More on this: IP addressing and subnetting, Networking fundamentals

OT and IoTAn IoT device on a fabric edge gets no IP address

What I check, in order

show authentication sessions interface Gi1/0/12 details (session? method 802.1X or MAB? authorised? VLAN and SGT returned?)
If no session: right auth template on the port, device linking (show interfaces), PoE (show power inline)
VLAN and pool. Returned VLAN name must exist on this edge for this VN (virtual network). A pool on the wrong site or VN leaves the port in limbo. Check host onboarding.
DHCP. show device-tracking database. Relay on the anycast SVI. Route from the DHCP server back into the VN (route leaking, firewall). DHCP option 82 sub option handling on the server (SD-Access needs it to return the offer to the right edge).
LISP. show lisp instance-id <id> ipv4 database and show lisp session. No registrations means control plane node or its reachability.
Path. Catalyst Centre path trace to the DHCP server and then the broker. SGACL denies via show cts role-based counters.

Worth knowing

ISE Live Logs for the MAC. Rejected (no authorisation rule, unknown endpoint group) or wrong profile. Fix profiling or add MAC to the group. CoA the session.

Stop it coming back

Auth template and pool assignment checks at onboarding. Runbook with this order.

More on this: Operational technology, SD-Access campus fabric, Security

OT and IoTThe device has an IP but cannot reach its broker or head end

Worth knowing

Different VN. Traffic must exit via the border to the fusion firewall. Check the border handoff BGP session, the VRF route on the fusion device, the firewall rule and SGT consumption.
Same VN, different pool. An SGACL. show cts role-based counters for both tags. Confirm the matrix cell in Catalyst Centre. Confirm ISE pushed it (show cts role-based permissions).
Same pool. L2. show lisp instance-id <L2 id> ethernet database, both endpoints registered, destination edge map cache.
Large packets fail, small succeed. MTU. Ping with DF bit and size 1500 between edge loopbacks across the underlay. Find the 1500 byte link.
Intermittent. Reauthentication (session timeout, reauth timer), CoA storms after a profiling change, PoE renegotiation resetting the device.
Broadcast discovery device (BACnet, some PLCs (programmable logic controller)): L2 flooding not enabled on that pool.

More on this: Operational technology, SD-Access campus fabric, Security

OT and IoTSensors on one floor drop off MQTT every few minutes and reconnect

Likely causes

Keep alive too long for a NAT or firewall idle timeout in the path (or too short for a sleepy device), so the session silently drops and LWT (Last Will and Testament) fires.
Wi-Fi roaming or RRM channel changes on that floor.
Duplicate client IDs kicking each other off in a loop.
Port or SSID reauthentication timer.
PoE budget shedding ports under load.
Broker connection limits or a load balancer idle timeout in front of the broker.

What I check, in order

Broker log for client IDs and reason codes.
show authentication sessions for reauth counts.
Firewall session timeout for 8883.
show power inline
Wi-Fi client detail for roams and RSSI.
Capture at the gateway to see if PINGREQ is answered.

Worth knowing

Pattern first. All sensors or one vendor, one floor, time of day, since a change. Client 360 for reauthentications. Broker logs for disconnect reasons.
Related case. Telemetry loss after a new SD-Access site: onboarding and ping fine, broker shows keep alive timeouts, captures show TLS stalling on large publishes. Cause an underlay link left at 1500 instead of 9100. Fix MTU and add an MTU check to fabric acceptance.

Fix

Keep alive (for example 60 s) below the shortest idle timeout in the path. Unique client IDs derived from serial numbers. Exclude IoT SSIDs from aggressive reauth.

Stop it coming back

"check duplicate client IDs" in the runbook.

More on this: Operational technology, SD-Access campus fabric, IoT and edge

OT and IoTA new batch of the same camera model will not onboard; the old ones are fine

Likely causes

New firmware or hardware revision changed the DHCP fingerprint or MAC OUI (Organisationally Unique Identifier).
Profiling no longer matches, authorisation falls to the default quarantine rule.

What I check, in order

ISE Live Logs show the MAC hitting a different policy.

Fix

Update the profiling policy or OUI list, or pre stage MACs in the endpoint group from the delivery note. CoA the sessions.

Stop it coming back

Move the camera class to EAP-TLS with IDevID (Initial Device Identifier). A "new hardware batch" checklist that confirms profile match on a lab port before installers arrive.

More on this: Operational technology, Security, Networking fundamentals

OT and IoTDevices get an IP and ping their gateway, but TLS to the broker fails

Likely causes

Causes in order. No NTP reachable (blocked by SGACL or firewall) so the clock reads 1970 and every certificate is "not yet valid". Certificate expired or device CA chain lacks the broker's intermediate. TLS version or cipher mismatch (device only TLS 1.2 with a cipher the broker disabled). SNI or hostname mismatch after a broker move. MTU (the certificate message is the first big packet, so a 1500 underlay link fails exactly here while pings work). A firewall doing TLS inspection on 8883.

What I check, in order

Device console for the TLS alert.
openssl s_client -connect broker:8883 -servername broker from a host in the same pool.
Ping with DF bit at 1500 across the underlay.
Firewall logs.

Fix

Permit NTP first, always. Fleet wide certificate expiry monitoring. MTU check in fabric acceptance.

More on this: Operational technology, IoT and edge, Networking fundamentals

OT and IoTSensor readings are wrong: jumps, out of order timestamps, impossible values

Worth knowing

Separate transport from sensor problems. Out of order data with QoS 1 and reconnects is duplicates and replays. Deduplicate on (device, sequence number). Use device timestamps not broker arrival. Devices without NTP produce nonsense times.
Drift over weeks. Calibration, self heating, placement near heat or vents.
Impossible values (minus 40 spikes, 65535): I2C read failures or Modbus sentinel values. Check units and scaling (tenths of a degree), signed versus unsigned, byte order (Modbus word swap).

Fix

Validation rules at the gateway (range, rate of change), a data quality flag in the payload, a calibration schedule. Prevention: an "impossible value" decision tree in the runbook.

More on this: Operational technology, IoT and edge

OT and IoTAfter a firmware update half the fleet publishes ten times the data and the WAN saturates

Likely causes

Root cause. Reporting interval changed or debug telemetry left on. The staged rollout should have caught it at 1 percent.

Fix

Contain: QoS that protects command and control and marks bulk telemetry as scavenger. Per client rate limits on the broker. ANC (Adaptive Network Control) quarantine of the noisy class if needed. A canary group and a bandwidth KPI (Key Performance Indicator) in the OTA (over the air) rollout gate. Prevention: aggregate at the gateway so WAN cost does not scale with firmware decisions.

More on this: Operational technology, IoT and edge

OT and IoTA Thread or Zigbee mesh works in one wing; devices in the far wing never join

Likely causes

No chain of mains powered routers in range (battery end devices do not route)
Fire doors, lift shafts and metal ceilings break the chain.
Channel overlap with Wi-Fi on 2.4 GHz.
Commissioning (join window closed, device bound to an old network)
Border routers on different pools without mDNS/SRP reachability make devices disappear from the controller.

Fix

Add a router or a second border router in the far wing. Choose a Thread/Zigbee channel between the Wi-Fi channels in use (for example 15, 20, 25 or 26) and coordinate with the Wi-Fi plan. Reopen commissioning. Put border routers in one pool or bridge discovery.

More on this: Operational technology, Networking fundamentals

OT and IoTCloud dashboards show devices online but no data

Likely causes

Connected but not authorised to publish (IoT policy denies the topic; AWS (Amazon Web Services) drops silently, CloudWatch AWSIotLogsV2 shows PUBLISH denied)
Device publishes to a topic the rule does not match (typo, case, missing site prefix)
Rule action failing (Lambda permission, Kinesis throttling)
Shadow or twin updates on the wrong topic version.
Message over the size limit (128 KB IoT Core, 256 KB IoT Hub) dropped.

Fix

Least privilege policies that still match the real topic tree. An end to end test client that publishes a known message after every change. Alarms on rule errors.

More on this: Operational technology, IoT and edge

OT and IoTMQTT broker or WAN failure on an OT site

Symptom. Plant data stops reaching the enterprise or a broker node fails.

What I check, in order

Broker cluster health, bridge state, store and forward buffer depth, LWT or NDEATH messages received.

Worth knowing

Expected behaviour by design. PLCs keep running because control never depends on the broker. Brokers run as clusters with one per site, so the site keeps working without the WAN. Bridges to the enterprise broker catch up when the link returns. Persistent sessions and QoS 1 queue messages for clients that drop. Edge gateways and historian collectors store and forward. Last Will messages or Sparkplug death certificates tell consumers the data is stale.

Fix

Restore the node or link. Buffers drain.

Stop it coming back

Clustered brokers, site autonomy, buffer sizing longer than the expected outage.

More on this: Operational technology, IoT and edge

OT and IoTConsumers trusting stale plant data

Symptom. Dashboards show old values as if live.

Likely causes

No Last Will or birth and death handling, so a dropped gateway's last retained value looks current.

Fix

Set LWT on every client, use Sparkplug NBIRTH, NDEATH, DBIRTH, DDEATH and STATE, check sequence numbers.

Stop it coming back

Payload schema with timestamp and quality on every value.

Commands by platform, and what each one tells me

Show, test and capture commands for Cisco IOS XE, NX-OS, ASA and FTD, FortiGate, PAN-OS, Junos, Check Point, F5, NSX and ESXi, wireless controllers, SD-WAN and host operating systems. A few rows are tools in a GUI (graphical user interface) rather than commands, and say so. Syntax can change between releases, so I check the release notes before I script anything.

Task	Command or tool	What it tells me
Cisco IOS and IOS XE
Configuration	`ip tcp adjust-mss 1360` `port-channel load-balance src-dst-ip`	MSS clamp, hashing.
IPsec phases	`show crypto isakmp sa` `show crypto ikev2 sa` `show crypto ipsec sa`	Phase 1 state for IKEv1 (Internet Key Exchange version 1) and IKEv2 peers, then the phase 2 SAs with their counters.
Interface overview	`show ip interface brief`	State and IP of every interface.
Port status	`show interfaces status`	VLAN, speed, duplex, connected / notconnect / `err-disabled`
Interface errors	`show interfaces Gi1/0/1`	Input errors, CRC, runts, output drops, resets, duplex, last cleared.
Errors, all ports	`show interfaces counters errors`	FCS (Frame Check Sequence), alignment, collisions per port: find the dirty port fast.
`err-disabled`	`show interfaces status err-disabled` `show errdisable recovery`	Which ports and why (BPDU Guard, port security, UDLD); recovery timers (default 300 s)
Optics	`show interfaces transceiver detail`	Temperature, voltage, Tx and Rx light against thresholds.
Neighbours	`show cdp neighbors detail` `show lldp neighbors detail`	What is plugged in where, platform, management IP, version.
Logs	`show logging \| include UPDOWN`	Flaps, `err-disable`, STP and routing changes.
CPU	`show processes cpu sorted` `show processes cpu history`	Busiest processes; 5 s / 1 min / 5 min; second figure means interrupt (punted traffic)
Punt path	`show platform hardware fed switch active qos queue stats internal cpu policer (3850/9000, varies)` `show policy-map control-plane`	What is hitting the CPU and what CoPP drops.
Route lookup	`show ip route 10.1.1.10`	Longest match, source, AD, metric, next hop.
Forwarding lookup	`show ip cef 10.1.1.10 detail`	What the forwarding plane will actually use.
OSPF	`show ip ospf neighbor` `show ip ospf interface brief` `show ip ospf database`	State (FULL, 2WAY, EXSTART), DR (designated router) role, dead timer; network type and cost; LSAs (Link State Advertisement) by type and area.
BGP	`show ip bgp summary` `show ip bgp neighbors 192.0.2.1 advertised-routes` `... routes` `show ip bgp 10.0.0.0/8`	Peer state or prefix count; what I send; what I accepted after policy; best path reason.
BGP policy refresh	`clear ip bgp 192.0.2.1 soft in`	New inbound policy without dropping the session.
EIGRP (Enhanced Interior Gateway Routing Protocol)	`show ip eigrp neighbors` `show ip eigrp topology`	Neighbours, SRTT (smoothed round trip time), Q count; successors and feasible successors.
STP overview	`show spanning-tree summary`	Mode, VLANs this switch is root for, guard and PortFast defaults.
STP per VLAN	`show spanning-tree vlan 10`	Root ID and priority, local ID, port roles, states, costs.
Guard blocks	`show spanning-tree inconsistentports`	Ports held by Root Guard, Loop Guard or PVID (port VLAN ID) mismatch.
TCN (Topology Change Notification) source	`show spanning-tree detail \| include ieee\|occurr\|from\|is exec`	Topology change count, time since last, port it came from.
EtherChannel	`show etherchannel summary` `show lacp neighbor`	Member flags P bundled, s suspended, D down, I stand alone; partner info.
MAC and ARP	`show mac address-table address 0011.2233.4455` `show ip arp 10.1.1.10`	Port and VLAN of a MAC; MAC behind an IP and entry age.
Trunks	`show interfaces trunk`	Mode, native VLAN, allowed, allowed and active, forwarding (pruned / STP blocked drop out)
Port mode	`show interfaces Gi1/0/48 switchport`	Admin and operational mode, DTP (Dynamic Trunking Protocol) negotiation, access and native VLAN.
L2 security	`show ip dhcp snooping binding` `show ip arp inspection statistics` `show port-security interface x` `show ip verify source`	Binding table; DAI drops; violation mode and count; IPSG state.
Stack	`show switch` `show switch stack-ports summary` `show redundancy`	Members, role, priority, state; ring state; SSO and switchover history.
VSS (6800)	`show switch virtual` `show switch virtual link` `show switch virtual dual-active summary`	Domain, chassis roles, VSL state, dual active detection ready.
HSRP / VRRP (Virtual Router Redundancy Protocol)	`show standby brief` `show vrrp brief` `show track`	Group, priority, preempt, state, VIP; tracked object state.
IP SLA	`show ip sla statistics` `show ip sla configuration`	Probe result, return code, RTT; ties to track and static routes.
VRF	`show vrf` `show ip route vrf RED` `ping vrf RED 10.1.1.10`	VRFs and interfaces; per VRF table; ping in the right table.
QoS	`show policy-map interface Gi0/0/0`	Per class matches, drops, queue depth; shaping and LLQ police drops.
NetFlow	`show flow monitor NAME cache`	Top talkers on the interface.
VPN phase 2	`show crypto ipsec sa peer 203.0.113.1` `show crypto session`	Encaps and decaps counters: one side rising means far end or return route.
DMVPN	`show dmvpn` `show ip nhrp`	Spoke to hub and spoke to spoke state; NHRP (Next Hop Resolution Protocol) mappings.
Conditional debug	`debug condition ip 10.1.1.10 then debug crypto ikev2` `debug ip packet`	Debug scoped to one peer; always `undebug all` after.
Capture (EPC)	`monitor capture CAP interface Gi1/0/1 both` `monitor capture CAP match ipv4 host 10.1.1.10 any` `monitor capture CAP start` `show monitor capture CAP buffer brief` `monitor capture CAP export flash:cap.pcap`	Define, filter, start, read, export pcap (classic IOS on ISR G2 differs)
SPAN	`monitor session 1 source interface Gi1/0/1 both` `monitor session 1 destination interface Gi1/0/24`	Mirror to an analyser such as Wireshark; RSPAN / ERSPAN for remote.
Unsaved changes	`show archive config differences nvram:startup-config system:running-config`	Exact diff saved versus running.
Safe change	`configure terminal revert timer 10 then configure confirm` `reload in 15 then reload cancel`	Auto rollback unless confirmed; scheduled reload as a lockout safety net.
Rollback	`configure replace flash:pre-change.cfg list`	Replace running with saved file, list every changed line.
Save	`copy running-config startup-config (plus SCP off box before and after)`	Save and back up.
Version	`show version`	Release, uptime, last reload reason, config register, INSTALL or BUNDLE mode.
Image hash	`verify /md5 flash:image (or /sha512)`	Compare with the vendor download page.
Upgrade (install mode)	`install add file flash:image activate commit` `show install summary`	IOS XE 16.x+ one step upgrade (varies)
Hardware health (PPM)	`show environment all` `show inventory` `show platform`	PSU, fans, temperature; PIDs (product ID) and serials for RMA and asset record.
Evidence pack	`show tech-support \| redirect flash:tech.txt`	Full state bundle for TAC.
Port status	`show interface status` `show ip route` `show ip bgp neighbors` `show mac address-table` `show spanning-tree` `show spanning-tree mst configuration` `show logging` `show policy-map interface`	Everyday checks.
Extended nodes	`show cdp neighbors` `show rep topology`	Extended Node and REP ring checks.
Underlay MTU	`ping <loopback> df-bit size 1500 (underlay)`	Find the 1500 byte link.
Interface errors	`show interface`	Errors, CRC, late collisions, drops.
Stop debugging	`undebug all`	Stop all debugs.
Reachability	`ping` `traceroute`	Reachability and path.
Routing table	`show ip route vrf <n>`	The routing table for one VRF.
BGP per VRF	`show ip bgp vpnv4 vrf <n> summary`	BGP peers and prefix counts for one VRF.
MST region and instances	`show spanning-tree mst <instance>`	Root, port roles and states for one MST instance.
EtherChannel	`show etherchannel summary (P bundled, s suspended, D down)`	Each bundle with member flags: P bundled, s suspended, D down.
IPsec phase 2	`show crypto ipsec sa (encaps and decaps counters)`	Encrypt and decrypt counters per SA: one side rising alone points at the far end or the return route.
Cisco NX-OS
vPC orphan ports	`show vpc orphan-ports`	Single attached ports, which lose service when their switch reloads.
Configuration	`feature nv overlay` `nv overlay evpn` `fabric forwarding anycast-gateway-mac` `interface nve1` `member vni ... associate-vrf` `evpn multisite border-gateway` `evpn multisite dci-tracking` `fabric-tracking` `system nve infra-vlans` `advertise-pip` `advertise virtual-rmac` `suppress-arp` `vpc orphan-port suspend` `dual-active exclude interface-vlan`	Fabric build and Multi-Site.
OSPF configuration	`interface Ethernet1/1` `ip router ospf 1 area 0.0.0.0`	NX-OS turns OSPF on per interface, not with network statements as on IOS.
Programmable access	`feature nxapi`	Turns on NX-API, so scripts can send CLI commands and get JSON (JavaScript Object Notation) back over HTTPS (Hypertext Transfer Protocol Secure).
Interface overview	`show interface brief`	State, mode, speed, port channel membership.
Interface errors	`show interface ethernet 1/1` `show interface counters`	Rates, CRC, drops, resets.
Logs and audit	`show logging last 50` `show accounting log`	Latest messages; who changed what and when.
CPU and memory	`show system resources` `show processes cpu sort`	Overall load, process count, busiest processes.
Features	`show feature`	Which features are enabled (NX-OS needs `feature x first`)
Route lookup	`show ip route 10.1.1.10 vrf PROD`	Best route in a VRF (default VRF only if no keyword)
OSPF	`show ip ospf neighbors vrf all`	Neighbour state in every VRF.
BGP	`show bgp ipv4 unicast summary vrf all` `show bgp ipv4 unicast neighbors 192.0.2.1 advertised-routes`	Peer state and prefixes; what I send.
STP	`show spanning-tree summary`	Mode, root per VLAN, port types (edge, network, normal)
Port channels	`show port-channel summary`	P up, s suspended, D down.
MAC and ARP	`show mac address-table address 0011.2233.4455` `show ip arp 10.1.1.10 vrf PROD`	"vPC `Peer-Link`" in the port column means learned from the peer.
vPC health	`show vpc brief`	Domain, role, peer and keepalive status, consistency, each vPC state and active VLANs.
vPC consistency	`show vpc consistency-parameters global` `show vpc consistency-parameters interface port-channel 10` `show vpc consistency-parameters vlans`	Type 1 and type 2 side by side; explains a suspended vPC.
vPC keepalive and role	`show vpc peer-keepalive` `show vpc role`	Keepalive source, destination, VRF, timers; configured versus operational role.
VDC (Virtual Device Context)	`show vdc` `show vdc membership` `switchto vdc PROD` `switchback`	VDCs and owned interfaces; move between VDCs from the admin VDC.
FEX	`show fex` `show fex detail`	FEX state, parent ports, pinning.
HSRP	`show hsrp brief`	Group, priority, state, VIP (both vPC peers forward for the vMAC)
VRF `ping`	`ping 10.1.1.10 vrf PROD (keepalive usually vrf management)`	Ping sourced in the right VRF.
VXLAN EVPN	`show nve peers` `show nve vni` `show bgp l2vpn evpn summary` `show l2route evpn mac all`	VTEP peers, VNI state, EVPN sessions, learned MACs.
Control plane capture	`ethanalyzer local interface inband capture-filter "host 10.1.1.10" limit-captured-frames 100`	Supervisor bound traffic only (routing, ARP, keepalive), not transit.
Checkpoint	`checkpoint PRE_CHANGE`	Named rollback point.
Compare and roll back	`show diff rollback-patch checkpoint PRE_CHANGE running-config` `rollback running-config checkpoint PRE_CHANGE`	Diff since checkpoint; restore.
Unsaved changes	`show running-config diff`	Running versus startup before saving.
Version	`show version` `show module`	Release, uptime, reload reason; module state.
Image hash	`show file bootflash:image sha512sum`	Compare with the download page.
Upgrade	`show install all impact nxos bootflash:image` `install all nxos bootflash:image`	Disruptive or ISSU per module, then upgrade (N7K: kickstart plus system images)
Hardware health (PPM)	`show environment`	Power, fans, temperature.
Configuration	`nv overlay evpn` `feature bgp` `feature interface-vlan` `feature vn-segment-vlan-based` `feature nv overlay` `feature fabric forwarding`	Enable VXLAN EVPN features.
Configuration	`fabric forwarding anycast-gateway-mac 2020.0000.00aa`	One anycast gateway MAC for the fabric.
Configuration	`vlan 3001` `vn-segment 50001` `vrf context PROD` `vni 50001` `rd auto` `route-target both auto` `route-target both auto evpn` `interface Vlan3001` `mtu 9216` `vrf member PROD` `ip forward`	Tenant VRF with L3VNI and transit SVI.
Configuration	`vlan 110` `vn-segment 10110` `interface Vlan110` `fabric forwarding mode anycast-gateway`	L2VNI (Layer 2 VNI) with anycast gateway SVI.
Configuration	`interface nve1` `host-reachability protocol bgp` `source-interface loopback1` `member vni 10110` `suppress-arp` `ingress-replication protocol bgp` `member vni 50001 associate-vrf`	VTEP definition.
Configuration	`evpn` `vni 10110 l2` `rd auto` `route-target import auto` `route-target export auto`	EVPN instance for an L2VNI.
Configuration	`router bgp 65101` `neighbor ... remote-as` `update-source loopback0` `address-family l2vpn evpn` `send-community` `send-community extended`	Overlay sessions to RRs.
Configuration	`route-reflector-client (spine template)`	Spine RR.
Configuration	`retain route-target all, next hop unchanged route map, rewrite-evpn-rt-asn`	eBGP overlay variants.
Configuration	`advertise virtual-rmac` `advertise-pip`	vPC pair routing behaviour.
Configuration	`interface Ethernet1/49.101` `encapsulation dot1q 101` `vrf member PROD` `ip address 172.16.101.0/31` `router bgp` `vrf PROD` `neighbor ... remote-as 65500` `bfd` `prefix-list ... in/out` `maximum-prefix 1000`	VRF lite border handoff.
Configuration	`interface Ethernet1/49` `mtu 9216`	Fabric MTU.
Configuration	`no lacp suspend-individual`	Let bare metal ports come up before the bond forms (provisioning)
Drain before upgrades	`system mode maintenance` `no system mode maintenance`	Graceful insertion and removal: routing protocols steer traffic away before the work, and back afterwards.
Zero touch provisioning	`POAP`	PowerOn Auto Provisioning: a new switch fetches its image and configuration over DHCP on the management network. A feature, not a command.
Fabric operations	`Nexus Dashboard Fabric Controller`	Templates, compliance and drift checks, pre change analysis and upgrade workflows for the whole fabric. A tool, not a command.
EVPN routes per VNI	`show bgp l2vpn evpn vni-id N`	EVPN routes for one VNI.
Type 2 routes	`show bgp l2vpn evpn route-type 2 vni-id X`	Did a type 2 route reach the RR.
EVPN MAC and IP routes	`show l2route evpn mac-ip all`	MAC and MAC/IP learnt via EVPN.
Tenant routes	`show ip route vrf X`	Tenant routing table.
ARP suppression	`show ip arp suppression-cache detail`	ARP suppression entries.
Underlay MTU	`ping <loopback> df-bit packet-size 9000 source <loopback>`	Underlay MTU proof.
Interface errors	`show interface`	Bundles and MTU.
Hardening	`feature tacacs+` `aaa authentication login default group TACACS local` `aaa authorization commands` `config-commands` `snmp-server user ... auth sha ... priv aes-128`	Hardening.
vPC health	`show vpc`	vPC role, peer link, keepalive, consistency.
Enable a feature	`feature <name>`	Enable a `feature before configuring it`
Enable a service	`feature <name> (for example ospf, vpc, lacp)`	NX-OS needs each `feature turned on` before it can be configured.
Safe change	`checkpoint` `rollback`	A named rollback point before a change, and the rollback to it.
NVE interface	`show nve interface nve1 detail`	NVE up, source loopback, anycast VTEP, BGP host reachability.
Flood lists	`show nve vni ingress-replication`	Flood list per VNI.
L3VNI mapping	`show nve vrf`	VRF to L3VNI association.
Remote MACs	`show mac address-table vlan 110`	Remote MACs point to nve1 (remote VTEP IP)
Underlay route	`show ip route 10.254.2.14`	Underlay route to remote VTEP with ECMP via every spine.
VTEP MTU test	`ping 10.254.2.14 source 10.254.2.11 df-bit packet-size 9000`	VTEP to VTEP MTU test.
Overlay `ping`	`ping nve` `traceroute nve (feature ngoam)`	Test through the overlay itself.
All EVPN routes	`show bgp l2vpn evpn`	All EVPN routes grouped by RD.
Routes for one VNI	`show bgp l2vpn evpn vni-id 10110`	Routes for one VNI: RD, RT, next hop, router MAC.
MAC and IP bindings	`show l2route evpn mac-ip all detail`	MAC and IP bindings, sequence numbers.
Type 3 routes	`show l2route evpn imet all`	Type 3 routes: flood list membership.
Tenant routes	`show ip route vrf PROD`	/32 host routes and type 5 prefixes via L3VNI.
Local hosts	`show fabric forwarding ip local-host-db vrf PROD`	Locally learned hosts (host mobility manager)
Cisco ASA and FTD
Interfaces	`show interface ip brief` `show interface Gi0/1`	State, IP; errors, overruns, "no buffer" (overloaded unit)
Logs	`show logging \| include 10.1.1.10`	Builds, teardowns, denies with ACL name.
CPU	`show cpu usage`	5 s, 1 min, 5 min.
Routing	`show route`	Needs a route to both source and destination.
Simulate	`packet-tracer input inside tcp 10.1.1.10 12345 192.0.2.10 443 detailed`	Each phase (route, ACL, NAT, inspection, VPN) and final allow or drop with reason.
Capture with trace	`capture CAPIN interface inside trace match ip host 10.1.1.10 host 192.0.2.10` `show capture CAPIN` `show capture CAPIN packet-number 1 trace`	Real packets and how each was processed.
Drops	`clear asp drop` `show asp drop` `capture ASP type asp-drop all`	Drop counters by reason; capture of dropped packets.
Connections	`show conn address 10.1.1.10` `clear conn address 10.1.1.10`	Flags (SYN only, established); clear stale flows after a NAT or route change.
NAT	`show nat detail` `show xlate`	Rules in order with hits; active translations.
ACL hits	`show access-list OUTSIDE_IN`	Hit count per line; never increments means not matching.
Failover	`show failover` `show failover history` `no failover active` `failover reload-standby`	Active and standby state, monitored interfaces and why the last failover happened; hand the active role over; reload the standby.
VPN	`show crypto ikev2 sa` `show crypto ipsec sa peer 203.0.113.1` `show vpn-sessiondb l2l`	Phase 1; encaps and decaps; L2L (LAN to LAN) sessions.
Contexts	`show context` `changeto context CTX1`	Multiple context mode, much like a VDC.
MSS / BGP through ASA	`sysopt connection tcpmss 1380` `tcp-map allowing TCP option 19 for BGP MD5`	MSS clamp; MD5 BGP through the ASA (else sessions fail)
Save and verify	`write memory` `show version` `verify /sha-512 disk0:/image`	Save; release, licence, failover; image hash.
FTD	`show running-config access-group` `packet-tracer ... detailed` `capture CAPIN trace ...` `system support trace` `system support firewall-engine-debug`	Same ASA style tools plus Snort engine traces; policy layers owned by FMC (prefilter, access control, intrusion and file, NAT)
Sessions	`show conn` `packet-tracer input ...`	Session table, policy trace.
Simulate	`packet-tracer`	Trace a synthetic packet through policy, NAT and routing.
FortiGate
Status	`get system status` `get system performance status`	Version, serial, HA and VDOM mode; CPU, memory, sessions.
Port errors	`diagnose hardware deviceinfo nic port1`	Link, speed, errors.
Routing	`get router info routing-table all` `get router info routing-table details 10.1.1.10`	Full table; route for one address.
OSPF and BGP	`get router info ospf neighbor` `get router info bgp summary`	Neighbour state and prefixes.
Debug flow	`diagnose debug flow filter addr 10.1.1.10` `diagnose debug flow show function-name enable` `diagnose debug flow trace start 100` `diagnose debug enable`	Route lookup, policy ID, NAT, drop reason; "policy 0" means implicit deny.
Stop debug	`diagnose debug disable` `diagnose debug reset`	Always run both.
Sniffer	`diagnose sniffer packet any "host 10.1.1.10 and host 192.0.2.10" 4 0 l`	Packets with interface names, no limit, local time.
Sessions	`diagnose sys session filter clear` `... filter src 10.1.1.10` `diagnose sys session list` `diagnose sys session clear`	Policy ID, NAT, state; clear filtered sessions.
Policy lookup	`diagnose firewall iprope lookup 10.1.1.10 12345 192.0.2.10 443 tcp port1`	Which policy a flow would match. Policy Lookup in the GUI does the same; the syntax varies by FortiOS release.
VPN	`get vpn ipsec tunnel summary` `diagnose vpn ike gateway list name P1` `diagnose vpn tunnel list name P1` `diagnose debug application ike -1`	Selectors and counters; phase 1 and 2 detail; full IKE debug.
HA	`get system ha status` `diagnose sys ha checksum cluster`	Members, roles; config checksums must match.
Backup and support	`execute backup config tftp fgt1.conf 10.0.0.5` `execute tac report`	Pre change backup; support bundle; firmware via the upgrade path tool one step at a time.
Packet trace	`diagnose debug flow`	Packet trace.
Palo Alto PAN-OS
Status	`show system info`	Version, model, serial, content versions.
Data plane load	`show running resource-monitor`	DP (data plane) CPU and packet buffers.
Interface	`show interface ethernet1/1`	State, zone, virtual router, errors.
Logs	`show log traffic direction equal backward`	Latest first: rule, app, end reason.
Policy match	`test security-policy-match from trust to untrust source 10.1.1.10 destination 192.0.2.10 destination-port 443 protocol 6`	Rule a flow would hit.
NAT match	`test nat-policy-match (same arguments)`	NAT rule and translation.
Sessions	`show session all filter source 10.1.1.10` `show session id N`	Session IDs; rule, app, NAT, state.
Packet diag	`debug dataplane packet-diag set filter match source 10.1.1.10 destination 192.0.2.10` `set filter on` `set capture stage receive` `firewall` `transmit` `drop file x.pcap` `set capture on` `then capture off and clear all`	Captures at each stage show where a packet dies.
Drop counters	`show counter global filter delta yes packet-filter yes`	Changed counters incl. drop reasons for filtered traffic.
Routing	`show routing route` `show advanced-routing route` `test routing fib-lookup virtual-router default ip 192.0.2.10`	Legacy versus Advanced Routing Engine; egress and next hop.
BGP	`show routing protocol bgp summary (ARE commands differ)`	Peer state.
VPN	`show vpn ike-sa` `show vpn ipsec-sa` `test vpn ike-sa` `ipsec-sa`	SAs; trigger negotiation and watch the system log.
HA	`show high-availability state` `request high-availability state suspend` `functional`	State and sync; force a failover and return.
Config	`show config diff` `save config to backup.xml`	Candidate versus running; named snapshot.
Software and content	`request system software info` `install` `request content upgrade check` `show jobs all`	Releases; content (App-ID, threat) updates; job status.
Policy match	`test security-policy-match`	Rule match test.
Juniper SRX
Interfaces	`show interfaces terse` `show interfaces ge-0/0/0 extensive`	State and addresses; full counters and zone.
Logs and RE (Routing Engine)	`show log messages \| last 50` `show chassis routing-engine`	Latest syslog; RE CPU, memory, uptime.
Routing	`show route 192.0.2.10` `show ospf neighbor` `show bgp summary`	Active route, preference, next hop; neighbour states.
BGP routes	`show route advertising-protocol bgp 192.0.2.1` `show route receive-protocol bgp 192.0.2.1`	Sent and received.
Policy match	`show security match-policies from-zone trust to-zone untrust source-ip 10.1.1.10 destination-ip 192.0.2.10 source-port 1024 destination-port 443 protocol tcp`	Policy a flow would hit.
Policy hits and shadows	`show security policies hit-count` `show security shadow-policies`	Hits per zone pair; rules that can never match.
NAT	`show security nat source rule all` `show security nat destination rule all` `show security nat static rule all`	NAT rules and hits.
Screens	`show security screen statistics zone untrust`	Screen (DoS) drops before policy.
Sessions	`show security flow session source-prefix 10.1.1.10 destination-port 443` `clear security flow session ...`	Live sessions both directions; clear stale.
Flow trace	`set security flow traceoptions file FLOW-TRACE` `... flag basic-datapath` `... packet-filter PF1 source-prefix 10.1.1.10/32 destination-prefix 192.0.2.10/32` `show log FLOW-TRACE`	Zone, policy, NAT, route and drop per packet; delete traceoptions after.
RE capture	`monitor traffic interface ge-0/0/0 matching "host 10.1.1.10"`	Routing Engine traffic only (use flow trace for transit)
VPN	`show security ike security-associations` `show security ipsec security-associations`	Phase 1 and 2 per peer.
Cluster	`show chassis cluster status`	Primary and secondary per redundancy group.
Change safety	`show \| compare` `commit check` `commit confirmed 5` `commit` `rollback 1`	Diff; validate; auto rollback in 5 min unless confirmed; previous config.
Upgrade	`file checksum sha-256 /var/tmp/pkg` `request system software add /var/tmp/pkg reboot`	Hash then install.
NSX Edge and ESXi
List gateways	`get gateways`	Every Tier-0 and Tier-1 service router and distributed router with its VRF ID. NSX 4.2 and later; `get logical-routers is deprecated`.
Enter a gateway	`vrf 1 (exit to leave)`	Prompt changes to `nsx-edge-1`(tier0_sr[1])>.
Routing table	`get route` `get route bgp`	Routes with source, including 100.64.0.0/16 router links and 169.254 transit.
Forwarding table	`get forwarding`	What the datapath actually uses (can differ from RIB during a fault)
BGP	`get bgp neighbor summary`	Peer state, uptime, prefixes: first command when north south fails.
BGP routes	`get bgp neighbor 192.0.2.1 advertised-routes` `get bgp neighbor 192.0.2.1 routes`	Are the segments advertised? Is the default route learned?
BFD	`get bfd-sessions` `get bgp-bfd diagnostics report (varies)`	BFD to peers and between Edge TEPs.
OSPF	`get ospf neighbor`	If the Tier-0 peers with OSPF.
Interfaces	`get interfaces`	Uplink, router link, downlink IPs, MTU, state in the current context.
SR HA	`get high-availability status` `get edge-cluster status`	Active or standby SR; health of each Edge.
Geneve tunnels	`get tunnel-ports` `get tunnel-port uuid stats`	Tunnels to remote TEPs; packet and error counters.
Datapath load	`get dataplane cpu stats`	DPDK core load: busy Edge versus routing fault.
Physical port	`get physical-port fp-eth0 stats`	Drops and errors on the Edge fast path NIC.
Capture	`start capture interface <name> [file <file>] [expression <filter>]`	Capture on an Edge interface. The syntax varies by release.
Manager / control plane	`get managers` `get controllers`	Edge connectivity to the Manager cluster and CCP (Central Control Plane).
Vmk NICs and TEPs	`esxcfg-vmknic -l`	Every vmk with IP, netstack, MTU; TEPs vmk10 and vmk11 on the vxlan netstack.
Physical NICs	`esxcli network nic list`	Vmnic link, speed, duplex, MTU.
VDS	`esxcli network vswitch dvs vmware list`	VDS MTU and uplinks (1600+ for Geneve)
TEP routes	`esxcli network ip route ipv4 list -N vxlan`	TEP gateway on the vxlan netstack.
TEP MTU test 1600	`vmkping ++netstack=vxlan -I vmk10 -d -s 1572 remote-TEP`	DF set `ping proves a 1600 byte path`
TEP MTU test 1700 / 9000	`... -s 1672 / -s 8972`	Same for 1700 or jumbo underlay; fails large but works small means MTU.
NSX datapath	`nsxdp-cli vswitch instance list` `nsxdp-cli bfd sessions list`	NSX switch instances and ports; TEP to TEP BFD state.
VM switch port	`net-stats -l`	Port IDs for each vNIC and vmk.
Capture on uplink	`pktcap-uw --uplink vmnic2 --capture UplinkSndKernel,UplinkRcvKernel -o - \| tcpdump-uw -enr -`	Geneve leaving and arriving on the NIC, live.
Capture at VM port	`pktcap-uw --switchport id --dir 0 -o /tmp/vm1.pcap`	Traffic before encapsulation (dir 0 from VM, 1 to VM)
Capture at DFW	`pktcap-uw --dvfilter name --capture PreDVFilter / PostDVFilter`	Before and after the DFW: proves a DFW drop.
Find DFW filter	`summarize-dvfilter`	Dvfilter names per vNIC (DFW ends `vmware-sfw.2`)
DFW rules on a vNIC	`vsipioctl getrules -f filter` `vsipioctl getaddrsets -f filter`	Rules and address sets actually programmed: did policy reach the host?
NSX CLI	`nsxcli then get firewall status` `get firewall vif-uuid ruleset rules` `get firewall packetlog last 20`	DFW enabled; ruleset per vNIC; recent log hits with rule IDs.
Segment tables	`(nsxcli) get segment vni mac-table` `vtep-table` `arp-table`	What the host learned for a segment from the CCP.
Host to managers	`(nsxcli) get managers` `get controllers`	Host connected to MP (management plane) and CCP.
NSX VIBs (vSphere Installation Bundle)	`esxcli software vib list \| grep -i nsx`	Installed NSX bundles and versions.
Host contention	`esxtop`	Press c for CPU and n for network. `%RDY` above 10 per vCPU or `%CSTP` above 3 points at host contention; `%DRPRX` and `%DRPTX` show vNIC drops.
Edge log	`get log-file syslog follow`	The Edge syslog, live.
Host log	`tail -f /var/log/nsx-syslog.log`	NSX messages on the ESXi host, live.
Support bundle	`System > Support Bundle (NSX Manager)`	One bundle of logs from managers, Edges and hosts for the vendor case.
Windows and Linux hosts
Controller and host checks	`curl -v https://<controller>/login` `arp -a`	Whether the controller answers over TLS from the runner, and whether the host has its gateway in the ARP table.
Path MTU (Windows)	`ping -f -l 1472 192.0.2.10`	DF set, 1472 + 28 = 1500; lower until it passes.
Path MTU (Linux)	`ping -M do -s 1472 192.0.2.10`	Same test on Linux.
Path	`tracert` `traceroute`	Hop by hop path (ICMP rate limiting caveat)
DNS	`nslookup name` `dig name @server +trace`	Resolution and which server answered.
Local IP settings	`ipconfig /all (Windows)` `ip route (Linux)`	No address (169.254), wrong mask or gateway, DHCP binding.
Capture	`tcpdump -ni eth0 host 10.1.1.10 -w host.pcap`	Capture at both ends and compare, to find where packets stop. Open the file in Wireshark.
Image hash	`sha512sum file` `certutil -hashfile file SHA512` `Get-FileHash file -Algorithm SHA512`	Compare with the vendor download page before upgrade.
Wireless controllers
Access points	`show ap summary`	Every access point that has joined: model, IP address, state and uptime.
Aruba client tables	`show user` `show ap debug client-table`	The controller user table with role and VLAN, and the client table on one access point.
Catalyst Centre assurance	`Path trace, Client 360, Assurance issues`	Path trace draws the hops and policy for a flow; Client 360 shows a client onboarding timeline; Assurance lists issues with suggested actions. Tools in the GUI, not commands.
Client detail	`show wireless client mac-address <mac> detail`	Everything about one client: state, access point, SSID, VLAN, policy and roam history.
Per client trace	`RadioActive trace on the client MAC`	A debug trace for one client MAC across the controller processes, safe in production.
Aruba Central	`Central client timeline`	Each step of a client joining, with the step that failed. A cloud tool, not a command.
Catalyst SD-WAN
Overlay first look	`show sdwan omp routes` `show sdwan bfd sessions` `show sdwan control connections`	Overlay first look.
Identity (cert, system IP, site ID, org)	`show sdwan control local-properties`	Certificate, system IP, site ID and organisation name: the identity the controllers check.
Tunnel state and SLA	`show sdwan bfd summary`	How many BFD tunnels are up, and their state.
OMP peering	`show sdwan omp peers`	OMP sessions to the controllers.
Overlay routes	`show sdwan omp routes vpn <n>`	Overlay routes learned for one VPN, with their TLOCs.
TLOCs	`show sdwan omp tlocs`	Transport locators known to this router.
Policy from controller	`show sdwan policy from-vsmart`	The policy the controller pushed to this router.
AAR stats	`show sdwan app-route stats`	Loss, latency and jitter per tunnel, used by application aware routing.
Tunnel stats	`show sdwan tunnel statistics`	Packets and drops per tunnel.
UTD performance	`show platform hardware qfp active feature utd stats`	Load and drops in the security container.
Service side	`show ip route vrf <n>` `show ip bgp vpnv4 vrf <n> summary`	The routing table for one VRF.
Drift check	`show sdwan running-config` `show running-config`	The configuration the controller pushed, to compare with the running configuration.
SD-Access and ISE
ISE tools	`Live Logs, CoA, ANC quarantine`	The result of each authentication, a change of authorisation to force a new one, and quarantine for one endpoint. GUI tools, not commands.
Who is on the port	`show authentication sessions interface Gi1/0/12 details`	Session, method, status, VLAN and SGT.
Link and PoE	`show interfaces` `show power inline`	Link and PoE.
Address learning	`show device-tracking database`	Did the switch learn the IP (SISF)
Control plane sessions	`show lisp session`	Edge to control plane sessions.
What the edge registered	`show lisp instance-id <id> ipv4 database`	What this edge registered.
Registered endpoints	`show lisp instance-id <id> ipv4 server`	Every registered EID (endpoint identifier).
What the edge resolved	`show lisp instance-id <id> ipv4 map-cache` `show lisp instance-id <L2 id> ethernet database`	What this edge resolved; L2 registrations.
Group policy	`show cts environment-data` `show cts role-based permissions` `show cts role-based counters`	SGT names, downloaded policy, which cells hit.
Cisco ACI
Troubleshooting tools	`Health scores, faults, endpoint tracker, atomic counters, moquery`	Where to start in the APIC (Application Policy Infrastructure Controller): health and faults, then where an endpoint has been, packet counts between two points, and object queries from the CLI.
Safe upgrade	`APIC first, then odd and even update groups`	Upgrade the APIC cluster first, then one switch group at a time, after the pre upgrade checks and a configuration export.
Policy models	`Network centric, application centric, vzAny`	Network centric maps one VLAN to one bridge domain and one EPG; application centric splits EPGs by tier and joins them with contracts.
SonicWall, Forcepoint and Sidewinder
Forcepoint logs	`SMC Logs view`	Which rule allowed or discarded, filtered by source, destination, service.
Forcepoint policy	`Edit in SMC, then install on the engines`	Never edit on the engine; failed install means engine on old policy.
Engine status	`sg-status` `sg-version` `sginfo`	Engine state, version; support info bundle.
Cluster	`sg-cluster`	Show or change node state (online, standby, offline)
Diagnostics	`sg-diagnostics` `sg-log-view` `sg-contact-mgmt`	Diagnostic logging; local logs; recontact the SMC (varies)
Capture	`tcpdump (engine shell or SMC traffic capture)`	Packets on the engine.
Sidewinder config	`cf policy q` `cf interface q` `cf zone q` `cf route status`	Rules, interfaces, zones (burbs on older), routing.
Sidewinder roles and audit	`srole (admin role)` `acat -k (kernel audit, denies)` `acat_acls -d / -a`	Switch to admin role; read audit; ACL denies and allows.
Sidewinder policy	`Admin Console or Control Center`	Rule changes and audit, through change control.
SonicWall packet capture	`Packet Monitor, filtered on both hosts`	Each packet with its status: forwarded, dropped or consumed, and the drop reason. A GUI tool.
SonicWall logs	`Log monitor, filtered on the source address`	Which rule allowed or dropped the flow, and the NAT applied. A GUI tool.
Check Point
Drops with reason	`fw ctl zdebug + drop`	Every packet the kernel drops, with the reason and the rule.
Capture	`fw monitor -e "accept host(10.1.1.10);"`	Packets at each inspection point, inbound and outbound, before and after NAT.
Firewall status	`cpstat fw`	Installed policy, install time and packet counters.
Cluster state	`cphaprob state`	ClusterXL members and which one is active.
F5 BIG-IP
Pool members	`tmsh show ltm pool <pool> members`	Member state, monitor results and connections: is the server marked down?
Connections for a client	`tmsh show sys connection cs-client-addr 10.1.1.10`	The client side and server side of each connection, and which pool member it went to.
Capture	`tcpdump -ni 0.0:nnn host 10.1.1.10`	Capture on every VLAN; the nnn flag tags each packet with its virtual server and flow.
Automation tools
Ansible checks	`ansible-playbook --check --diff` `ansible-lint` `yamllint` `ansible-builder` `molecule test`	Preview, lint, EE (execution environment) build, role test.
pyATS (Python Automated Test Systems) state checks	`pyats run job` `device.parse('show ...')` `device.learn('bgp')` `genie diff`	Snapshots and assertions.
Batfish analysis	`bfq.bgpSessionStatus()` `bfq.reachability()` `bfq.traceroute()` `bfq.undefinedReferences()` `bfq.differentialReachability()`	Pre change analysis.
Python push	`netmiko ConnectHandler(...).send_config_set(cmds)` `requests token then GET inventory`	Bulk push, controller API.
MQTT test client	`paho mqtt.Client(...).tls_set(...).will_set(...)` `requests to the Intent API`	Test clients and bulk onboarding.
Terraform	`terraform plan` `terraform apply` `providers CiscoDevNet/aci, CiscoDevNet/mso, CiscoDevNet/dcnm, vSphere, AWS`	Provisioning with reviewable plan.
Ansible collections	`Collections cisco.nxos, cisco.aci, cisco.dcnm, cisco.nd` `Ansible Automation Platform` `Event Driven Ansible`	Configuration and day 2 workflows.
Other tools and ports
MQTT broker	`mosquitto_sub -t '#' -v` `$SYS topics` `MQTT Explorer`	Watch topics and broker health.
TLS to a broker	`openssl s_client -connect broker:8883 -servername broker`	Certificate chain, SNI, cipher.
AWS IoT logs	`CloudWatch AWSIotLogsV2 (PUBLISH denied)` `rule error metrics`	Silent policy drops.
Device bench tools	`UART serial console` `USB 802.15.4 sniffer` `multimeter` `logic analyser` `oscilloscope` `JTAG/SWD with OpenOCD`	Device layer debugging.
Pre and post change checks	`pyATS (before and after state diff), Batfish (model based reachability and segmentation), Nexus Dashboard and ACI pre change analysis`	Pre and post change checks.
Pipeline guardrails	`OPA (Rego), HashiCorp Sentinel, Kyverno, Checkov`	Pipeline guardrails.
Short lived credentials	`HashiCorp Vault, CyberArk`	Short lived credentials.
Bare metal and firmware	`Redfish (DMTF REST on BMC), iDRAC10, PXE, iPXE, UEFI HTTP boot, Metal3 and Ironic, Dell repository tools`	Bare metal and firmware.
Cluster networking	`MetalLB (BGP mode with BFD), Multus, OVN-Kubernetes, CSI, NetworkPolicy, AdminNetworkPolicy, egress IPs, MTV`	Cluster networking and migration.
AI pod checks	`DCGM diagnostics, NCCL tests, UFM, Base Command Manager, GPU Operator, Network Operator, Run:ai, Cable Validation Tool`	AI pod build and checks.
Cable certification	`OLTS (tier 1), OTDR (tier 2), IEC 61300-3-35 inspection`	Cable certification.
Ports to know	`VXLAN 4789` `Geneve 6081` `RoCEv2 4791` `BGP 179` `BFD 3784 (single hop)` `NETCONF 830` `TACACS+ 49` `RADIUS 1812/1813` `LDAP 389, LDAPS 636` `DNS 53` `NTP 123` `DHCP 67/68` `SNMP 161/162` `syslog 514, TLS 6514` `iSCSI 3260` `NVMe/TCP 4420 and discovery 8009` `NFS 2049` `SMB 445` `S3 443` `KMIP 5696` `NX-OS BFD 50 ms x 3` `BGP keepalive 60 s, hold 180 s` `OSPF dead 40 s` `LACP fast 1 s` `ARP 1500 s, MAC 1800 s` `Kerberos 5 min skew` `AD eBGP 20, OSPF 110, IS-IS 115, iBGP 200, static 1`	Numbers to hold.

Projects

Where this method has been tested

I started in hands on support and still troubleshoot in my own lab. The same discipline let me change live, regulated networks without outages.

Home lab

A rack built for troubleshooting

My Cisco and HP (Hewlett Packard) home lab is where I practise troubleshooting, prove fixes and study, on real routers, switches and firewalls across three simulated sites. See the lab topology.

AVEVA · 2020 to 2021

Every site, upgraded remotely

Upgraded every global site remotely during the COVID lockdowns through local enablement, with no delay, and kept over £200M of transactional services at 99.99% uptime through data centre transitions.

XIDA · 2021 to 2024

Migrations with no critical incidents

Planned and ran remote site migrations across regulated environments with no critical incidents.

BNY Mellon · 2018 to 2019

Refresh without an outage

Delivered the EMEA (Europe, Middle East and Africa) network refresh, replacing end of life equipment with minimal impact on global operations and no service outages.

JJ Food Wholesale · 2007 to 2011

Hands on support

Supported LAN, WAN, VoIP (voice over IP), virtualisation and security across the group, including warehouse, cold chain and temperature monitoring systems at the food depots.

Certification

Cisco depth

CCNP (Cisco Certified Network Professional) Enterprise and CCNP Routing and Switching, and I have passed the CCIE (Cisco Certified Internetwork Expert) Routing and Switching and CCIE Security written exams.

Explore

More areas I design and deliver

Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.

Finding the fault, one layer at a time

The same method, whatever has broken

Read before I touch

Explain every symptom

Leave a trail

Six ways into a fault, and when I use each

What I check at each layer

A priority one lands. The first five minutes.

The tools I reach for, and how I read them

Ping, properly

Show before debug

Logs in one place

Flow records

Monitoring tells me where to look

IP SLA probes

Simulate before changing

Tools that prove where a packet was dropped

When I need proof, I capture the packets

Wireshark filters I use most

The fault is technical. The incident is about people.

The evidence pack I send

How I keep people informed

206 faults, with the checks that find them

Commands by platform, and what each one tells me

Where this method has been tested

A rack built for troubleshooting

Every site, upgraded remotely

Migrations with no critical incidents

Refresh without an outage

Hands on support

Cisco depth

More areas I design and deliver

Foundations (this area)

Networks and security

Data centre and cloud

Industry and governance