Each layer has a name for its data
Data at the top, then a segment (TCP) or datagram (UDP), a packet, a frame and finally bits. Using the right word tells a colleague which header I am talking about.
Networking fundamentals
How data really moves from one host to another
Every fault I fix and every design I sign off comes back to these basics. This page walks a packet through the layers and the services every network depends on, with the numbers I use most. You can step through each diagram at your own pace.
Models
Two models, one shared language
The Open Systems Interconnection (OSI) model has seven layers and the TCP/IP model, named after the Transmission Control Protocol (TCP) and Internet Protocol (IP), has four. Real networks run TCP/IP, but I still use the OSI layers every day, because they give everyone the same words for where a problem sits. When I troubleshoot, I work from layer 1 upwards.
Swipe sideways to see the whole diagram.
Layer 1 moves raw bits as electrical, light or radio signals. Many faults I find are simple: a bad patch lead, a dirty fibre connector or the wrong optic. That is why I always start at the bottom.
Layer 2 moves frames between devices on the same local area network (LAN) or virtual LAN (VLAN), using media access control (MAC) addresses. Switches learn which MAC address sits behind which port. Spanning tree and the Link Aggregation Control Protocol (LACP) work here. Loops, VLAN mismatches and duplex problems live here too.
Layer 3 moves packets between networks using IP addresses. Routers choose the path from their routing table, which protocols such as Open Shortest Path First (OSPF) and the Enhanced Interior Gateway Routing Protocol (EIGRP) fill in. Every router a packet crosses lowers its time to live (TTL) by one.
Layer 4 carries data between applications using port numbers. TCP adds reliability with a handshake and acknowledgements. The User Datagram Protocol (UDP) sends without either, which suits voice, the Domain Name System (DNS) and tunnels.
Layer 5 opens, keeps and closes conversations between applications. The TCP/IP model folds it into the application layer, so I rarely treat it on its own.
Layer 6 deals with how data is formatted, compressed and encrypted. TLS (Transport Layer Security) is often placed here, although it does not sit neatly in any one layer.
Layer 7 is where applications talk: web, email, name lookups and management traffic. Layer 7 firewalls and load balancers read this far into the packet to make their decisions.
Devices span layers
A switch works at layer 2 and a router at layer 3. A stateful firewall reads layers 3 and 4, and often layer 7. A load balancer works at layer 4 or layer 7, depending on how it is set up.
Some protocols sit between layers
Multiprotocol Label Switching (MPLS) is often called layer 2.5, because its label sits between the frame and the packet. Virtual Extensible LAN (VXLAN) carries a whole layer 2 frame inside UDP port 4789, which is why it needs a larger maximum transmission unit (MTU).
Encapsulation
Each layer wraps the one above
Before data leaves a host, each layer adds its own header. The receiver removes them in reverse. Knowing exactly which header holds which field is what lets me read a packet capture quickly.
Swipe sideways to see the whole diagram.
The application creates its data, for example a web request. It knows nothing about addresses or links yet.
The transport layer adds a header with port numbers, so the receiver knows which application the data is for, and sequence numbers, so nothing is lost or reordered.
The network layer adds source and destination IP addresses and a time to live. These stay the same from end to end, unless network address translation (NAT) changes them.
The data link layer adds MAC addresses for this hop only, plus a frame check sequence (FCS) to catch corruption. The frame is rebuilt at every router.
The physical layer sends the frame as signals. A standard Ethernet frame is 64 to 1,518 bytes, or 1,522 bytes with a VLAN tag.
The receiver reverses the process, checking and removing one header at a time until the data reaches the right application.
Packet walk
Following one packet, hop by hop
This is the walk I use to explain almost any network fault. It shows what really changes between a personal computer (PC) and a server on another subnet, and what never changes.
Swipe sideways to see the whole diagram.
The PC compares the destination with its own address and mask. The server is on another subnet, so the frame must go to the default gateway, not straight to the server.
The PC needs the gateway’s MAC address. It broadcasts an Address Resolution Protocol (ARP) request, the switch floods it within the VLAN, and Router 1 answers by unicast.
The frame carries the gateway’s MAC but the server’s IP address. The switch learns the PC’s MAC on the way and forwards the frame out of one port only.
Router 1 removes the frame, looks up the longest matching route, lowers the TTL from 64 to 63, recalculates the IP header checksum and builds a fresh frame for the next hop.
The destination is on a connected subnet. Router 2 resolves the server’s MAC with ARP if it is not cached, lowers the TTL again and delivers the frame.
The server sends its reply to its own gateway, and the whole process runs in reverse. Return traffic can take a different path, so stateful firewalls must see both directions.
MAC addresses change at every routed hop and the TTL drops by one per router. IP addresses and ports stay the same unless NAT or a proxy rewrites them.
Same subnet, no gateway
If the destination is inside the PC’s own subnet, the PC uses ARP for the destination itself and never touches the gateway. A wrong mask breaks exactly this decision.
Longest match wins
A router picks the most specific route that matches. A /24 beats a /16, and the default route, 0.0.0.0/0, is only used when nothing else matches.
Captures either side of a router
Both captures show the same IP addresses but different MAC addresses and a TTL one lower. If the IP addresses differ, something on the path is doing NAT.
Ethernet
Frames, MAC addresses and how switches decide
Ethernet carries almost every packet I touch, on copper, fibre and inside data centre overlays. Two things are worth knowing in detail: what is inside a frame, and how a switch decides where to send it.
Swipe sideways to see the whole diagram.
Standard frame. Five fields follow the preamble. The switch only needs the first six bytes, the destination MAC, to decide where the frame goes. The frame check sequence (FCS) at the end lets the receiver throw away corrupted frames.
802.1Q tagged frame. On a trunk, a 4 byte tag sits between the source MAC and the EtherType. It holds a tag protocol identifier (TPID) of 0x8100, a 3 bit priority code point (PCP), a 1 bit drop eligible indicator (DEI) and the 12 bit VLAN ID. VLANs 0 and 4095 are reserved, so 4,094 VLANs are usable.
Swipe sideways to see the whole diagram.
Host A sends to Host B. The switch learns that A lives on port 1. It has never seen B, so it floods the frame out of every other port in the VLAN.
Host B replies. The switch learns B on port 2, and because it already knows A, it sends the reply out of port 1 only.
From now on, frames between A and B use only ports 1 and 2. Hosts C and D never see them. A frame for a host on the same port it arrived on is dropped.
Host C sends an ARP broadcast. Broadcasts and multicasts the switch has not been told about are flooded to every port in the VLAN, so the VLAN is the broadcast domain.
Entries for silent hosts are removed after 300 seconds on Cisco switches by default. A MAC address that keeps moving between ports usually means a loop.
Unicast, broadcast and multicast
A unicast frame goes to one MAC address. A broadcast, ffff.ffff.ffff, goes to every host in the VLAN. A multicast MAC goes to a group, and Internet Group Management Protocol (IGMP) snooping stops it reaching hosts that did not ask for it.
Collision and broadcast domains
A hub is one collision domain, while every switch port is its own. Full duplex links have no collisions at all, so collisions today point to a duplex mismatch. A VLAN is one broadcast domain, and a router ends it.
Ethernet has no TTL
Nothing stops a frame circling a layer 2 loop, so one loop can take down a whole VLAN in seconds. That is why spanning tree exists and why I keep VLANs small. The detail is on the switching page.
To find where a host is plugged in, I take its MAC address from the gateway’s ARP table. Then I follow show mac address-table address from switch to switch, using Cisco Discovery Protocol (CDP) or Link Layer Discovery Protocol (LLDP) neighbours, until it lands on an access port.
Cables and optics
| Medium | Typical use | Reach | What to watch |
|---|---|---|---|
| Cat 6 copper | 1 Gb/s to desks and access points | 100 m | Bend radius and patch lead quality |
| Cat 6A copper | 10 Gb/s, and Power over Ethernet (PoE) for Wi-Fi 6 and 7 access points | 100 m | Heat in large cable bundles carrying power |
| OM4 multimode fibre | 10GBASE-SR, 25 and 100 Gb/s short reach, at 850 nm | about 400 m at 10 Gb/s | Dirty connectors and polarity |
| OS2 single mode fibre | 10GBASE-LR (10 km) and ER (40 km), at 1310 or 1550 nm | up to tens of km | Optic type must match at both ends |
| Direct attach copper (DAC) | Server to top of rack switch | a few metres | Vendor coding on third party cables |
Form factors: small form factor pluggable (SFP) for 1 Gb/s, SFP+ for 10, SFP28 for 25, quad SFP (QSFP28) for 100 and QSFP double density (QSFP-DD) for 400 Gb/s. Digital optical monitoring (DOM) shows transmit and receive power, which is the first thing I read when a fibre link will not come up.
ARP
Finding a MAC address for an IP address
The Address Resolution Protocol (ARP) links layer 3 to layer 2. Every packet that leaves a host on Ethernet needs it, so a broken or poisoned ARP cache stops traffic as surely as a cut cable.
Swipe sideways to see the whole diagram.
ARP. To send to 10.1.10.1, the PC needs its MAC address. It broadcasts a request to the whole VLAN, the owner replies by unicast, and both sides cache the answer. Cisco routers keep ARP entries for 4 hours by default, while most hosts keep them for only a few minutes.
Gratuitous ARP. A device announces its own IP and MAC without being asked. After a first hop redundancy failover, the new active router sends one so every switch learns the virtual MAC on its port. Firewalls, load balancers and moved virtual machines do the same.
Proxy ARP. This PC has the wrong mask, so it thinks 10.1.20.50 is local and asks for it directly. The router, which has a route to it, answers with its own MAC and quietly fixes the mistake. Proxy ARP is on by default on Cisco IOS, and it can hide bad masks for years, so I turn it off with no ip proxy-arp.
ARP spoofing. ARP has no authentication, so an attacker can claim the gateway’s address and sit in the middle. Dynamic ARP Inspection (DAI) checks each reply against the DHCP snooping table of known address and port pairs, and drops forged ones.
Incomplete means no reply
An ARP entry that shows Incomplete means the request went out and nothing answered: the host is down, it is in another VLAN, or layer 2 is broken in between. I check with show ip arp and the switch MAC table.
Stale entries after a failover
If a firewall or server fails over without a gratuitous ARP, upstream routers keep the old MAC until it times out. clear ip arp for that address fixes it at once.
Timers that work together
On Nexus switches ARP entries last 1,500 seconds and MAC entries 1,800, so ARP refreshes before the MAC ages out. Get them the wrong way round and the switch floods unicast traffic it should know.
IPv6 does not use ARP. It uses Neighbour Discovery over ICMPv6 (the Internet Control Message Protocol for IPv6), which I cover on the IP addressing page.
DHCP
How a device gets its address, in four messages
The Dynamic Host Configuration Protocol (DHCP) gives every client its address, mask, gateway and DNS servers. The four messages are often remembered as DORA: discover, offer, request, acknowledge.
Swipe sideways to see the whole diagram.
A new client has no address, so it broadcasts a Discover from 0.0.0.0 to 255.255.255.255. Every device in the VLAN sees it, but only DHCP servers and relays act on it.
Most servers sit in a data centre, not in every VLAN. The router’s
ip helper-addressforwards the broadcast as a unicast and fills in giaddr, the relay’s own address, so the server picks the right scope.The server offers an address with its settings: mask, gateway, DNS servers and lease time. The relay passes the offer back to the client.
The client asks for the offered address, by broadcast, so that any other server that also made an offer knows it was not chosen.
The server confirms the lease. The client sets its address, and many operating systems check that nobody else is using it first.
Halfway through the lease the client renews directly with its server. If that fails, it tries any server at 87.5%. When the lease ends, it gives the address up and starts again.
Options I use most
| Option | Name | What it is for |
|---|---|---|
| 1 | Subnet mask | The mask for the offered address. |
| 3 | Router | The default gateway. |
| 6 | DNS servers | The resolvers the client should use. |
| 15 | Domain name | The DNS suffix added to short names. |
| 42 | Network Time Protocol (NTP) servers | Time sources for the client. |
| 43 | Vendor specific | For example, how Cisco access points find their wireless controller. |
| 51 | Lease time | How long the address is valid. |
| 66 and 67 | Trivial File Transfer Protocol (TFTP) server and boot file | Network boot (Preboot Execution Environment, PXE) and switch zero touch provisioning. |
| 82 | Relay agent information | Added by the switch or relay: which port and which switch the request came from. |
| 150 | TFTP server (Cisco) | Where Cisco IP phones fetch their configuration. |
No helper, no address
A client that gives itself 169.254.x.x (Automatic Private IP Addressing, APIPA) never reached a server. On a new VLAN the first thing I check is the ip helper-address on the gateway interface, then show ip dhcp binding or the server’s scope.
The option 82 trap
DHCP snooping on an access switch adds option 82 with giaddr set to 0.0.0.0. An upstream Cisco relay drops that by default, so clients get nothing. The fix is no ip dhcp snooping information option on the switch, or trusting it on the relay.
Snooping stops rogue servers
DHCP snooping lets only trusted ports, the uplinks to real servers, send offers. It also builds the table of address and port pairs that ARP inspection and IP source guard rely on.
By default the helper forwards eight UDP services: DHCP on 67 and 68, plus Time, TACACS (Terminal Access Controller Access Control System), DNS, TFTP and the two NetBIOS ports. I switch off the ones that are not needed with no ip forward-protocol udp <port>. In the data centre, DHCP mostly matters at build time, for server network boot and switch zero touch provisioning.
DNS
Turning names into addresses, and caching the answer
The Domain Name System (DNS) sits underneath almost every application. When DNS fails, users rarely report a DNS problem. They report that email is down, logins fail or a website will not load.
Swipe sideways to see the whole diagram.
The laptop sends a recursive query to its resolver, usually on UDP port 53: please find the whole answer for me. The resolver checks its cache first.
With nothing cached, the resolver asks a root server. The root does not know the address, but it refers the resolver to the servers for .com.
The .com servers refer the resolver to the name servers that hold the example.com zone. These referrals are iterative queries: each server points to the next.
The authoritative server for example.com returns the A record, 198.51.100.80, with a time to live of 300 seconds.
The resolver passes the answer to the laptop and caches it. The laptop caches it too, for no longer than the TTL.
Anyone who asks for the same name in the next 300 seconds gets the cached answer at once. That is why I lower the TTL a day before I move a service.
Record types
| Record | Holds | Example of use |
|---|---|---|
| A | An IPv4 address | www.example.com to 198.51.100.80 |
| AAAA | An IPv6 address | The same name over IPv6 |
| CNAME (canonical name) | An alias to another name | A friendly name pointing at a load balancer or cloud service |
| MX (mail exchange) | Mail servers, with a preference | Where email for the domain is delivered |
| NS (name server) | The name servers for a zone | Delegating a subdomain |
| SOA (start of authority) | Zone settings and serial number | Zone transfers and negative caching |
| PTR (pointer) | A name for an address (reverse lookup) | Readable names in logs and traceroutes |
| SRV (service) | Service, port, priority and weight | How Windows clients find domain controllers |
| TXT (text) | Free text | Sender Policy Framework (SPF) and domain ownership checks |
UDP first, TCP when needed
Most queries use UDP port 53. Zone transfers and answers too large for UDP use TCP port 53, so blocking TCP 53 breaks things in odd ways. Encrypted DNS uses port 853 (DNS over TLS) or 443 (DNS over the secure Hypertext Transfer Protocol, HTTPS).
Split horizon and anycast
Split horizon gives different answers inside and outside the organisation. Anycast resolvers share one address announced by the Border Gateway Protocol (BGP) from several sites, so clients reach the nearest healthy one. The route must be withdrawn when the DNS service fails, not only the server.
How I test it
nslookup www.example.com or dig @10.9.0.53 www.example.com A shows exactly what a given resolver answers. On Cisco devices, no ip domain lookup stops a mistyped command hanging while the router tries to resolve it.
TCP and UDP
Reliable or fast: TCP and UDP
The transport layer decides how data travels between two applications. TCP checks everything arrives, in order. UDP just sends. Reading the TCP flags in a capture tells me more about a failing application than almost anything else.
Swipe sideways to see the whole diagram.
The client picks a random high source port and an initial sequence number, then sends a synchronise (SYN) segment. It offers its maximum segment size (MSS) and, usually, window scaling and selective acknowledgement.
The server sets the acknowledge (ACK) flag, acknowledges the client’s number plus one and sends its own starting sequence number. A SYN that gets no answer at all usually means a silent drop or a routing problem.
The client acknowledges the server’s number. The connection is now established on both sides. A stateful firewall builds its session from these three packets.
Sequence numbers count bytes, not packets. After 1,460 bytes starting at 1001, the server acknowledges 2461, which means: I have everything up to 2460, send 2461 next.
The receiver advertises how much it can take before the sender must wait for an acknowledgement. One acknowledgement can cover several segments. A window of zero means the receiving application is not keeping up.
The segment at 6841 is lost, but 8301, 9761 and 11221 arrive. Each of them triggers another acknowledgement for the missing byte. After three duplicates, the sender retransmits at once (fast retransmit) and slows down to ease congestion.
Each side sends a finish (FIN) and acknowledges the other’s, so a clean close takes four packets. The side that closes first waits in TIME WAIT. A reset (RST) tears the connection down at once, for example when a port is closed.
How fast can one TCP flow go?
One flow can never send more than one window per round trip, however fast the link is. Change the numbers to see why long distance transfers crawl.
- The most this flow can reach
- about 26.2 Mb/s
- Window needed to fill the link
- about 2,441 KB (bandwidth delay product)
Throughput is at most window ÷ round trip time. Window scaling, agreed in the handshake, lets modern hosts use windows of many megabytes.
TCP flags I read in a capture
| Flag | What it means |
|---|---|
| SYN (synchronise) | Opens a connection and carries the first sequence number. |
| ACK (acknowledge) | The acknowledgement number is valid. Set on every segment after the first. |
| FIN (finish) | The sender has finished sending. |
| RST (reset) | Abort the connection at once. |
| PSH (push) | Hand the data to the application now. |
| URG (urgent) | The urgent pointer is valid. Rarely used. |
| ECE and CWR | ECE is the explicit congestion notification (ECN) echo: the receiver reports congestion on the path. CWR, congestion window reduced, confirms the sender has slowed down. |
TCP and UDP side by side
| TCP | UDP | |
|---|---|---|
| Connection | Three way handshake first | None, it just sends |
| Header | 20 to 60 bytes | 8 bytes |
| Reliability | Acknowledges and resends lost data | None. The application handles loss, if it cares |
| Order | Delivered in order | May arrive out of order |
| Flow and congestion control | Yes, with the window | No |
| Typical uses | Web, email, Secure Shell (SSH), file transfer, databases, BGP | DNS, DHCP, NTP, Simple Network Management Protocol (SNMP), syslog, voice and video, VXLAN, QUIC |
| Firewall behaviour | Session follows the handshake and the FIN or RST | Session kept on a timer, so long silent flows can time out |
Reading TCP when something fails
| What the capture shows | What it usually means |
|---|---|
| SYN with no reply, resent several times | A silent drop: a firewall rule, an access list or a routing problem on the way there or back. |
| SYN answered by RST | Nothing is listening on that port, or a firewall is rejecting rather than dropping. |
| Handshake completes, then the transfer stalls | Often MTU: small packets pass and full size ones are lost. Otherwise, the application itself. |
| Zero window from the receiver | The receiving application cannot keep up. The network is fine. |
| Many retransmissions and duplicate ACKs | Packet loss somewhere on the path: errors, congestion or a bad link. |
Small, battery powered Internet of Things (IoT) devices often avoid the cost of TCP. The Constrained Application Protocol (CoAP) runs over UDP and adds its own simple acknowledgements, while Message Queuing Telemetry Transport (MQTT) runs over TCP and keeps one long lived connection. The IoT page covers that choice.
Ports worth knowing
| Port | Transport | Service | Note |
|---|---|---|---|
| 20 and 21 | TCP | FTP (File Transfer Protocol) | Data and control. Needs an application layer gateway (ALG) through NAT. |
| 22 | TCP | SSH (Secure Shell) and SFTP (SSH File Transfer Protocol) | Secure device and server management. |
| 23 | TCP | Telnet | Plain text. I replace it with SSH. |
| 25 | TCP | SMTP (Simple Mail Transfer Protocol) | Mail between servers. |
| 49 | TCP | TACACS+ (Terminal Access Controller Access Control System Plus) | Device administration, encrypts the whole body. |
| 53 | UDP and TCP | DNS | TCP for zone transfers and large answers. |
| 67 and 68 | UDP | DHCP | Server 67, client 68. |
| 69 | UDP | TFTP (Trivial File Transfer Protocol) | Firmware, phone and boot files. |
| 80 | TCP | HTTP (Hypertext Transfer Protocol) | Plain web traffic. |
| 88 | TCP and UDP | Kerberos | Windows authentication. Not the same as IP protocol 88. |
| 110 | TCP | POP3 (Post Office Protocol version 3) | Mail download. |
| 123 | UDP | NTP (Network Time Protocol) | Time. |
| 143 | TCP | IMAP (Internet Message Access Protocol) | Mail access. |
| 161 and 162 | UDP | SNMP (Simple Network Management Protocol) | Polling on 161, traps on 162. |
| 179 | TCP | BGP (Border Gateway Protocol) | Routing between networks. |
| 389 | TCP and UDP | LDAP (Lightweight Directory Access Protocol) | Directory lookups. |
| 443 | TCP and UDP | HTTPS (HTTP Secure) | TCP for HTTP/1.1 and HTTP/2, UDP for HTTP/3 (QUIC). |
| 445 | TCP | SMB (Server Message Block) | Windows file sharing. |
| 500 | UDP | IKE (Internet Key Exchange) | Key exchange for IP security (IPsec). |
| 514 | UDP | Syslog | Logs. TCP 6514 for syslog over TLS. |
| 546 and 547 | UDP | DHCPv6 (DHCP for IPv6) | Client 546, server 547. |
| 636 | TCP | LDAPS (LDAP Secure) | LDAP over TLS. |
| 853 | TCP | DNS over TLS | Encrypted DNS. |
| 1812 and 1813 | UDP | RADIUS (Remote Authentication Dial In User Service) | Authentication and accounting, used by 802.1X. Encrypts only the password. Older kit uses 1645 and 1646. |
| 1985 | UDP | HSRP (Hot Standby Router Protocol) | Gateway redundancy hellos. |
| 3389 | TCP | RDP (Remote Desktop Protocol) | Windows remote desktop. |
| 3784 and 3785 | UDP | BFD (Bidirectional Forwarding Detection) | Fast failure detection. |
| 4500 | UDP | IPsec NAT traversal | Encapsulating Security Payload (ESP) wrapped in UDP to cross NAT. |
| 4789 | UDP | VXLAN (Virtual Extensible LAN) | Data centre overlay. |
| 4791 | UDP | RoCEv2 (RDMA over Converged Ethernet version 2) | Remote direct memory access (RDMA) between servers, used by artificial intelligence (AI) and storage fabrics. |
| 5060 and 5061 | TCP and UDP | SIP (Session Initiation Protocol) | Voice call set up, 5061 with TLS. |
| 6081 | UDP | Geneve (Generic Network Virtualisation Encapsulation) | VMware NSX overlay. |
Some protocols are not carried by TCP or UDP at all. They have their own IP protocol number instead, which is what a firewall rule or access list has to permit.
| IP protocol number | Protocol |
|---|---|
| 1 | ICMP |
| 6 | TCP |
| 17 | UDP |
| 47 | GRE (Generic Routing Encapsulation) |
| 50 | ESP (IPsec Encapsulating Security Payload) |
| 51 | AH (IPsec Authentication Header) |
| 58 | ICMPv6 |
| 88 | EIGRP |
| 89 | OSPF |
| 112 | VRRP (Virtual Router Redundancy Protocol) |
ICMP
Ping, traceroute and the messages behind them
The Internet Control Message Protocol (ICMP) is how the network reports problems. It is also behind the two tools I use first on almost every fault. Blocking all of it causes more trouble than it prevents.
Swipe sideways to see the whole diagram.
Traceroute sends its first probes with a time to live of 1. The first router lowers it to zero, discards the probe and replies with ICMP time exceeded.
The next probes have a TTL of 2, so they expire one hop further on. Increasing the TTL by one each time maps the path hop by hop.
Some routers rate limit ICMP or do not send it at all. If later hops still answer, stars on one hop do not mean traffic is lost there.
When a probe finally reaches the server, the TTL has not run out. Linux and IOS traceroute use UDP to high ports, so the server replies port unreachable. Windows tracert uses ICMP echo instead.
Ping is simpler: an echo request and an echo reply, with the round trip time. On Cisco IOS an exclamation mark means a reply and a full stop means a timeout.
ICMP messages worth knowing
| Type and code | Meaning | Where I see it |
|---|---|---|
| 0 | Echo reply | A successful ping. |
| 3, code 0 | Network unreachable | No route to the destination network. |
| 3, code 1 | Host unreachable | The last router could not reach the host, often because ARP failed. |
| 3, code 3 | Port unreachable | Nothing listening on that UDP port. The normal end of a Linux traceroute. |
| 3, code 4 | Fragmentation needed and the don’t fragment (DF) bit set | Path MTU discovery. It carries the MTU of the next hop. Never block it. |
| 3, code 13 | Administratively prohibited | An access list or firewall rejected the packet. |
| 5 | Redirect | A router telling a host about a better next hop on the same subnet. |
| 8 | Echo request | Ping. |
| 11, code 0 | Time exceeded in transit | Traceroute, or a routing loop when it appears on its own. |
Cisco IOS ping symbols
| Symbol | Meaning |
|---|---|
| ! | Echo reply received |
| . | Timed out waiting for a reply |
| U | Destination unreachable received |
| M | Could not fragment: the packet was too big with DF set |
| Q | Source quench (destination too busy) |
| ? | Unknown packet type |
| & | Packet lifetime exceeded |
An extended ping on IOS lets me set the source interface, the size and the DF bit, which proves both the return route and the path MTU in one test. IPv6 relies on ICMPv6 even more, for neighbour discovery and for its packet too big messages.
NAT
Translating addresses at the edge
Network address translation (NAT) lets thousands of hosts with private addresses share a few public ones. It also causes some of the strangest faults I see, because it changes the very fields people rely on when they trace a flow.
Swipe sideways to see the whole diagram.
Port address translation (PAT), or NAT overload. Many inside hosts share one public address. The router gives each flow its own source port on the public address, and remembers every mapping in its translation table.
Static NAT. A permanent one to one mapping publishes an inside server on a public address. Because the entry always exists, connections can start from the outside.
The return path. A reply arrives for 203.0.113.10 port 1025. The router finds the matching entry, rewrites the destination back to 10.1.10.21 port 51544, then routes it inside. Without that entry, the reply is dropped.
The four NAT terms on Cisco IOS
| Term | Meaning | In the example |
|---|---|---|
| Inside local | The inside host’s real, private address | 10.1.10.21 |
| Inside global | How the inside host appears to the outside | 203.0.113.10 |
| Outside global | The outside host’s real address | 198.51.100.80 |
| Outside local | How the outside host appears to the inside. Usually the same as outside global | 198.51.100.80 |
Order matters
On IOS, traffic going from inside to outside is routed first, then translated. Traffic coming in is translated first, then routed. Most firewalls have their own order, which changes how rules are written.
Configuration and common faults
The minimum PAT configuration is ip nat inside and ip nat outside on the interfaces, plus ip nat inside source list 10 interface Gi0/1 overload. Faults come from a missing interface command, an access list that does not match, no route back to the inside global address, or PAT running out of ports. I check with show ip nat translations and show ip nat statistics.
NAT is not a firewall
NAT hides addresses, but it is not a security control. It also breaks protocols that carry addresses inside their payload, such as FTP and SIP, unless an application layer gateway fixes them.
MTU
When big packets quietly disappear
The maximum transmission unit (MTU) is the largest packet a link carries without fragmenting it. Standard Ethernet allows 1,500 bytes. Every tunnel and overlay adds headers, and when nobody plans for that, the result is one of the most confusing faults in networking.
Swipe sideways to see the whole diagram.
The sender sets the DF bit and sends a full size packet. A router whose next link is smaller must drop it and report the size that would fit.
The sender remembers the smaller size for that destination and resends. Nothing is fragmented, and the transfer carries on.
If a firewall or access list blocks that ICMP message, the sender keeps sending packets that are too big. Small packets work, so the fault looks like a slow or broken application.
Lowering the TCP maximum segment size (MSS) at the tunnel edge avoids the problem for TCP. Allowing ICMP type 3, code 4 through firewalls fixes it for everything else.
MTU and MSS calculator
Pick the link MTU and every header the packet gains on the way. The calculator shows what is left for the inner packet and the MSS to clamp to.
- Largest inner IP packet
- 1,500 bytes
- TCP MSS to clamp to (IPv4)
- 1,460 bytes
- TCP MSS for IPv6
- 1,440 bytes
- Largest ping that should pass with DF set
- 1,472 bytes of data
Overheads are typical figures for IPv4 outer headers. The IPsec figure is the worst case for the Advanced Encryption Standard (AES) in cipher block chaining mode with Secure Hash Algorithm 1 (SHA 1) integrity. IPsec overhead varies with the cipher and padding, so I leave a margin: on tunnels I usually set ip mtu 1400 and ip tcp adjust-mss 1360.
How I prove the path MTU
| Platform | Command | What it tests |
|---|---|---|
| Cisco IOS | ping 198.51.100.80 size 1500 df-bit | A full 1,500 byte packet that must not fragment |
| Windows | ping -f -l 1472 198.51.100.80 | 1,472 bytes of data plus 28 bytes of headers is 1,500 |
| Linux | ping -M do -s 1472 198.51.100.80 | The same test, with fragmentation forbidden |
| VMware ESXi | vmkping ++netstack=vxlan -d -s 1572 <peer> | The overlay path between hosts, for NSX |
MSS is the TCP view of MTU
The MSS is the MTU minus 40 bytes of IPv4 and TCP headers, so 1,460 on a 1,500 byte link. Each side announces its MSS in the SYN, which is why clamping it there works so well.
When DF is clear
If the DF bit is clear, an IPv4 router splits a packet that is too big into fragments. Each fragment carries the same ID, an offset and a more fragments (MF) flag. Only the destination puts them back together. Lose one fragment and the whole packet is lost, so I design to avoid fragmentation, not to rely on it.
IPv6 never fragments in transit
Only the sender may fragment an IPv6 packet, and every IPv6 link must carry at least 1,280 bytes. ICMPv6 packet too big messages must always be allowed through.
Jumbo frames in the data centre
I run data centre switches at 9,216 bytes and hosts at 9,000, so a VXLAN or Geneve overlay fits with room to spare. Where traffic leaves the fabric at 1,500, I clamp the MSS at the border.
Web page
Opening a web page, end to end
This brings everything on this page together. When someone tells me a website is slow or broken, these are the steps I check, in this order.
Swipe sideways to see the whole diagram.
Before anything crosses the network, the laptop needs an address and a resolver, usually from DHCP, and the browser checks whether it already knows the answer.
The name is turned into an address. If this step fails, users report that the site is down, even though the network may be perfect.
An address outside the laptop’s subnet means the first hop is the default gateway, found with ARP. A wrong gateway or mask breaks everything from here on.
The three way handshake crosses the firewall, which checks its rules, creates a session and applies NAT. A drop here shows as a SYN with no answer.
The encrypted session is set up and the server proves who it is with a certificate. The ClientHello names the site with Server Name Indication (SNI), so one address can serve many sites. An expired certificate or a missing intermediate stops the page here, with a browser warning.
The request and response finally flow. One page can trigger dozens more lookups and connections for scripts, images and fonts.
I can prove or rule out every step with a command or a log, which turns a vague complaint into a short, ordered check list.
How I prove each step
| Step | What I prove | Command or evidence |
|---|---|---|
| Settings | Address, mask, gateway and DNS server are right | ipconfig /all, ip addr, the DHCP binding |
| Name lookup | The name resolves to the expected address | nslookup www.example.com, dig www.example.com |
| Gateway | The laptop has the gateway’s MAC, and the gateway is up | arp -a, show ip arp, show standby brief |
| Path | There is a route there and back | traceroute, show ip route 198.51.100.80 |
| Firewall and NAT | The rule allows it and a session exists | packet-tracer on a Cisco Adaptive Security Appliance (ASA), session table, show ip nat translations |
| TCP and MTU | The handshake completes and full size packets pass | A packet capture, a ping with DF set at full size |
| TLS | The certificate is valid for the name | openssl s_client -connect www.example.com:443 -servername www.example.com |
| HTTP | The server answers quickly and correctly | Browser developer tools, curl -v https://www.example.com, proxy and server logs |
My full method, with worked faults, is on the troubleshooting page.
Delivery
One, all, some or the nearest
A packet can be meant for one host, every host, a group of hosts or the nearest of several identical hosts. Each type behaves differently on a switch and a router, and each one fails in its own way.
Swipe sideways to see the whole diagram.
Unicast. One sender, one receiver. Almost all traffic is unicast.
Broadcast. Every host in the broadcast domain receives it, needed or not. ARP and DHCP rely on it. Routers do not forward it, and IPv6 has no broadcast at all.
Multicast. One stream to a group address in 224.0.0.0/4. Hosts join with IGMP, switches use IGMP snooping to send it only where it was asked for, and routers use Protocol Independent Multicast (PIM) to carry it between subnets. Video, market data and routing protocols use it.
Anycast. The same address is announced from more than one place, and routing delivers each packet to the nearest copy. DNS resolvers, root servers and content delivery networks rely on it.
Time
Every device agreeing on the time
Logs, logins, certificates and some regulations all depend on accurate time. Two protocols deliver it: NTP for most devices, and the Precision Time Protocol (PTP) where microseconds matter.
Swipe sideways to see the whole diagram.
NTP. The Network Time Protocol runs on UDP port 123 and keeps clocks within milliseconds. Each step away from the reference adds one to the stratum. I give every device at least four sources, so one bad clock can be outvoted even when another is unreachable, and I authenticate them with keys or NTS (Network Time Security).
PTP. The Precision Time Protocol (IEEE 1588) reaches sub microsecond accuracy with hardware timestamps. A grandmaster, usually fed by a Global Navigation Satellite System (GNSS) receiver, serves boundary clocks in PTP aware switches, which serve the hosts. Every hop must support it, and asymmetric paths ruin the accuracy.
Why time matters. Without good time, logs cannot be lined up during an incident, Kerberos logins fail and certificates look invalid. Regulated trading systems also break rules that tie their clocks to Coordinated Universal Time (UTC), such as the European Markets in Financial Instruments Directive (MiFID II). Satellite signals can be jammed or spoofed, so critical sites add a terrestrial source and a good holdover clock.
Projects
Where I use the basics
These are the places where I have relied on this material most.
A network I can break on purpose
My three site lab runs MPLS, Dynamic Multipoint VPN (DMVPN), a route out to the real internet, and Windows DNS and DHCP. I use it to rehearse packet walks, captures and faults before I meet them on a live network. See the topology.
Where it started
Cisco based telecoms infrastructure for over 4,000 clients across East Africa and South America, hands on from the first day of my career.
Running a whole group network
Local and wide area networks (LAN and WAN), voice over IP (VoIP), virtualisation and security across the group, including warehouse, cold chain and temperature monitoring systems at the food depots.
A regulated refresh with no outages
The Europe, Middle East and Africa (EMEA) network refresh: audit, high and low level design, and implementation across LAN, WAN, software defined WAN (SD-WAN) and data centre, replacing end of life equipment with no service outages.
Backed by CCNP Enterprise, CCNP Routing and Switching and CCNA, and the CCIE Routing and Switching written exam. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.