I design and fix the links between sites: the circuits underneath, MPLS (Multiprotocol Label Switching) from carriers, and the IPsec, GRE and DMVPN tunnels I build on top. This page shows how each one works, how I keep traffic moving when a link fails, and how I find the fault when it does not.
Every wide area network (WAN) starts with the circuits underneath it. I pick each one for the job it does: the main path, the backup, or the link between data centres. Choose a service to compare them.
Swipe sideways to see the whole diagram.
Leased line and dedicated internet access (DIA). An uncontended circuit, the same speed in both directions, with a service level agreement (SLA) on availability and fix time. I use it for hubs, data centres and central internet breakout. It is not encrypted, so anything sensitive still rides inside IPsec (Internet Protocol Security) or MACsec (media access control security).
Broadband. Full fibre to the premises (FTTP) is fast and cheap, but contended and best effort, with slower fixes. Many lines use PPPoE (Point to Point Protocol over Ethernet), which takes 8 bytes and leaves a 1,492 byte MTU (maximum transmission unit). I treat it as an untrusted transport: always tunnel over it, and never let one broadband line be the only path for a site that matters.
4G and 5G. Ideal on day one before fixed lines arrive, as a failover path and for sites that move. Speed depends on signal and cell load, plans often have data caps, and most services sit behind carrier grade NAT (CGNAT), so nothing can connect in. IPsec over it needs NAT traversal.
Carrier Ethernet. The MEF (Metro Ethernet Forum, now the Mplify Alliance) defines the service types. E-Line is point to point, as an Ethernet private line or a VLAN (virtual local area network) based virtual private line. E-LAN is multipoint, with every site in one Layer 2 network, and E-Tree is hub and spoke. I route over these links rather than stretching Layer 2, to keep failure domains small.
MPLS IP VPN. The carrier runs a Layer 3 VPN (virtual private network): any site reaches any site, with QoS (quality of service) classes and an SLA (service level agreement). It is private but not encrypted. I still use it where voice or regulated traffic needs a carrier SLA, often alongside internet circuits in an SD-WAN (software defined wide area network).
DWDM. Dense wavelength division multiplexing puts many wavelengths on one fibre pair, each its own high speed link. A ROADM (reconfigurable optical add drop multiplexer) adds or drops wavelengths without manual patching. It gives the lowest latency between data centres, where latency is set mostly by fibre distance.
I size the WAN (wide area network) from what crosses it. That means bandwidth at the busiest hour, latency and loss limits for voice and video, the number of sites, how much routing control we need, and what the security policy says about encryption.
Two circuits are only diverse if they use separate ducts, exchanges and points of presence. Two carriers on the same last mile cable fail together, so I ask for route diversity evidence in writing.
MTU, VLAN tagging or QinQ (IEEE 802.1ad), speed and duplex, and whether LACP (Link Aggregation Control Protocol) or CDP (Cisco Discovery Protocol) pass through. A carrier link can stay up while the far end is dead, so I run BFD (Bidirectional Forwarding Detection) or Ethernet OAM (operations, administration and maintenance) over it.
| Service | What it is | Watch out for |
|---|---|---|
| Ethernet private line (EPL) | Point to point on a dedicated port at each end | Single path unless you buy a protected service |
| Ethernet virtual private line (EVPL) | Point to point, several services sharing one port, split by VLAN | VLAN numbers agreed with the carrier, and shared port bandwidth |
| E-LAN or VPLS (virtual private local area network service) | Multipoint Layer 2: every site in one broadcast domain | A loop or broadcast storm at one site reaches every site. I route over it and set storm control |
| MPLS L3VPN (Layer 3 VPN) | Carrier routes your traffic between sites in a private VRF (virtual routing and forwarding table) | Routing policy at the carrier edge, and the AS (autonomous system) number plan |
| DWDM (dense wavelength division multiplexing) wavelength | A dedicated wavelength on carrier or owned fibre | Protected or unprotected, and the real fibre route |
A branch with two links only helps if the router knows when the main one has stopped working. I use IP SLA probes and object tracking on Cisco routers, so failover follows real reachability, not just the light on the port.
Swipe sideways to see the whole diagram.
The branch router sends an ICMP (Internet Control Message Protocol) echo every 10 seconds to a target inside ISP (internet service provider) A, beyond the next hop and sourced from the primary interface. While replies arrive, track object 1 stays up and the default route through ISP A stays in the table.
ISP A fails inside its own network. My interface and the carrier's edge router are still up, so a plain static route would keep sending traffic into a black hole. This is the failure that catches simple designs out.
Probes time out. After the delay I set, 10 seconds here, track 1 goes down. The delay stops one lost probe from moving all the traffic.
The tracked route is withdrawn, so the floating static route with administrative distance 250 is installed. Traffic now leaves through ISP B or the 4G router. The same track object can lower an HSRP (Hot Standby Router Protocol) priority, so the gateway moves with the routing.
When probes succeed again, the track waits 30 seconds before coming back up. A slower return than failover stops a flapping circuit from moving traffic back and forth.
The configuration behind this animation, on Cisco IOS XE
ip sla 1
icmp-echo 198.51.100.1 source-interface GigabitEthernet0/0/0
frequency 10
ip sla schedule 1 life forever start-time now
!
track 1 ip sla 1 reachability
delay down 10 up 30
!
ip route 198.51.100.1 255.255.255.255 203.0.113.1
ip route 0.0.0.0 0.0.0.0 203.0.113.1 track 1
ip route 0.0.0.0 0.0.0.0 192.0.2.1 250
!
! check it
show ip sla statistics 1
show track 1
show ip route 0.0.0.0Pinging the carrier's own router proves very little. I probe something inside the carrier or at the far site, and pin the probe to the primary path with a host route.
If a site has two routers running HSRP (Hot Standby Router Protocol), the same track object lowers the priority on the router that lost its WAN, so users follow the working path.
Where the carrier runs BGP (Border Gateway Protocol) with me, the routes disappear on their own when the far side fails. I add BFD so that happens in under a second instead of after the BGP hold timer.
An EEM (Embedded Event Manager) applet watches for the track going down, then saves the interface, route and probe output with a timestamp. The evidence for the carrier ticket exists before anyone logs in.
MPLS (Multiprotocol Label Switching) forwards on short labels instead of full IP lookups, and lets one carrier network keep thousands of customers apart. My home lab runs this exact design: two P (provider core) routers, five PE (provider edge) routers and two customers in separate VRFs. You can see it on the home lab topology.
Swipe sideways to see the whole diagram.
CE B, the customer edge router in New York, sends its network to PE2, the provider edge, using eBGP, OSPF (Open Shortest Path First) or a static route. PE2 keeps it in VRF ITS (a virtual routing and forwarding table for that customer), so other customers never see it.
PE2 sends the route to PE1 over MP-BGP (multiprotocol BGP). A route distinguisher makes it unique, a route target says which VRFs may import it, and a VPN label (33) tells PE2 which VRF to use later. Large providers use route reflectors instead of a full mesh.
LDP (Label Distribution Protocol) builds labels for each PE loopback, hop by hop. P1 asks for 16 and P2 asks for 17. PE2 advertises implicit null, which tells P2 to pop the label before the last hop: penultimate hop popping (PHP).
A packet from London reaches PE1 as plain IP. PE1 finds the route in VRF ITS and pushes two labels: the VPN label 33 inside, and the transport label 16 on top to reach PE2.
P1 only looks at the top label. It swaps 16 for 17 and forwards. The core routers carry no customer routes at all, which is why one core can carry thousands of customers with overlapping addresses.
P2 pops the transport label, as PE2 asked. PE2 reads the VPN label 33, knows it belongs to VRF ITS, removes it and forwards the original IP packet to CE B.
Swipe sideways to see the whole diagram.
Route distinguisher (RD). Both customers use 10.1.1.0/24. PE1 adds an 8 byte RD, so the routes become 65000:1:10.1.1.0/24 and 65000:2:10.1.1.0/24, two different 12 byte VPNv4 (the VPN address family for IPv4) routes. The RD only makes routes unique. It does not decide where they go.
Route target (RT). Each route carries export route targets as extended communities. A VRF imports any route whose RT is on its import list. ITS imports 65000:1 and GAP imports 65000:2, so the customers stay apart even on shared PE routers. The PEs must send extended communities to each other. IOS adds send-community extended when the VPNv4 neighbour is activated, so I check nobody has removed it.
Shared services. A shared VRF exports 65000:99 and both customer VRFs import it. The shared VRF imports the customers' routes so it can reply. That only works when every imported range is unique. Here both customers use 10.1.1.0/24, so one of them is NATed before it reaches the shared VRF.
| CE to PE routing | How it works | Where I use it |
|---|---|---|
| Static routes | Simple and predictable, but the carrier cannot tell me when its side fails | Small single homed sites, with IP SLA tracking |
| eBGP | Each site has its own AS, or every site shares one private AS | My default. With one shared AS the PE needs as-override or the CE needs allowas-in, or BGP drops the routes as a loop |
| OSPF or EIGRP (Enhanced Interior Gateway Routing Protocol) | The carrier joins my IGP (interior gateway protocol) at each site | Where the customer cannot run BGP. Needs care with domain IDs, the down bit and sham links |
| BFD on the session | Detects a dead neighbour in under a second | Every dynamic CE to PE session, because the BGP hold timer is 180 seconds by default |
The simplest handoff is VRF lite: one sub interface and one routing session per zone between the border leaf and the PE router. The boundary is clean and a firewall fits in easily, but the configuration grows with every VRF. A combined border can instead turn EVPN (Ethernet VPN) routes into L3VPN routes, with fewer links but a more complex shared device.
Whichever handoff I use, each zone maps to the same VRF and L3VPN at every site, DSCP (Differentiated Services Code Point) maps to the MPLS traffic class the same way, and both teams agree the MTU. The data centre team owns up to the border leaf and the WAN team owns the PE. More on the data centre fabrics page.
Segment routing removes LDP: the IGP (interior gateway protocol) carries the labels itself, so there is one protocol fewer to run. It can keep the MPLS data plane or carry the path in IPv6 (Internet Protocol version 6) headers (SRv6), and topology independent loop free alternates give fast reroute without RSVP (Resource Reservation Protocol) tunnels.
IPsec (Internet Protocol Security) protects traffic between two sites over a network I do not trust. IKE (Internet Key Exchange) agrees the keys first, then ESP (Encapsulating Security Payload) protects every packet. Knowing exactly what each message does is what makes VPN faults quick to find.
Swipe sideways to see the whole diagram.
IKEv1. Phase 1 main mode takes six messages: agree the proposal, swap Diffie-Hellman values, then prove identity inside the new encrypted channel. That builds the IKE SA (security association), shown as QM_IDLE on Cisco. Phase 2 quick mode then agrees the proxy IDs and transform set and builds a pair of one way IPsec SAs. Aggressive mode does phase 1 in three messages but exposes the identity, so I do not use it.
IKEv2. Four messages build both the IKE SA and the first child SA. NAT traversal, dead peer detection (DPD) and certificate or EAP (Extensible Authentication Protocol) authentication are built in, and failures come back as clear notifications such as NO_PROPOSAL_CHOSEN or TS_UNACCEPTABLE. It is not backwards compatible, so both ends must agree. I use it for every new tunnel.
NAT traversal (NAT-T). ESP is IP protocol 50 and has no ports, so a NAT device cannot track it. IKE detects the NAT in its first exchange, both sides move to UDP (User Datagram Protocol) 4500 and wrap ESP in UDP, and keepalives hold the NAT entry open. Firewalls in the path must allow UDP 500, UDP 4500 and ESP.
Swipe sideways to see the whole diagram.
A plain packet. A 1,500 byte IPv4 (Internet Protocol version 4) packet carries 20 bytes of IP header, 20 bytes of TCP (Transmission Control Protocol) header and up to 1,460 bytes of data. That 1,460 is the TCP maximum segment size (MSS) on an ordinary Ethernet path.
ESP tunnel mode. The whole original packet is encrypted and wrapped in a new IP header. ESP (Encapsulating Security Payload, IP protocol 50) adds its header, an initialisation vector (IV), padding, a 2 byte trailer and a 16 byte integrity check value (ICV). AH (protocol 51) gives integrity only and breaks with NAT, so I do not use it.
GRE over IPsec. GRE (Generic Routing Encapsulation, protocol 47) adds 4 bytes, or 8 with a tunnel key as DMVPN (Dynamic Multipoint Virtual Private Network) uses, plus the 20 byte delivery header. IPsec then protects it in transport mode, so no second IP header is needed. GRE carries multicast, which is why routing protocols can run over it.
With NAT traversal. A UDP header on port 4500 sits in front of ESP so NAT devices can track the flow. It costs another 8 bytes, which matters when the path is already close to its limit.
| Setting | What I use today | What I remove |
|---|---|---|
| IKE version | IKEv2 (Internet Key Exchange version 2) | IKEv1 (Internet Key Exchange version 1) aggressive mode |
| Encryption | AES-256-GCM, which also gives integrity | DES (Data Encryption Standard) and 3DES |
| Integrity and PRF (pseudorandom function) | SHA-256 or stronger where GCM (Galois/Counter Mode) is not used | MD5 (Message Digest 5) and SHA-1 |
| Diffie-Hellman | Groups 19, 20 or 21 (elliptic curve), at least group 14 | Groups 1, 2 and 5 |
| Authentication | Certificates at scale, long random pre shared keys for a few tunnels | Short or reused pre shared keys |
| Perfect forward secrecy | On, with the same strong group | Off, or a weak group on phase 2 only |
| Lifetimes | Cisco defaults: 86,400 seconds for the IKE SA, 3,600 for IPsec SAs | Mismatched lifetimes between vendors |
A policy based VPN uses a crypto ACL (access control list) and a crypto map: one pair of SAs per subnet pair, and no routing protocols. A route based VPN uses a virtual tunnel interface (VTI): anything routed into it is encrypted, OSPF or BGP can run over it, and it shows up in monitoring like any link. I build route based tunnels.
When a route based end meets a policy based end, the proxy IDs must match exactly, or phase 2 fails with TS_UNACCEPTABLE. I set specific proxy IDs on the route based side, for example on a Palo Alto firewall, and keep a record of every pair.
DPD (dead peer detection) sends a check when no traffic has arrived for a while, so a dead peer is noticed and the SAs are cleared. Without it, a tunnel can look up for hours while nothing passes.
A route based IKEv2 tunnel on Cisco IOS XE. Keys and addresses are examples.
crypto ikev2 proposal PROP-GCM
encryption aes-gcm-256
prf sha384
group 20
crypto ikev2 policy POL-1
proposal PROP-GCM
crypto ikev2 keyring KR-DC
peer DC
address 203.0.113.10
pre-shared-key <long random key>
crypto ikev2 profile IKEV2-DC
match identity remote address 203.0.113.10 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local KR-DC
dpd 10 3 on-demand
crypto ipsec transform-set TS-GCM esp-gcm 256
mode tunnel
crypto ipsec profile IPSEC-DC
set transform-set TS-GCM
set pfs group20
set ikev2-profile IKEV2-DC
!
interface Tunnel10
ip address 172.16.10.1 255.255.255.252
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/0
tunnel destination 203.0.113.10
tunnel mode ipsec ipv4
tunnel protection ipsec profile IPSEC-DC| Link | What I use | Why |
|---|---|---|
| Any to any MPLS L3VPN | GETVPN (Group Encrypted Transport VPN) with a key server | Encrypts with group keys and keeps the original IP header, so carrier routing and QoS still work |
| Dark fibre, owned DWDM or carrier Ethernet between data centres | MACsec (IEEE 802.1AE), or WAN MACsec on carrier Ethernet | Line rate AES (Advanced Encryption Standard) with GCM in hardware, at most 32 extra bytes per frame, decrypted at each hop |
| Routed WAN, internet and cloud backup paths | IPsec | Works over any IP path and meets the PCI DSS (Payment Card Industry Data Security Standard) rule for strong cryptography over public networks, but costs more per Gbps and adds the bytes counted in the MTU section |
| Between data centre fabrics over any routed network | CloudSec between fabric border gateways or ACI (Application Centric Infrastructure) spines | Encrypts the VXLAN (Virtual Extensible Local Area Network) overlay itself, so the transport in between can be anything |
| Application to application | TLS (Transport Layer Security) 1.2 or 1.3, and mutual TLS between services | Still needed end to end, because link encryption stops at every device |
| Every option | Keys from a key management system, rotated on a schedule | Encryption also hides traffic from intrusion detection, so I plan where inspection happens |
GRE gives me a tunnel that can carry routing protocols. DMVPN adds multipoint GRE, NHRP (Next Hop Resolution Protocol) and IPsec, so a hub with one configuration serves every spoke and spokes talk directly when they need to. My home lab uses DMVPN as the backup path between its three sites.
Swipe sideways to see the whole diagram.
Each spoke registers its tunnel address and its real internet address, the NBMA (non broadcast multiple access) address, with the hub, which is the NHRP (Next Hop Resolution Protocol) server. The hub needs no per spoke configuration, which is why DMVPN (Dynamic Multipoint VPN) scales to hundreds of sites.
In phase 1, and at the start of every new flow in phase 3, spoke to spoke traffic goes through the hub. It works, but the hub carries everything and adds latency.
In phase 3 the hub notices traffic going out the same tunnel it came in on, and sends an NHRP redirect to spoke 1: there is a better path. This needs ip nhrp redirect on the hub.
Spoke 1 sends a resolution request, which the hub forwards to spoke 2. Spoke 2 replies straight to spoke 1 with its real address. This needs ip nhrp shortcut on the spokes.
Spoke 1 builds an IPsec protected GRE tunnel straight to spoke 2 and installs a shortcut route. The hub can now advertise just a summary or a default route, which phase 2 cannot do.
| Phase | Spoke to spoke traffic | Routing at the hub | Notes |
|---|---|---|---|
| 1 | Always through the hub | Can summarise | Simple hub and spoke. With EIGRP, turn off split horizon on the hub |
| 2 | Direct, once resolved | Must keep each spoke's next hop: no summaries | EIGRP needs no ip split-horizon eigrp and no ip next-hop-self eigrp on the hub. OSPF needs broadcast type with the hub as DR (designated router) and spokes at priority 0 |
| 3 | First packets via the hub, then direct | Can summarise to a default | Redirect on the hub, shortcut on the spokes. OSPF point to multipoint works. My default for new builds |
GRE adds 24 bytes and carries multicast, so OSPF and EIGRP can run over it. It does not encrypt anything. Keepalives such as keepalive 10 3 bring the tunnel down when the far end stops answering.
If the route to the tunnel destination is learned through the tunnel itself, the tunnel flaps with %TUN-5-RECURDOWN. I keep the destination reachable only through the underlay, with a static route or a filter.
Every DMVPN router in a cloud needs the same tunnel key, and the same NHRP authentication string if one is set. A mismatch looks like a spoke that never registers. The NHRP network ID only matters on the router itself, but I keep it the same everywhere so configurations read alike.
DMVPN phase 3 on Cisco IOS XE, hub and spoke
! hub
interface Tunnel0
ip address 10.0.0.1 255.255.255.0
ip nhrp network-id 100
ip nhrp map multicast dynamic
ip nhrp redirect
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 100
tunnel protection ipsec profile DMVPN
! spoke
interface Tunnel0
ip address 10.0.0.11 255.255.255.0
ip nhrp network-id 100
ip nhrp nhs 10.0.0.1 nbma 203.0.113.1 multicast
ip nhrp shortcut
ip mtu 1400
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 100
tunnel protection ipsec profile DMVPN
! check it
show dmvpn
show ip nhrp
show crypto sessionThe maximum transmission unit (MTU) is the largest packet a link carries without splitting it. Tunnels eat into it, and getting it wrong causes one of the most confusing faults there is. Try the calculator with the layers you plan to stack.
Pick the path and the layers you stack on it. The numbers are worst case for IPv4, including the largest padding each cipher can add.
| Overhead, worst case | 65 bytes |
|---|---|
| Largest inner packet | 1,435 bytes |
| Largest TCP segment (MSS) | 1,395 bytes |
| My standard ip mtu 1400 and MSS 1360 | Fits, with 35 bytes to spare |
This is a DMVPN style tunnel: GRE with a key, protected by IPsec in transport mode.
Every tunnel adds headers, so a full size packet no longer fits the path. If the do not fragment bit is set, the router drops it and should send back ICMP type 3 code 4. When a firewall blocks that message, path MTU discovery (PMTUD) fails silently: pings and logins work, file copies and large web pages stall.
On each tunnel I set ip mtu and ip tcp adjust-mss so TCP never sends a segment that is too big, and I allow ICMP type 3 code 4 through firewalls. MSS clamping only helps TCP, so UDP applications need the right MTU as well.
A ping with the do not fragment bit at decreasing sizes finds the real limit: ping 10.2.1.10 source Vlan10 size 1400 df-bit on Cisco, or ping -f -l 1372 on Windows, where 1,372 plus 28 bytes of headers makes 1,400.
Quality of service (QoS) cannot add bandwidth. It decides who goes first and who waits when a link is congested, which on a WAN is most busy afternoons. I design it as a small set of classes, applied the same way at every site.
Swipe sideways to see the whole diagram.
At the edge of my network I classify traffic and set a DSCP (Differentiated Services Code Point) value. I use EF (expedited forwarding) 46 for voice, AF41 (assured forwarding, class 4) for video, AF21 (assured forwarding, class 2) for business applications and 0 for the rest. Markings from user devices are not trusted unless I choose to.
Class based weighted fair queuing (CBWFQ) gives each class its own queue and a guaranteed share of the link when it is congested. When the link is not busy, nothing is held back.
Low latency queuing (LLQ) adds a strict priority queue for voice. It is always served first, so jitter stays low. During congestion it is policed to its configured rate, so voice can never starve everything else. I keep all priority classes to about a third of the link.
The other classes get their guaranteed shares. In the default class, weighted random early detection (WRED) drops a few packets before the queue is full, so TCP flows slow down gradually instead of all at once.
The carrier port runs at 1 Gb/s but the contract is 200 Mb/s. Without a shaper the carrier polices my traffic and drops it blindly. Shaping to 200 Mb/s, with the queues nested inside the shaper (hierarchical QoS), means I choose what waits and what drops.
Swipe sideways to see the whole diagram.
Policing. Traffic above the rate is dropped or re marked immediately, in either direction. TCP sees the loss and backs off hard, so throughput saws up and down. Carriers police at their edge, which is why I never send them more than the contract.
Shaping. Traffic above the rate waits in a buffer and is sent when there is room. It adds a little delay but avoids loss, and it only works outbound. I shape at the WAN edge to the contracted rate, or a little under it if the carrier counts Layer 2 overhead differently, then queue inside the shaper.
| Class | DSCP | Queue | Typical share |
|---|---|---|---|
| Voice | EF (46) | Priority queue, policed | About 10% |
| Interactive video | AF41 (34) | Bandwidth guarantee | About 20 to 30% |
| Routing and network control | CS6 (48) | Small guarantee so keepalives survive congestion | About 5% |
| Business applications | AF21 (18) or AF31 (26) | Bandwidth guarantee | About 25 to 30% |
| Default | 0 | Fair queue with WRED | What is left, at least 25% |
| Scavenger, such as backups | CS1 (8) | Minimum share, dropped first | About 1 to 5% |
Hierarchical QoS on Cisco IOS XE: queue inside a 200 Mb/s shaper
policy-map CHILD-WAN
class VOICE
priority percent 10
class VIDEO
bandwidth percent 30
class BUSINESS
bandwidth percent 30
class class-default
bandwidth percent 25
fair-queue
random-detect dscp-based
policy-map PARENT-SHAPE
class class-default
shape average 200000000
service-policy CHILD-WAN
interface GigabitEthernet0/0/0
service-policy output PARENT-SHAPE
! check it
show policy-map interface GigabitEthernet0/0/0Remote access VPN (virtual private network) connects a user, not a site. The design choice is how much of their traffic goes through the company, and how tightly access is checked.
Swipe sideways to see the whole diagram.
Full tunnel. Every packet from the laptop goes to the headend, so the same firewall policy and logging apply at home as in the office. The cost is bandwidth and latency at the headend, and SaaS (software as a service) traffic takes the long way round.
Split tunnel. Only corporate address ranges use the tunnel. Internet and trusted SaaS traffic, such as Microsoft 365, go direct. It saves headend capacity, but the laptop needs its own protection for the traffic that skips the tunnel.
Zero trust network access (ZTNA). A broker checks the user and the device on every request, then connects them to one application, never to a whole network. Combined with cloud security for internet traffic, this is the SASE (secure access service edge) model. More on the security page.
Remote access uses multi factor authentication (MFA), usually through SAML (Security Assertion Markup Language) single sign on to the identity provider, and a certificate on managed devices so a stolen password alone is not enough.
Posture checks confirm the operating system is patched and protection is running before access is given. A device that fails gets a remediation network, not the corporate one.
Cisco Secure Client (formerly AnyConnect) uses TLS with DTLS (Datagram Transport Layer Security) for the data, or IKEv2. I size the headend for licences, concurrent sessions and throughput, and keep a second headend in another site.
A tunnel fails at one specific stage, and each stage leaves its own evidence. I work through them in order, from the underlay up, and never change two things at once. Choose a symptom to see where it fails and what I run. My general method is on the troubleshooting page, and vendor firewall detail is on the firewall platforms page.
Swipe sideways to see the whole diagram.
No SA at all. Nothing has tried to build the tunnel. Either no interesting traffic has arrived, the crypto map or tunnel protection is not applied, or there is no route to the peer. I check underlay reachability from the outside interface first, then send traffic from an inside host to trigger the tunnel.
Phase 1 never completes. An IKEv2 SA stuck in IN-NEG, or MM_NO_STATE on IKEv1, with retransmissions means the peer is not answering: UDP 500 blocked, the wrong peer address, or the peer not configured. atts are not acceptable or NO_PROPOSAL_CHOSEN is a proposal mismatch. Stuck at MM_KEY_EXCH or AUTHENTICATION_FAILED points to the pre shared key, identity or certificate.
Only fails behind NAT. The first exchange on UDP 500 works, but once NAT is detected both sides move to UDP 4500, which a firewall is blocking. I allow UDP 4500, check NAT traversal is on, or move the tunnel to IKEv2 where it is built in.
Phase 1 up, phase 2 fails. Usually the proxy IDs or crypto ACLs are not exact mirrors, the most common VPN fault of all. Otherwise the transform sets differ or perfect forward secrecy is on at one end only. I compare both configurations line by line on a call with the far end.
One way only. Encaps rising with no decaps means my side is sending but nothing returns. The far end may be missing a route back, its NAT may be translating replies before encryption, or ESP may be blocked on the way back. Encaps at zero means my own traffic is not reaching the tunnel, often because NAT needs an exemption for VPN traffic.
Large transfers hang. The tunnel overhead pushes full size packets over the path MTU, and ICMP type 3 code 4 is blocked, so path MTU discovery fails. I set ip mtu 1400 and ip tcp adjust-mss 1360 on the tunnel and allow the ICMP message through.
| Symptom | What I check | Usual cause | Fix |
|---|---|---|---|
| A site has lost its MPLS routes | One site or many? Carrier notices and my change calendar. Interface errors, then show ip bgp summary and the routes received and advertised | Carrier fault, BGP down, an inbound filter or maximum prefix limit, or a loop rejection when every site shares one AS | Carrier ticket with circuit ID, timestamps, counters and BGP state. allowas-in or as-override for a shared AS |
| A remote site has lost its WAN | Reach the router over the backup or out of band path. show interfaces: down/down is physical, up/down is Layer 2, up/up with no routes is routing. show track | Carrier fault, a dead router, or a failover that never triggered | Carrier ticket, local hands to check the network terminating equipment, a spare router with the saved configuration |
| An application is slow at one site | Which app, which users, since when. traceroute for a silent move to the backup path, show policy-map interface for class drops, a capture for retransmissions | Running on the backup path, the wrong QoS class, MTU, or latency limiting the TCP window | Restore the primary, fix the classification, move bulk jobs, or add capacity with data to back it |
| Voice breaks up at busy times | Drops in the priority class, or voice landing in the default class | Wrong marking at the trust boundary, a priority class that is too small, or no shaper | Mark at the edge, size the priority class, shape below the contract |
| A GRE tunnel keeps flapping | show ip route for the tunnel destination, and %TUN-5-RECURDOWN in the log | The destination is learned through the tunnel itself | Reach the destination only through the underlay |
| A DMVPN spoke never registers | show dmvpn, show ip nhrp nhs, then the IPsec state | A tunnel key or NHRP authentication mismatch, the wrong NHS (next hop server) or NBMA address, or IPsec failing underneath | Align the values, then fix IPsec phase by phase |
| What I want to know | Cisco IOS XE | Other platforms |
|---|---|---|
| Is phase 1 up? | show crypto ikev2 sa or show crypto isakmp sa | ASA (Adaptive Security Appliance) show crypto ikev2 sa, FortiGate diagnose vpn ike gateway list, Palo Alto show vpn ike-sa |
| Is traffic being encrypted both ways? | show crypto ipsec sa peer 203.0.113.10 | ASA show vpn-sessiondb l2l, Palo Alto show vpn ipsec-sa |
| Why did it fail? | debug crypto condition peer ipv4 203.0.113.10, then debug crypto ikev2, and undebug all after | FortiGate diagnose debug application ike -1, Palo Alto tail follow yes mp-log ikemgr.log |
| DMVPN state | show dmvpn, show ip nhrp | |
| MPLS and L3VPN | show mpls ldp neighbor, show mpls forwarding-table, show ip bgp vpnv4 all summary, show ip route vrf ITS | |
| Did failover work? | show ip sla statistics, show track, show ip route 0.0.0.0 | |
| Is QoS dropping? | show policy-map interface GigabitEthernet0/0/0 |
WAN work is slow to change and expensive to get wrong, because circuits take weeks to arrive and contracts last years. I follow the same path every time: agree the outcome, prove the facts, design for failure, then hand over something the support teams can run.
Swipe sideways to see the whole diagram.
I start by agreeing why the WAN is changing: a contract ending, a new site, poor application performance or a security finding. I write the goals as measures, and agree who decides on cost, risk and timing (a RACI: responsible, accountable, consulted, informed).
I group sites into tiers, such as data centre, hub, large branch and small branch, and agree requirements per tier. Functional requirements say what each site must reach. Non functional ones set bandwidth, latency, loss, failover time and encryption.
I build the circuit inventory from bills and carrier portals, then check it against what the routers actually show. Flow data shows what crosses each link, and the routing tables show how failover really behaves, which is often not what the drawings say.
I design one pattern per site tier: circuits and real diversity, CE to PE routing, the VPN and encryption standard, QoS classes and failover. The high level design (HLD) and low level design (LLD) go to the design authority before anything is ordered.
I test the configuration and every failure case in a lab, then pilot on a few sites. Each cutover has a runbook, a rollback point and a test list: routing, failover, voice quality and the key applications, checked before the change is closed.
Before go live, every circuit is in monitoring with its ID, carrier contacts and escalation route. The network operations centre (NOC) gets runbooks for the common faults, and hypercare ends only when the agreed exit criteria are met.
WAN contracts, router software and hardware all have end dates. I keep a renewal calendar, a patch plan for security advisories and an end of life register, and review capacity every quarter so upgrades are ordered before they are urgent.
What it must do.
How well it must do it, with a number wherever one can be agreed.
These are typical starting targets for a WAN. Each organisation sets its own per site tier, and each number is traced to a carrier SLA or a test.
| Tool or source | What it tells me |
|---|---|
| NetFlow or IPFIX (IP Flow Information Export) collector | Which applications cross each circuit in the busy hour, to size bandwidth and QoS classes. |
| Carrier turn up test reports (ITU-T Y.1564 or RFC 2544) | Whether each circuit met its contracted throughput, loss and delay at handover. |
show ip bgp neighbors 192.0.2.1 advertised-routes and received-routes | What each site really tells the carrier, and what it hears back. |
| Carrier Ethernet OAM and the carrier's diversity records | Whether two circuits really take separate ducts and exchanges. |
| Carrier portals and bills | Circuit IDs, speeds, service levels and contract end dates. |
show ip sla statistics and show track | Whether failover probes exist and what they really test. |
I find WAN faults from the circuit upwards, and separate my side from the carrier's side quickly, because a carrier ticket needs evidence to move.
| Layer | What I check |
|---|---|
| 1 · Physical | Interface errors and drops, optics, the carrier's terminating equipment, show controllers. |
| 2 · Data link | VLAN tagging on the hand off, ARP (Address Resolution Protocol) to the next hop, speed and duplex, carrier Ethernet OAM. |
| 3 · Network | show ip bgp summary, routes in and out, show ip route vrf ITS, show track, and show crypto ipsec sa counters for the tunnels. |
| 4 · Transport | MSS with ping 10.0.0.1 size 1400 df-bit, and TCP retransmissions in a capture. |
| 5 to 7 · Application | QoS class of the traffic, DNS (Domain Name System), and the application response time users really see. |
My general method is on the troubleshooting page, and the IPsec fault finder is further up this page.
Most WAN faults sit with a carrier, so the support model starts with who raises the ticket, and with what evidence.
Who owns each part: the service desk, the network operations centre, each carrier and any managed service provider, with the escalation times written into each contract.
Circuit state, BGP neighbours, IP SLA probe results, tunnel state, QoS drops and certificate expiry, with thresholds tuned during hypercare.
Circuit IDs, contacts and escalation routes in the monitoring tool, and a monthly service review with each carrier.
Circuit down, failover that did not trigger, a tunnel down at each IPsec phase, MTU faults, and replacing a branch router.
Configuration from per tier templates, changes through change control, a pilot site first, and a rollback point for every change.
Repeat faults get a root cause and a known error record, and recurring carrier faults go to the service review.
A WAN has three clocks running at once: carrier contracts, router software and router hardware. I plan all three from the start.
WAN work has run through my whole career, from carrier networks to global refreshes. My lab keeps the hands on side of it current.
Two P routers and five PE routers carry two customers in separate VRFs. Each of the three main sites fails over to DMVPN tunnels across a simulated internet. See the topology.
Audit, high and low level design and implementation of LAN (local area network), WAN, SD-WAN and data centre infrastructure, replacing end of life equipment with minimal impact on global operations.
Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center, and upgraded every global site remotely during the lockdowns through local enablement, with no delay.
High and low level designs for enterprise customers including Rolls Royce, Virgin Money, Rio Tinto and Primark, including manufacturing and mining site connectivity with control networks kept apart from corporate IT (information technology).
LAN, WAN, VoIP (voice over IP), virtualisation and security across the group, including the warehouse and cold chain systems at the food depots, linked to the group ERP (enterprise resource planning).
Cisco based telecoms infrastructure for over 4,000 clients across East Africa and South America.
Backed by CCNP (Cisco Certified Network Professional) Enterprise and CCNP Routing and Switching, with the CCIE (Cisco Certified Internetwork Expert) Routing and Switching and CCIE Security written exams passed. See all credentials
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.