Separate the planes
Management, control and data each do one job. Controllers decide, edges forward, and a controller outage does not stop traffic for up to 12 hours by default.
SD-WAN and SDN
Software defined networks, from branch to data centre
I design and deliver software defined networks across the WAN, the campus and the data centre. This page shows how they work, the decisions that matter, and where I have used them. You can step through most diagrams at your own pace.
Overview
Why software defined changes the job
Traditional networks are run one box at a time. Software defined networks are run by intent: I describe what the network should do once, and controllers make every device do it.
Swipe sideways to see the whole diagram.
Box by box. Every router holds its own control plane and its own configuration. A change means logging in to each device in turn. Across 50 sites that is 50 changes, and each one is a chance for a typo. Over time the network drifts from its design.
Software defined. The overlay control plane and the policy move to controllers. I write the intent once, as templates and policy, and the controllers apply it to every edge. Edges still run BFD (bidirectional forwarding detection) and local routing, but take their overlay routes and policy from the controllers. Changes are faster, identical everywhere and easy to audit.
Policy, not commands
Templates and central policy replace hand typed configuration, so 5 sites and 500 sites are run the same way.
Any transport
MPLS (Multiprotocol Label Switching), broadband and 5G become interchangeable. Each application uses whichever path meets its needs right now.
Planes
Four planes, one fabric
Cisco Catalyst SD-WAN, which grew out of Viptela, splits the network into orchestration, management, control and data planes. The clearest way to see how they fit is to follow a new branch from power on to live traffic.
Swipe sideways to see the whole diagram.
The new branch edge boots with only an address and a certificate. Plug and Play points it at the Validator, so nobody has to travel to the site.
The Validator checks the device certificate against the allowed list, discovers any NAT on the path, and tells the edge where the Manager and Controllers live. It needs a public address every edge can reach, because it is the first thing a new edge talks to.
The Manager pushes the device template: interfaces, segments, routing and security. Every site is built from the same versioned intent, which is how I keep a large estate consistent.
The edge opens DTLS (Datagram Transport Layer Security) sessions to the Controllers and exchanges OMP (Overlay Management Protocol) routes, transport locators (TLOCs) and encryption keys. The Controllers act like route reflectors, with policy applied centrally.
Edges build IPsec (IP Security) tunnels directly to each other over every transport, then run BFD inside each tunnel to measure loss, latency and jitter every second.
User traffic goes straight from site to site. If the Controllers become unreachable, edges keep forwarding on their last known state for the graceful restart period, 12 hours by default.
Overlay
Any transport, one secure overlay
SD-WAN is a stack of layers. Each one hides the detail of the layer below, which is what makes the circuits interchangeable and the policy simple.
Swipe sideways to see the whole diagram.
I start with the circuits a site can actually get: MPLS where it is still justified, business internet, and 4G or 5G for real path diversity. Two circuits are only diverse if their last mile is, so I check ducts and exchanges, not just carrier names.
Each circuit becomes a TLOC (transport locator): the edge's system IP, a colour such as
mplsorbiz-internet, and the encapsulation. Colours decide which tunnels may form. The restrict option stops a TLOC building tunnels to any other colour, for example MPLS to internet.Edges build an IPsec tunnel for every allowed pair of TLOCs and run BFD across each one. This encrypted fabric is what every application rides on.
Inside the fabric, each VPN is its own routing table from end to end. Corporate, guest and OT traffic share the circuits but never share a route unless policy says so.
Policy sits on top, written once on the Manager and enforced by the Controllers and edges: which application uses which path, which segments may talk, and where traffic must be inspected.
Routing
How routes travel through the fabric
Inside Catalyst SD-WAN, routes move by OMP (the Overlay Management Protocol). It works much like BGP (Border Gateway Protocol) with route reflectors: edges tell the Controllers what they can reach, and the Controllers decide who hears what.
Swipe sideways to see the whole diagram.
Edge A learns its LAN prefix from OSPF (Open Shortest Path First), BGP, EIGRP (Enhanced Interior Gateway Routing Protocol) or a static route inside a service VPN. Connected and static routes go into OMP by default, and so do OSPF internal routes. BGP, EIGRP and OSPF external routes stay out until I advertise them under the OMP address family, and I filter anything the fabric should not learn.
Edge A sends three kinds of route over its control connection. OMP routes carry its prefixes, TLOC routes describe each circuit (colour, preference and weight), and service routes point to any firewall or IPS (intrusion prevention system) attached to it in a service VPN. OMP also carries the keys for the data plane.
The Controller is a route reflector. It is never in the data path. It picks the best path, then applies control policy: the topology (full mesh or hub and spoke), route filtering and service chaining, such as sending OT traffic through the hub firewall.
Edge B receives 10.1.0.0/16 with TLOC A as the next hop. On IOS XE edges OMP routes have an administrative distance of 251, so a route learned locally from OSPF or BGP still wins unless I change it.
Edge B already has an IPsec tunnel to TLOC A, checked by BFD, so traffic for 10.1.0.0/16 rides it. If every Controller is lost, the edges keep forwarding on what they last learned, for up to 12 hours by default.
The numbered VPNs
| VPN | What it carries |
|---|---|
| VPN 0 | Transport. The circuits, TLOCs, public addresses and NAT, the default route to each carrier and the control connections. Direct internet breakout leaves from here. |
| VPN 512 | Out of band management, kept apart from both transport and user traffic. |
| Service VPNs | User traffic, one VPN per segment, each mapped to a VRF (virtual routing and forwarding table): corporate, guest, OT, PCI (payment card) and so on. |
Where each protocol runs
| Where | Protocol |
|---|---|
| Between edges and Controllers | OMP, inside the DTLS or TLS control connection. |
| Transport side, VPN 0 | Static default routes, or BGP to an MPLS provider. |
| Service side, each VPN | BGP, OSPF, EIGRP or static routes to the LAN, the data centre or the old WAN. |
| Every tunnel | BFD (Bidirectional Forwarding Detection), for liveness and path quality. |
How OMP chooses between two routes
- Active before stale. Then only valid routes, whose TLOC is known and reachable.
- Administrative distance. The lower distance wins.
- OMP route preference. The higher preference wins. This is the main tool I use to steer a prefix.
- TLOC preference. The higher preference wins, which is how I make one circuit or hub the primary.
- Origin. Connected, then static, then the routing protocols in a fixed order, then the lower origin metric.
- Tie breakers. A route from an edge beats one from a Controller, then the lowest system IP, then the lowest private TLOC address.
Public and private colours
The colours mpls, metro-ethernet and private1 to private6 are private. Two private colours build tunnels to each other's private address. Every other colour, such as biz-internet or lte, is public, and tunnels to it use the public address the Validator saw through NAT.
Redistribution with care
A prefix that goes from OMP into OSPF at one site and back into OMP at another comes back with the wrong metric and can loop. I tag every route that comes from the fabric and filter it on the way back.
Two sites, one BGP AS
If two SD-WAN sites also meet through the old MPLS network in the same BGP autonomous system (AS), routes can leak through the underlay and loop. The overlay AS path, Site of Origin or tag filtering stops it.
Keep the hierarchy
Redistributing OMP into BGP at a regional core can flatten a hub and spoke design into one big mesh. I use route maps and OMP tags so the regions stay as designed.
Path selection
Each application takes the best path
Application aware routing measures every tunnel and moves each application to a path that meets its service level. Try it: change the circuits and watch the traffic move.
Swipe sideways to see the whole diagram.
MPLS
Internet
| Application | Service level | Path |
|---|
Shown with fallback to best path turned on.
Measured, not assumed
BFD probes run inside every tunnel and report loss, latency and jitter. Decisions use measured averages, so a brownout is handled as well as an outage. I tune the timers so the fabric reacts quickly.
Service levels per class
I group applications into a few classes, such as voice, video, business critical and default, with clear thresholds. Too many classes make the design hard to run.
Metered links stay quiet
5G is a good backup, but it is charged per gigabyte, so I keep bulk traffic off it unless nothing else is left. I write cost into the policy.
Service level classes I start from
| Class | Loss | Latency | Jitter |
|---|---|---|---|
| Voice | under 1% | under 150 ms | under 30 ms |
| Video | under 1% | under 200 ms | under 50 ms |
| Business critical | under 3% | under 300 ms | no limit set |
| Default | under 5% | no limit set | no limit set |
Indicative starting values. Each organisation agrees its own, and the simulator above uses the same numbers.
When no path meets the service level
| Setting | What happens |
|---|---|
| Nothing set | Traffic is shared across every tunnel that is up. |
| Fallback to best path | The least bad path carries it. I use this for most classes. |
| Backup preferred colour | A named transport takes over, such as the internet when MPLS fails its service level. |
| Strict | The traffic is dropped. I keep this for traffic that must never use a poor path. |
BFD sends a hello every second by default and declares a tunnel down after seven missed hellos. Path quality is averaged over 10 minute poll intervals by default, which is too slow for voice, so I shorten the timers or use enhanced application aware routing on newer releases.
Quality of service on the edge
| Piece | What it does | How I set it |
|---|---|---|
| Classification | A central data policy, or a local access list, matches the application, DSCP (Differentiated Services Code Point) value or addresses, and sets a forwarding class. | A few classes only, the same names at every site. |
| Queues | Each forwarding class maps to one of eight queues on the WAN interface. Queue 0 is the low latency queue, used by control traffic and voice. | Voice and control in queue 0, then video, business critical and default by weight. |
| Shaping | The transport interface in VPN 0 is shaped to the circuit's real rate, so queuing happens on my edge, not in the carrier's network. | Shape to the contracted rate, not the port speed. |
| Marking in the tunnel | The inner DSCP is copied to the outer IPsec header, so an MPLS carrier can honour it. | Map the classes to the carrier's own classes at the MPLS hand off. |
Quality of service only matters when a link is full. Before I rebuild a policy, I check utilisation and queue drops with show policy-map interface.
Segmentation
Separate networks on shared circuits
Segmentation has to be designed. Each business function gets its own VPN, with its own routes and its own rules for leaving it.
Swipe sideways to see the whole diagram.
Corporate, guest and OT each get a VPN, which is a VRF (virtual routing and forwarding table) on the edge. They share circuits and tunnels but keep completely separate routing tables.
Corporate users reach business applications in the data centre over the best path for each application. Nothing else can see those routes.
Guest traffic leaves through the local internet breakout. There is no route from VPN 20 to the data centre, so a misconfigured device cannot reach it even by accident.
OT and IoT traffic is chained through a firewall at the hub before it reaches the historian. The hub advertises the firewall as a service, and a control policy on the Controllers sends VPN 30 traffic through it, without touching each edge.
Some services, such as DNS and identity, are needed by more than one segment. I leak those prefixes deliberately with central extranet policy, prefix by prefix, so every exception is visible and auditable.
Security
Breaking out locally, without losing control
Where internet traffic leaves the network is one of the biggest SD-WAN decisions. There are three patterns, and each has a cost.
Swipe sideways to see the whole diagram.
Backhaul. All internet traffic goes through the data centre firewall. It is simple to secure, but SaaS is slower and the data centre circuit and firewall carry everyone's browsing.
Local breakout. Internet traffic leaves at the branch. SaaS gets faster, but every branch becomes an internet edge, so it needs a zone based firewall, IPS (intrusion prevention system) and URL filtering on the edge, sized for that load.
Cloud security. The branch tunnels internet traffic to a security service edge: web gateway, CASB (cloud access security broker), DNS security and zero trust access. Run with SD-WAN under one policy, this is SASE (secure access service edge). It is fast, consistent and protects remote users the same way. I still send Microsoft 365 traffic in its Optimize category straight out, as Microsoft advises. For most estates this is the pattern I recommend now.
Security features on the edge, and what they cost
| Feature | What it does | Sizing note |
|---|---|---|
| Zone based firewall | Stateful policy between zones built from the VPNs. The default control at every branch. | Light. Runs on every edge. |
| DNS security (Umbrella) | Blocks bad domains at the DNS request, before a connection exists. | Very light. A good first layer everywhere. |
| IPS and IDS (intrusion detection system) | Snort in a security container on the edge, using Talos signatures. | Heavy. Needs memory and CPU headroom, so the platform is sized for it. |
| URL filtering | Blocks sites by category and reputation, with a cloud lookup. | Moderate. Needs the same security container. |
| Malware protection | Checks file reputation, and alerts if a file later turns out to be bad. | Moderate. Often better done in the cloud. |
| Cloud security tunnels | IPsec or GRE (Generic Routing Encapsulation) tunnels to a security service edge such as Umbrella or Zscaler, built automatically. | Light on the edge. The heavy inspection moves to the cloud. |
Before IPS goes on every edge
I ask which threat it stops and whether IPS is the right control. Most HTTPS needs decryption before IPS sees anything, which costs far more. I usually run IPS where the threat justifies it, cloud security for the rest, and roll it out in waves with a rollback each time.
Cloud OnRamp for SaaS
The edge probes Microsoft 365, Webex and other SaaS through each exit, then sends users out of the best one, whether that is local internet, a regional hub or a cloud security tunnel.
Cloud OnRamp for multicloud
The Manager builds virtual edges and connections into AWS, Azure or Google Cloud, directly or through Megaport or Equinix, so cloud regions join the fabric like any other site. The cloud page shows the hybrid patterns.
Campus
Policy that follows the user
SD-Access brings the same idea to the campus. Identity decides the policy, not the port or the IP address, so a user gets the same access wherever they plug in.
Swipe sideways to see the whole diagram.
Catalyst Center builds the routed IS-IS underlay with LAN Automation, then assigns the fabric roles. No spanning tree, no VLANs stretched across the campus, and every switch configured the same way.
The laptop authenticates with 802.1X. Cisco ISE (Identity Services Engine) checks who and what it is, then assigns a scalable group tag (SGT), here finance, which carries the policy from now on.
Edge node A tells the control plane node where the laptop lives. The control plane node is a LISP (Locator/ID Separation Protocol) map server: a live directory of every endpoint in the fabric.
When the laptop talks to the finance app, edge A asks the control plane node where 10.20.1.5 is and caches the answer. No flooding and no giant MAC (media access control) address tables.
Edge A wraps the traffic in VXLAN (virtual extensible LAN) and stamps the group tag into the header, so the user's identity travels with every packet across the fabric.
Edge B checks the group policy matrix: finance to the finance app is permitted, guest to the same app is denied. The rule follows the user, wherever they connect.
Data centre
Fabrics that grow leaf by leaf
In the data centre, software defined usually means a leaf and spine fabric with a VXLAN overlay and a BGP EVPN (Ethernet VPN) control plane. The other common choice is Cisco ACI (Application Centric Infrastructure), which also uses VXLAN but is run end to end by its own controllers. The data centre page covers the physical side.
Swipe sideways to see the whole diagram.
Every leaf connects to every spine, so traffic between any two leaves crosses one spine, over equal cost paths. Adding capacity means adding a leaf or a spine, not redesigning the network.
When VM A comes up, leaf 1 learns its MAC and IP address locally from its first packets. Other leaves learn it from BGP, not by flooding.
Leaf 1 advertises a type 2 route through the spines, which act as route reflectors. Every leaf now knows that VM A lives behind leaf 1's tunnel endpoint (VTEP).
VM B sends to VM A. Leaf 3 wraps the frame in VXLAN with the segment's VNI (VXLAN network identifier) and sends it across either spine. Leaf 1 unwraps it and delivers it.
Cisco ACI also carries traffic in VXLAN, but works differently. The spines keep a database of every endpoint instead of using BGP EVPN, and an APIC (Application Policy Infrastructure Controller) cluster configures the whole fabric and applies policy as groups of workloads and contracts between them. I have worked with both. The right choice depends on the team that will run it.
Delivery
Migrate in waves, run it by intent
Most of the work is moving a live estate across without an outage, then keeping it tidy.
Swipe sideways to see the whole diagram.
I start with what is really there: circuits, devices, routing, and above all the application flows. You cannot migrate what you have not mapped.
The data centre edges move first, so the SD-WAN fabric and the old MPLS or DMVPN (Dynamic Multipoint VPN) network meet at the hubs. I align OMP and BGP preferences there so return traffic never takes a different path through a stateful firewall.
One branch per hub goes next. We prove the template, the policy and the support process on real users before scaling.
The rest move in planned waves, lowest risk first, each with pre and post checks, a tested rollback and a go or no go decision.
Once every site is on the fabric, MPLS is thinned to what is still justified, or retired.
How I keep it running well
- Template everything. Every device is attached to a versioned template. Any CLI add on is tracked, justified and removed over time.
- Controllers first. Upgrade the controllers, let them settle, then move edges in waves: lab, low risk, standard, critical.
- Watch for drift. Compare running configuration with intent through the API every day. I report drift each month, and review the trend each quarter as a governance measure.
- Troubleshoot from the bottom up. Check control connections, then BFD and OMP, then policy. The fault is usually upstream of where it shows.
- Evidence for every change. Pre and post state, a tested rollback and agreed go or no go gates, presented at change board.
- Tier the sites. Not every site needs the same resilience. Hubs and major sites get two edges and two diverse transports. Small offices get one edge with broadband and 4G or 5G backup. Each tier is a standard pattern, so resilience is built in every time.
- Test it in the lab. My home lab runs the MPLS and DMVPN designs that SD-WAN replaces, so I can test routing and failover on the legacy side before a cutover.
Operations
Templates, drift and upgrades
An SD-WAN estate stays healthy only if the configuration on every edge matches the intent. This is how I keep the two in step, and how I upgrade without surprises.
Swipe sideways to see the whole diagram.
Every edge is attached to a versioned template, and the templates live in Git, each with one owner. A template change is a change to every site that uses it, so template edits go to the change board, not just device pushes.
The Manager fills each template with the site's own values and pushes it. Anything typed on an edge's command line is overwritten at the next push, which is why a local fix must go back into the template.
Drift starts with a manual change during a fault, a fix that was never reverted, a push that failed part way or a device left in CLI mode. The Manager's out of sync flag only compares with the last attached template, so it misses some of these.
Every day I pull each running configuration through the API and compare it with the intent in Git. The report lists each difference, with the device and the time it appeared.
An emergency fix that was right is written into the template. An unauthorised change is reversed and reviewed. A gap in the templates goes on the backlog. I report the count each month: the aim is that every difference is known and closing, not that the number is zero.
Template types
| Type | How I use it |
|---|---|
| Feature templates | Small building blocks, such as an interface, OSPF, AAA (authentication, authorisation and accounting) or logging. The standard pieces, owned and versioned. |
| Device templates | A set of feature templates that defines one kind of device. One per site type and platform. |
| CLI templates and add ons | Raw configuration text. Only when nothing else covers a feature, with an owner, a reason and an end date. Every add on is future drift. |
| Configuration groups | The newer, parameter driven model on recent Manager releases. New builds start here. Existing estates move only for a real feature reason, in waves. |
Upgrading without surprises
- Check the compatibility matrix. Cisco publishes which Manager, Validator, Controller and edge versions work together. A mismatch can fail quietly.
- Controllers in their own window. The Manager, then the Validators, then the Controllers. Never in the same window as the edges.
- Let it settle. A soak period before any edge moves.
- Edges in waves. Lab, low risk, standard, then critical sites, with a snapshot before each wave and a go or no go decision.
- Pick the release train. A long life release for most estates, a newer one only when a feature needs it.
Template governance
One owner per template and a naming rule of region, site type, function and version, such as UK BRANCH WAN v3.2. Templates are mirrored in a test environment, and obsolete ones are cleared out each quarter.
When templates do not cover a feature
First choice is a feature request to the vendor and a tracked, time limited add on. Next is a configuration group. A pure CLI template is the last resort, with a written reason, an owner and an end date.
Segmentation as evidence
For an auditor, the policy intent, the Manager's configuration, the live state on each edge and the test results must all agree. Drift between intent and live state is usually the first finding, so the daily comparison doubles as evidence.
Faults
Faults I plan for, and how I find them
Pick a fault to see where it sits, what I check first and how I fix it. I always work from the underlay up: circuit, control connections, tunnels, then policy.
Swipe sideways to see the whole diagram.
Bad template push. Many sites drop straight after a template change. I stop further pushes, work out the blast radius and find the exact difference. Sites still reachable through VPN 512 show what landed. I attach the previous template version, and recover unreachable sites by console, out of band access or zero touch provisioning. The review asks why the lab and the first wave did not catch it.
Controller out of sync. The Manager marks a Controller out of sync when its configuration or policy no longer matches the Manager. The usual causes are a failed template or policy push, a Controller changed in CLI mode, or a Manager cluster problem. I also check NTP (Network Time Protocol) and certificates, because they break control connections. Until it is fixed, I treat every change pushed in the meantime as suspect.
No control connections. The edge is up but has no control connections, so it learns nothing. I check the transport first, then show sdwan control connections and show sdwan control local-properties: certificate, system IP, site ID, organisation name, clock, and whether the Validator is reachable through any NAT in the path.
Missing tunnel. Control is fine, but one tunnel never forms. I check show sdwan omp tlocs and show sdwan bfd sessions. The cause is usually a colour or restrict setting, a TLOC that cannot reach the other one through NAT, or a firewall on the path blocking UDP 12346 to 12445, which the tunnels use.
Slow after inspection. Throughput falls or CPU climbs after IPS, URL filtering or a TLS proxy is turned on. I check the security container's load and the traffic mix. The fix is to trim features, move inspection to cloud security or use a bigger platform. Sizing before switching on avoids it.
The edge commands I use most
| What I want to know | Command on an IOS XE edge |
|---|---|
| Are the controllers reachable? | show sdwan control connections |
| Who is this edge: certificate, system IP, site ID, organisation | show sdwan control local-properties |
| Are the tunnels up, and how good are they? | show sdwan bfd sessions and show sdwan bfd summary |
| Is OMP peering up? | show sdwan omp peers |
| Which routes and TLOCs has it learned? | show sdwan omp routes vpn 10 and show sdwan omp tlocs |
| What policy did the Controller send? | show sdwan policy from-vsmart |
| Path quality per tunnel | show sdwan app-route stats |
| What is really configured? | show sdwan running-config |
Engagement
How I run an SD-WAN programme, from first workshop to refresh
Every SD-WAN programme I lead follows the same path. I start with the people and the requirements, prove the facts with discovery tools, then design, test and hand over something the operations team can support and keep current.
Swipe sideways to see the whole diagram.
I start by agreeing why the programme exists: lower circuit cost, faster site moves, better application experience or tighter security. I write the goals as measures we can test later, and agree who decides what (a RACI: responsible, accountable, consulted, informed).
I split requirements into functional ones, what the network must do, and non functional ones, how well it must do it. Each non functional requirement gets a number, such as loss, latency, recovery time or sites per region, so it can be tested.
I check the facts against data, not memory: circuit bills and contracts, NetFlow and SNMP history, routing tables and firewall rules. This shows what really crosses the WAN, which sites are fragile and where the drawings are wrong.
I design a small number of site templates, the segmentation model, the routing between the fabric and the old network, and the application policies. The high level design (HLD) and low level design (LLD) go to the design authority, and every decision goes in the decision log.
I test the templates and failover in a lab, then run a pilot on a few real sites and measure the results against the non functional targets. Each migration wave then goes through change control with a rollback point.
Before the first wave I agree who supports what: the service desk, the network operations centre (NOC), the carrier and the vendor. Runbooks, alerts and escalation are tested in hypercare, which ends only when the agreed exit criteria are met.
Controller and edge software, security fixes, circuits and hardware all have a life. I keep a release plan, an end of life register and a quarterly capacity review, so the estate never drifts out of support.
Functional requirements
What it must do.
- Connect every site type (branch, hub, data centre and cloud) over at least two transports.
- Keep business, guest, OT and management traffic in separate segments from end to end.
- Steer each application class onto the best path, within set limits for loss, latency and jitter.
- Send trusted SaaS traffic straight to the internet, and the rest through cloud or hub security.
- Join the fabric to the existing WAN during migration, with clear routing between the two.
- Bring new sites online from templates with zero touch provisioning.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Availability
- 99.99% at hub sites and 99.9% at branches with two transports.
- Failover
- Traffic moves to the other transport within seconds of an outage, and within a minute of a brownout with tuned application aware routing timers.
- Performance
- Voice under 150 ms one way, loss under 1% and jitter under 30 ms.
- Scale
- Sites per region, tunnels per edge and growth headroom agreed up front.
- Security
- Encrypted tunnels, segmentation, firewalling at the edge and logs sent to the SIEM (security information and event management).
- Operability
- Configuration from templates, drift reported, and every change through change control.
- Cost
- Circuit and licence cost per site agreed and tracked.
Typical starting targets for an SD-WAN estate. Each organisation agrees its own numbers, and each one becomes a test in the pilot.
Who I talk to
- Network, security and cloud teams
- Application owners for voice, enterprise resource planning (ERP) and SaaS
- Site managers and regional IT
- Carriers, account managers and the current supplier
- Finance, for circuit contracts and renewal dates
How I find out
- Workshops by site type and region
- Site surveys of power, cabinets and where each circuit is handed off
- Traffic and application profiling from flow data
- Reading the current routing, firewall and QoS (quality of service) configuration
- Contract and circuit review: bandwidth, renewal dates and exit costs
Discovery tools
| Tool or source | What it tells me |
|---|---|
| NetFlow or IPFIX (IP Flow Information Export) collector | Which applications cross the WAN, when, and how much. |
| SNMP monitoring history (SolarWinds or similar) | Link use, errors and outages over months, not days. |
Router and firewall configuration, show ip route | The real routing, NAT and QoS, compared with the drawings. |
| Circuit inventory and carrier bills | Every circuit, its speed, cost and contract end date. |
| Packet captures (Wireshark) | Proof of loss, retransmission or MTU problems on a real flow. |
| Catalyst SD-WAN Manager in a pilot | Measured loss, latency and jitter on each path before the rollout. |
I find SD-WAN faults the same way as any network fault, but I check the underlay (the circuits) and the overlay (the tunnels and policy) separately. Most problems are a circuit, a control connection or a policy, in that order.
My approach
- Define. What is affected: one site, one application, one segment or everything? Since when, and what changed?
- Underlay. Check the transport: interface, carrier, public IP address and NAT, and whether the edge can reach the Validator.
- Control. Check control connections to the Manager and Controllers, then OMP peers and the routes each edge has learned.
- Data plane. Check BFD sessions for each tunnel and colour, then measured loss, latency and jitter on each path.
- Policy. Check which application aware routing or data policy matched, and which path the flow really took.
- Fix and record. Fix the template, not just the edge, through change. Confirm with the user and the monitoring, and record the cause for problem management.
Faults I see most
- A circuit brownout that shows in loss and latency but not on the interface
- Control connections down after a certificate or clock problem
- Tunnels missing because of a colour or restrict setting
- Asymmetric routing through a stateful firewall at the hub
- Large transfers stalling because of the MTU inside the tunnel
- A policy matching the wrong application or segment
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Interface state, errors, optics, the carrier's equipment and circuit tests. |
| 2 · Data link | VLAN tagging on the transport, ARP (Address Resolution Protocol) to the next hop, speed and duplex. |
| 3 · Network | Underlay routes and the default route, NAT type, show sdwan control connections and show sdwan omp routes. |
| 4 · Transport | BFD sessions with show sdwan bfd sessions, IPsec counters, MTU and TCP MSS (maximum segment size). |
| 5 to 7 · Application | Application recognition, the policy match, DNS and the SaaS path, and what the user really sees. |
My general method, with worked faults, is on the troubleshooting page.
I agree the support model before the first site moves, so nobody learns their role during an outage.
Service model
Who owns each part: service desk, network operations centre, carrier, vendor and any managed service provider, with escalation times.
Monitoring
Alerts on path quality, tunnels, control connections and certificate expiry, tuned during hypercare so each alert means something.
Runbooks
Step by step guides for common faults, site moves, circuit swaps and replacing an edge.
Change
Template changes go through change control, to a test site first, with a rollback point.
Hypercare
Extra cover after each wave, with exit criteria agreed up front: no open priority one faults, alert noise under control and runbooks proven.
Problem management
Repeat faults get a root cause, a known error record and a fix that stops them coming back.
An SD-WAN estate is never finished. I plan its life from the start, using the vendor's release and end of life dates.
- Plan. A release strategy: which software train, how often to upgrade and how far behind the latest release we may sit.
- Upgrade. Controllers first, then edges in waves, checking the version compatibility matrix before each step.
- Patch. Cisco PSIRT (Product Security Incident Response Team) advisories for IOS XE SD-WAN are triaged by CVSS (Common Vulnerability Scoring System) score and exposure. Critical fixes go through an emergency change, the rest join the next wave.
- Review. Quarterly reviews of capacity, circuit cost, template drift and application policy.
- Renew. Circuit contracts and licences are tracked against renewal dates, so changes are planned rather than forced.
- Refresh. Edge hardware is replaced before end of support, in the same wave pattern as the first rollout.
Projects
Where I have delivered it
Software defined networking has been part of my work since the early Viptela days, across banking, industrial software, telecoms and consulting.
Global network and security transformation
Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center, and modernised every site worldwide remotely during lockdown, leading 7 engineers and coordinating 5 vendors.
EMEA network refresh
Audit, high and low level design and implementation across LAN, WAN, SD-WAN and data centre, kept compliant with MiFID II throughout, with no service outages.
Technical design authority
Shaped SD-WAN, SD-Access and Catalyst Center planning so the backbone, data centre and edge could carry high throughput, low latency workloads.
Lead architect and design authority
Network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access and multi cloud, plus multi vendor SD-WAN and SD-Access designs for enterprise clients.
Early software defined networking
Cloud, SDN and security architecture on Nexus, Viptela, Palo Alto, F5 and Aruba, then architecture and delivery across Catalyst SD-WAN, SD-Access and disaster recovery.
Software defined data centre
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI and Nexus.
Backed by CCNP Enterprise, the Cisco Enterprise Design and Enterprise Core specialist certifications, and the CCIE Security and CCIE Routing and Switching written exams. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.