Day 0, day 1, day 2
Day 0 builds and bootstraps the device. Day 1 adds the first services. Day 2 is everything after that, for the life of the network: changes, patching, scaling and checks. Day 2 is where automation pays back.
Network automation
Automating networks safely, from data to device
I automate networks so that every change is repeatable, tested and recorded. Intent lives in a source of truth and in Git, a pipeline (CI/CD: continuous integration and continuous delivery) checks each change before and after it runs, and the network reports its own state back. This page walks through the tools and the method, from NETCONF (Network Configuration Protocol), YANG (Yet Another Next Generation) data models and gNMI (gRPC Network Management Interface) to the tests that guard each change.
Overview
Why I automate, and how far to take it
Most network outages start with a change made by hand. Automation takes away the typing, but its real value is that every change follows the same tested, recorded process. I move teams up these levels one step at a time.
Swipe sideways to see the whole diagram.
Manual. An engineer logs in to each device and types the change. That works for one box. Across hundreds, small typing errors and forgotten steps build up, and the running network slowly drifts away from its design.
Scripted. A Python script pushes the same commands to every device. It is fast, but a script without tests repeats a mistake everywhere at once, and nothing records who changed what, or why.
Pipeline. The change starts as data in a source of truth and a pull request in Git. A pipeline tests it, waits for approval, makes the change and proves the result. This is where I take most teams first.
Closed loop. The network streams its state back. When something drifts or degrades, analytics and policy raise a ticket or trigger a known fix through the same pipeline. Engineers set the intent and the limits, not every command.
Declarative or imperative
A declarative tool is told the end state and works out the steps, as Terraform does. An imperative tool is given the steps in order, as an Ansible playbook is. I use both, each for the jobs it suits.
Idempotent by design
An idempotent task changes nothing the second time it runs. That makes retries safe, lets a dry run show drift, and means a failed job can simply run again.
Source of truth
One place that says what the network should be
Automation is only as good as the data behind it. I design around a source of truth, usually NetBox, that holds the intended state: sites, racks, devices, cables, VLANs (virtual local area networks), VRFs (virtual routing and forwarding tables) and IP addresses. Templates turn that data into configuration, so nobody edits a device by hand.
Swipe sideways to see the whole diagram.
The network is modelled as data in NetBox: sites, racks, device roles, interfaces, cables, VLANs, VRFs and prefixes. Custom fields hold what my designs need, such as fabric role, VXLAN (virtual extensible LAN) network identifier (VNI) or migration wave. Addresses are handed out through the API (application programming interface), so two people can never take the same one.
Templates, playbooks and the data for each service live in Git. A change is a branch and a pull request, reviewed by a second engineer and linked to the change ticket. Git keeps the full history, so rolling back is a revert.
Jinja2 templates combine the data and the design into device configuration. The same data and the same template always give the same result. That is what makes a change predictable, and a review meaningful.
Only the pipeline pushes to devices, through a controller or the device's API. Engineers keep read access for fault finding, plus an emergency route for the rare case where the pipeline itself is down.
NetBox creates the configuration management database (CMDB) records in ServiceNow, with the links between them: rack, device, port, cable and service. Scheduled discovery then checks the real network against the record and flags any difference.
NetBox or Infoblox
NetBox is open source DCIM (data centre infrastructure management) and IPAM (IP address management) with a strong API, built for automation. Infoblox is commercial DDI: DNS (domain name system), DHCP (Dynamic Host Configuration Protocol) and IPAM together, on resilient appliances, common in banks. Where both exist, I agree which one owns each record.
Check the data first
A wrong entry in IPAM is a production change waiting to happen. Every data file is checked against a JSON (JavaScript Object Notation) Schema: required keys, types, names, and ranges such as VLAN IDs between 2 and 4094, before anything is rendered.
Filled in the factory
Serial numbers, MAC (media access control) addresses and rack positions are captured as kit is built. LLDP (Link Layer Discovery Protocol) then proves the real cabling matches the plan before a rack ships.
Zero touch provisioning
A new switch boots with no configuration, gets an address and a script location from DHCP, then pulls its day 0 configuration. Cisco calls this ZTP (zero touch provisioning) on IOS XE and POAP (power on auto provisioning) on Nexus. The serial number in NetBox decides which configuration each switch gets.
Discover, then reconcile
Discovery finds what exists, through CDP (Cisco Discovery Protocol), LLDP, SNMP (Simple Network Management Protocol) or a controller sweep. Inventory records it, and provisioning pushes the intended state. Reconciliation compares the two and raises every difference.
Brownfield first
Most networks already exist. The running estate is imported into NetBox with discovery tools and scripts, and the fixes are agreed with the owners. Only then does the pipeline take over each part.
Python
Python for the jobs no product does well
I use Python as glue: pulling data from APIs, checking it, parsing show output and making the changes no product covers. I keep custom code small, reviewed and tested, and lean on proven libraries rather than writing my own.
Swipe sideways to see the whole diagram.
Nornir reads the device list and its variables from NetBox, so the script never holds a hard coded list. Credentials come from a secrets vault at run time, never from the code.
Each thread connects and takes a snapshot of neighbours, routes and interface errors. If a device is already unhealthy, it is skipped and reported. I never change something that is already broken.
The rendered configuration goes to the devices in parallel, within a limit I set, so a mistake cannot reach the whole estate in one go.
The same checks run again and are compared with the snapshot. A device that fails is rolled back on its own, and the run ends with a report attached to the change record.
The libraries I design around
| Library | What it is for |
|---|---|
| Netmiko (built on Paramiko) | SSH (secure shell) sessions to devices that have no API. It handles prompts, paging and configuration mode for many vendors. |
| NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) | One interface across vendors: read facts, load a candidate configuration, show the diff, then commit or discard it. |
| Nornir | Runs tasks across an inventory in parallel threads, in plain Python, with the inventory taken from NetBox. |
| Jinja2 | Templates that turn data into configuration. |
| TextFSM, TTP (Template Text Parser) and Genie parsers | Turn show command output into structured data that a script can test. |
| pyATS and Genie | Snapshots before and after a change, whole feature models and the comparison between them. |
| requests, httpx and pynetbox | REST (representational state transfer) APIs on controllers, ServiceNow and NetBox. |
| ncclient | NETCONF sessions to devices that support it. |
Push a change with Netmiko
from netmiko import ConnectHandler
device = {
"device_type": "cisco_ios",
"host": "10.0.0.11",
"username": user, # read from the vault at run time
"password": secret,
}
commands = ["ntp server 10.0.0.1", "ntp server 10.0.0.2"]
with ConnectHandler(**device) as conn:
before = conn.send_command("show ntp associations")
conn.send_config_set(commands)
conn.save_config()
after = conn.send_command("show ntp associations")Run across the fleet with Nornir
from nornir import InitNornir
from nornir.core.filter import F
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result
nr = InitNornir(config_file="config.yaml") # inventory from NetBox
leaves = nr.filter(F(groups__contains="leaf"))
result = leaves.run(
task=netmiko_send_command,
command_string="show bgp l2vpn evpn summary",
)
print_result(result)Review, then commit, with NAPALM
from napalm import get_network_driver
driver = get_network_driver("nxos")
with driver("leaf101", user, secret) as dev:
dev.load_merge_candidate(filename="leaf101.cfg")
print(dev.compare_config()) # the diff a reviewer reads
if approved:
dev.commit_config()
else:
dev.discard_config()Model the intent, not the commands
I describe a service as data, such as a tenant with its VRFs, VLANs and ports, and validate it before any template runs. The same model drives the configuration, the tests and the documentation.
Scripts that last
One off scripts soon become tools other people rely on. Scripts that last have typed data models, such as Pydantic classes, plus timeouts, retries with back off, and pagination for large API results. They log what they did, have tests, and ship as a container image built with Docker or Podman.
Integration patterns
Systems talk in four ways: REST calls on demand, webhooks when something changes, scheduled syncs for bulk data, and a message bus such as Kafka for high volume events. Each call is idempotent and carries a correlation ID, so one request can be traced across every system.
Ansible
Ansible for ordered, repeatable tasks
Ansible is agentless, and its playbooks are written in YAML (a plain text data format). It connects over SSH or an API, so nothing is installed on the switch. I use it for work that must happen in order: building a device, upgrading software, failing over a pair, and checking state before and after. Ansible Automation Platform runs it safely for a whole team.
Swipe sideways to see the whole diagram.
The automation controller pulls playbooks and roles from Git. What runs in production is always a reviewed commit, never a copy on someone's laptop.
A dynamic inventory plugin reads devices, groups and variables from NetBox when the job starts. I never cache it for production runs, so a device marked as planned or retired is never touched.
A credential plugin fetches secrets from the vault at run time. Engineers launch jobs they are allowed to run, without ever seeing the passwords.
The job runs in an execution environment: a container image with pinned collections and Python libraries, so it behaves the same in testing and production. I run check mode with a diff first, then the real change.
Each host returns ok, changed or failed. The job log and the results are attached to the ServiceNow change. Event Driven Ansible can also launch approved job templates from monitoring alerts.
A playbook using a resource module
# playbooks/vlans.yml
- name: Add the data centre VLANs from the source of truth
hosts: leaf_switches
gather_facts: false
tasks:
- name: Merge the VLANs
cisco.nxos.nxos_vlans:
config: "{{ vlans }}"
state: mergedansible-playbook playbooks/vlans.yml --check --diff --limit leaf101What I hold to in every role
- Data apart from logic. Variables live in group_vars and host_vars, or come from NetBox, and are reviewed like code.
- Resource modules first. Modules such as
nxos_vlanstake a state: merged, replaced, overridden, deleted or gathered. They are idempotent, unlike raw commands. - One role per job. Pre checks, the change, post checks and the evidence pack are separate roles, tested with Molecule.
- Collections per platform.
cisco.ios,cisco.nxos,cisco.dcnmfor Nexus Dashboard,cisco.aci,fortinet.fortiosandpaloaltonetworks.panos, all pinned to versions. - Pairs changed together. Both members of a vPC (virtual port channel) pair are always in the same run, so a pair is never left half changed.
Terraform
Terraform for long lived resources
Terraform is declarative. I describe the end state, Terraform works out the steps and records what it built in a state file. I use it for things with good APIs that live for years: cloud networks, controller tenants, VRFs and firewall objects. OpenTofu, the open source fork, works the same way.
Swipe sideways to see the whole diagram.
I describe the end state in HashiCorp Configuration Language (HCL). Repeated patterns, such as a landing zone spoke or a data centre pod, become modules with a few inputs, so every copy is built the same way.
Terraform reads the state file and the real platform, then lists what it will create, change or destroy. The plan is the thing reviewers and the change board approve, so it is saved with the change.
Apply makes exactly the planned changes, in dependency order, then updates the state. The state file is stored remotely, locked during a run and encrypted, because it can hold secrets.
A scheduled plan with no code change should show nothing. If it shows a change, someone altered the platform outside Terraform. I either bring that change into code through a pull request or put it back. I never ignore it.
A small module input
resource "aws_vpc" "prod" {
cidr_block = "10.20.0.0/16"
tags = { Name = "prod", Owner = "network" }
}
resource "aws_subnet" "app_a" {
vpc_id = aws_vpc.prod.id
cidr_block = "10.20.1.0/24"
availability_zone = "eu-west-2a"
}terraform plan -out=change.tfplan
terraform apply change.tfplanTerraform or Ansible
| Terraform | Ansible | |
|---|---|---|
| Style | Declarative, state based | Tasks in order |
| Best for | Long lived resources with good APIs | Builds, upgrades and ordered operations |
| Drift view | A plan shows it | Check mode shows it |
| State | A state file to protect | No state file |
| Typical use | Cloud networks, tenants, VRFs | Bootstrap, software upgrades, checks |
Both read the same source of truth. I choose per job, not per team.
APIs
Talking to devices through models, not screens
Screen scraping breaks when a vendor changes one line of output. Model driven interfaces give structured data, transactions and rollback. YANG, defined in RFC (Request for Comments) 7950, is the data modelling language that describes the data. NETCONF, RESTCONF (the same ideas over HTTPS, the secure web protocol) or gNMI carries it. OpenConfig models work across vendors, and native models cover each vendor's extra features.
Swipe sideways to see the whole diagram.
The automation host opens an SSH session to port 830 and asks for the NETCONF subsystem. NETCONF is defined in RFC 6241.
Both sides send a hello listing what they support, such as a candidate datastore, validation and the YANG models the device knows. The client now knows what it may ask for.
The client locks the candidate configuration, so no other session can change it half way through.
An edit config message carries the change as XML (extensible markup language), shaped by the YANG model. Nothing is live yet. The candidate is a draft.
The device checks the whole candidate for errors before anything is applied.
The client commits with a confirm timer of 120 seconds, and the device replies with ok. The change is live in one transaction, but the device will roll it back on its own unless it is confirmed in time. That protects against a change that cuts off the management path.
Post checks pass, so the client sends a plain commit before the timer runs out, and the change is kept. If the session drops before this, the device rolls the change back on its own.
The client releases the lock and closes the session. Every step returned a structured reply, so the script knows exactly what happened.
Interfaces compared
| Interface | Transport and port | Encoding | Best for |
|---|---|---|---|
| NETCONF (RFC 6241) | SSH, port 830 | XML | Configuration with transactions: lock, edit a candidate, validate, commit. Edit operations: merge, replace, create, delete, remove. Datastores: running, candidate, startup. |
| RESTCONF (RFC 8040) | HTTPS, port 443 | JSON or XML | Simple reads and writes of YANG data from scripts and tools. |
| gNMI | gRPC over HTTP/2. IANA (Internet Assigned Numbers Authority) lists port 9339, but vendors vary. | Protobuf or JSON | Streaming telemetry, and configuration on some platforms. |
| Controller REST APIs | HTTPS | JSON | Catalyst Center, SD-WAN Manager, Nexus Dashboard, Meraki and APIC (Application Policy Infrastructure Controller). |
| NX-API | HTTPS | JSON or XML | CLI (command line interface) commands with structured output on Nexus switches. |
| SSH and CLI | SSH, port 22 | Text | Older devices only, with the output parsed by TextFSM or Genie. |
Read an interface with RESTCONF
curl -s -u "$USER:$PASS" \
-H "Accept: application/yang-data+json" \
"https://router1/restconf/data/ietf-interfaces:interfaces/interface=GigabitEthernet1"The reply is JSON shaped by the IETF (Internet Engineering Task Force) interfaces YANG model, so a script reads a field by name instead of hunting through text.
Log in to Catalyst Center from Python
import requests
base = "https://catalyst-center.example.net"
token = requests.post(f"{base}/dna/system/api/v1/auth/token",
auth=(user, secret)).json()["Token"]
devices = requests.get(f"{base}/dna/intent/api/v1/network-device",
headers={"X-Auth-Token": token}).json()["response"]
for d in devices:
print(d["hostname"], d["managementIpAddress"], d["reachabilityStatus"])Controller APIs I work with
| Platform | How a session starts | Typical calls |
|---|---|---|
| Catalyst Center Intent API | POST /dna/system/api/v1/auth/token, then send the token as X-Auth-Token. | Device inventory, host onboarding for SD-Access, templates, assurance issues. |
| Catalyst SD-WAN Manager | POST /j_security_check for a session cookie, then GET /dataservice/client/token for the X-XSRF-TOKEN header. | Device inventory and state, templates and configuration groups, path quality for dashboards. |
| Nexus Dashboard, with its fabric controller | POST /login for a token. | Fabrics, inventory, networks and VRFs, then preview and deploy. The cisco.dcnm collection wraps it for Ansible. |
| Cisco Meraki Dashboard | An API key sent as a bearer token to https://api.meraki.com/api/v1. | Organisations, networks, devices and clients, within about 10 calls a second per organisation. |
| Cisco APIC | POST /api/aaaLogin.json for a token cookie. | The object model: tenants, VRFs, bridge domains, endpoint groups (EPGs) and contracts. |
Telemetry
Telemetry that sees what polling misses
SNMP polling every five minutes averages away short spikes. Model driven streaming telemetry pushes data every few seconds, or the moment a value changes, using the same YANG models as configuration. Pick a method and compare what each one sees.
Swipe sideways to see the whole diagram.
SNMP polling. The collector asks every five minutes and gets a counter. The average looks healthy at about a third full, so a 20 second burst that dropped packets never shows. SNMP polls use UDP (user datagram protocol) 161 and traps use UDP 162. I only allow SNMPv3, which adds authentication and encryption.
Streaming telemetry. The router pushes interface counters every ten seconds, or on change, over gNMI. The burst to nearly 100 percent is plain to see, and the drops line up with it. A collector such as Telegraf feeds a time series database and dashboards. A subscription samples on a timer or sends on change. Dial in means the collector connects to the device, and dial out means the device connects to the collector.
Metrics, logs and flows
Metrics come from streaming telemetry and SNMPv3. Flow records from NetFlow, IPFIX (IP Flow Information Export) or sFlow show who talks to whom. Synthetic probes, such as IP SLA (service level agreement) or ThousandEyes tests, measure the path the way a user sees it.
Syslog that can be trusted
Syslog runs on UDP 514, or over TLS (transport layer security) on TCP (transmission control protocol) 6514. UDP can lose messages, so I send to two collectors, keep every device on accurate time and protect the logs from tampering.
Syslog severities
0 emergency, 1 alert, 2 critical, 3 error, 4 warning, 5 notice, 6 informational and 7 debug. I alert on 0 to 3 and keep the rest for investigation.
Pipeline
Every change through the same tested pipeline
This is the pipeline I design for network changes. Code and data meet in a pull request, automated checks run first, people approve what the checks cannot judge, and the network proves the result. If the proof fails, the change rolls back on its own.
Swipe sideways to see the whole diagram.
Every change starts as a branch and a pull request that names the change ticket. Commits are signed by named engineers, and the description explains why the change is needed, not only what it does.
Automated checks run in seconds: yamllint, ansible-lint and ruff for code style, JSON Schema for the data, and a secret scanner so no password or key ever lands in Git.
The pipeline renders the new configuration and shows the diff. Batfish builds a model of the network from the candidate configuration and answers queries, such as whether the payments network can still reach its database. Terraform plan and Ansible check mode show exactly what will change.
People approve what checks cannot judge. A named code owner reviews the pull request, no one approves their own change, the change board approves it in ServiceNow, and the production environment only runs once both approvals exist.
Before touching anything, pyATS takes a snapshot of routes, neighbours and interface counters. If the fabric is already degraded or the controller is out of sync, the run stops. I never change a network that is already broken.
The change goes through the controller where there is one, such as Nexus Dashboard, Catalyst Center or APIC, and straight to devices only for older kit. Each run is scoped to one pod or site, with a rate limit and a kill switch.
The same checks run again. Expected values come from intent, not memory: the number of routes, neighbours and endpoints the data says there should be. Any failure rolls the change back on its own and reopens the ticket.
Diffs, approvals, test results and snapshots go to the change record as one evidence pack. Auditors can see exactly what changed, who approved it and how it was proved.
Try it: which gate catches the mistake?
Swipe sideways to see the whole diagram.
Choose a mistake
Batfish finds the payments subnet can no longer reach its database on TCP 5432. The pull request fails at gate 3 and nothing is deployed.
- YAML typo: caught at gate 2 by yamllint.
- VLAN 5000: caught at gate 2 by the JSON Schema range check.
- A rule that blocks payments: caught at gate 3 by Batfish.
- A change during a freeze: stopped at gate 4 by the change board.
- A leaf already down: stopped at gate 5 by the pre check.
- A faulty optic under load: caught at gate 7, then rolled back.
GitOps
Git holds the intended state and is the only way to change it. A merge is the deploy, and a revert is the rollback. A reconciler or a scheduled job compares the network with the repository, and flags or fixes any difference.
Runners and environments
I specify GitHub Actions or GitLab CI with self hosted runners inside the management network, because cloud runners should never reach devices. A protected production environment needs named approvers. A manual dispatch lets an engineer run an approved job on demand, with the same checks.
Policy as code
Standards are written as tests, not wiki pages: no permit any rules, every interface described, only approved NTP (Network Time Protocol) and syslog servers. Tools such as Open Policy Agent or Batfish assertions run them as a gate on every pull request.
Testing and drift
Proving a change, and keeping it true
A validation framework is different from running a few show commands. The expected values come from intent, the checks are named and versioned, the results are stored, and a failure has an automatic consequence. I design testing in three layers: static checks, a lab that matches production, and the live network.
Before and after with pyATS
from genie.testbed import load
tb = load("testbed.yaml") # generated from NetBox
leaf = tb.devices["leaf101"]
leaf.connect(log_stdout=False)
before = leaf.learn("bgp")
# ... the change runs here ...
after = leaf.learn("bgp")pyats learn bgp interface --testbed-file testbed.yaml --output pre
pyats learn bgp interface --testbed-file testbed.yaml --output post
pyats diff pre postAsk the model with Batfish
from pybatfish.client.session import Session
bf = Session(host="batfish")
bf.set_network("dc")
bf.init_snapshot("snapshots/candidate", name="candidate", overwrite=True)
peers = bf.q.bgpSessionStatus().answer().frame()
down = peers[peers["Established_Status"] != "ESTABLISHED"]
assert down.empty, downBatfish reads configuration files and builds an offline model. An engineer can then ask whether every BGP (Border Gateway Protocol) session will come up, or whether a subnet can still reach a server, before the change exists anywhere real.
Swipe sideways to see the whole diagram.
A scheduled job pulls the running configuration through the device API or the controller, and compares it with what the source of truth and the templates say it should be. Controllers such as Nexus Dashboard and APIC report their own compliance as well.
Drift is any difference between code and the running network. It usually comes from an emergency change at the command line, a replaced device, or a setting edited in a controller but never brought into code.
I class each difference: legitimate but not yet in code, unauthorised, or a gap in the template. The aim is drift that is known, justified and closing, not a number that hides problems.
A legitimate change comes into code through a pull request. An unauthorised one is put back through the pipeline, after checking it is not quietly keeping something running. I never redeploy a whole fabric just to clear the warnings.
A lab that matches production
The same pipeline runs against a lab first: Cisco Modeling Labs or containerlab with the same software versions. A digital twin catches problems a model cannot, such as a parser that changed in a new release.
Rollback is a tested path
Every change carries its rollback, and the rollback is tested in the lab too. A rollback that finishes cleanly and returns the snapshot to its first state is a success, even if the change was not.
Measure the pipeline
I track the share of production changes that go through the pipeline, the false alarm rate of the checks and how long changes wait for approval. The target is every change, except a recorded break glass.
Security
Automation is a privileged user
A pipeline that can change every device is the most powerful account on the network. I govern it like one: least privilege, short lived credentials, two people for every production change, and a trail for everything, including emergencies.
Swipe sideways to see the whole diagram.
The job runs on a self hosted runner inside the network, so it can reach the controllers. Runners for production are labelled, kept to approved repositories and rebuilt for every job.
The runner asks the Git platform for a signed OpenID Connect (OIDC) token. The token states which repository, branch and environment the job belongs to. No password is stored anywhere in the pipeline.
Vault compares the token with a role. Only the main branch, in the protected production environment, of this one repository, may receive production network credentials.
Vault returns a credential for the automation account that expires after 15 minutes, for example a dynamic directory account from its LDAP (Lightweight Directory Access Protocol) secrets engine that TACACS+ (Terminal Access Controller Access Control System Plus) accepts. If it leaks, it is soon useless.
The runner uses the credential on the controller. TACACS+ accounting logs every command against the automation identity, so each pushed line can be traced to a pull request.
- Named code owners. CODEOWNERS files name who must review each part of the repository, and branch protection blocks merges without them.
- Signed and linear history. Signed commits, no direct pushes to main, and no one approves their own change.
- Least privilege. Credentials are scoped to one pod or site, and the automation account can only run the commands its job needs.
- A small blast radius. Each run is limited in scope, with a rate limit and a kill switch, and destructive actions need an extra approval.
- No secrets in code. Passwords and keys live in Vault or CyberArk, are rotated without editing scripts, and never appear in logs.
- Break glass with a trail. In an emergency an engineer may use the command line with a named account. The change is logged, and a pull request brings the code in line the next morning.
Orchestration
From a customer order to a configured network
In service provider and large enterprise networks, automation sits inside a wider orchestration stack. I design to the TM Forum and MEF standards (MEF, first formed as the Metro Ethernet Forum, is now the Mplify Alliance), so ordering, inventory and activation use agreed interfaces instead of one off scripts between systems.
Swipe sideways to see the whole diagram.
The customer orders a service, such as a new SD-WAN site, through the Cantata reference point. Product ordering is a standard API, not an email or a spreadsheet.
The business support systems (BSS) pass the order to service orchestration through Legato. Orchestration, part of the OSS (operations support systems), splits it into parts: an access circuit, an edge device, a policy and monitoring.
Where a partner provides the access circuit, orchestration orders it through Sonata for business requests and Interlude for operational ones, using the same standard APIs.
Orchestration asks each domain controller to deliver its part through Presto: SD-WAN Manager for the edge, the fabric controller for the data centre, an optical controller for transport.
Each controller configures its network elements through Adagio and reports back. Orchestration tracks every part until the whole service is up.
The service inventory is updated and assurance starts watching the service. The customer sees progress and status through Allegro, again through standard APIs.
TM Forum Open APIs
Standard REST APIs for each job, such as TMF622 Product Ordering, TMF641 Service Ordering, TMF638 Service Inventory and TMF640 Service Activation Management. The Open Digital Architecture (ODA) groups them into replaceable components.
MEF Lifecycle Service Orchestration
MEF LSO names the reference points between customer, provider, partner and network, as in the diagram. Its APIs build on the TM Forum ones, so carriers and partners can order and track services automatically.
Autonomous networks
TM Forum describes six levels, from level 0 (fully manual) to level 5 (fully autonomous). Most operators aim for level 3 or 4 in chosen domains, where the network handles known faults within limits people set.
ServiceNow in the loop
The pipeline opens the change request with the plan and the diff attached, waits for approval, and closes it with the evidence. NetBox feeds the CMDB, and a MID (management, instrumentation and discovery) server runs discovery inside the network.
Closed loop assurance
Intent, render, assure. Streaming telemetry feeds analytics, policy decides, and an approved action runs. An SD-WAN path change during a brownout is a closed loop that already runs inside the controller.
Self service catalogue
A ServiceNow catalogue item, such as a new VLAN for an application, asks only for what the requester knows. The pipeline validates it, reserves the VLAN and subnet in NetBox, raises the change and runs the job. Nothing reaches the network outside the checks.
Event Driven Ansible
Rulebooks listen to alerts from monitoring and launch approved job templates: collect diagnostics, bounce a stuck session or open a ticket. Anything risky still waits for a person.
Troubleshooting
Finding faults in the automation itself
Automation fails in its own ways. I read the first failing step in the job log, not the last error, and check the chain in order: identity and reachability, inventory, data, then the device. Pick a symptom.
Swipe sideways to see the whole diagram.
- Likely causes
- The Vault role no longer matches the token after a branch or environment rename, or a rotated password is still cached in the platform. Other causes: a renewed controller certificate the runner does not trust, a runner registered without its label, or a firewall change.
- Fix
- Correct the one failing link the job log points to.
- Prevent
- A daily smoke job that logs in to every platform, plus certificate expiry alerts.
- Likely causes
- The inventory plugin filters it out: status still "planned", a missing tag or role, no primary IP address, or a stale inventory cache.
- Fix
- Correct the data in NetBox and sync the inventory again.
- Prevent
- A NetBox report that fails if an active device lacks its role, platform or primary IP, and no caching for production runs.
- Likely causes
- Raw commands instead of a resource module, or a template that writes a line differently from how the device stores it, such as an abbreviated interface name.
- Fix
- Use resource modules and match the device's own form of each line.
- Prevent
- An idempotency test in Molecule: the second run must report no changes.
- Likely causes
- A resource renamed in code, an attribute that cannot change in place, or the wrong state file or workspace.
- Fix
- Add a
movedblock for renames, or move the resource in the state. Never apply a plan that replaces something you did not mean to touch. - Prevent
- Plan review as a gate, versioned and backed up state, and no hand edits to state.
- Likely causes
- A parser output that changed after a software upgrade, a wrong expected value in the data, a check that ran before the fabric had converged, or a check aimed at the wrong device.
- Fix
- Correct the check or the data, prove it in the lab, then run the change in the next window.
- Prevent
- Schema checks on parser output, expected values counted from the source of truth, and bounded waits for convergence.
- Likely causes
- Command line changes made outside the pipeline, a policy edited but not deployed, a fabric wide setting changed, or a switch replaced or reimaged.
- Fix
- Bring each change into code through a pull request, or put it back, then deploy switch by switch with checks. Never press a fabric wide deploy just to turn the screen green.
- Prevent
- A nightly API job that alerts on anything out of sync.
- Likely causes
- The job limit or a failed task left one member changed and the other not, so a type 1 consistency check fails and VLANs are suspended.
- Fix
- Complete the change on the peer with an idempotent rerun, or roll both back.
- Prevent
- Inventory groups per pair, both members in every run, and a "pair consistent" post check.
- Likely causes
- A confirmation prompt the script did not expect, slow command output, or a changed banner or prompt.
- Fix
- Read the session log, then answer the prompt with
expect_stringor a timing based send, or raiseread_timeoutfor slow commands. - Prevent
- Use an API on any device that has one.
My general method for network faults, with worked examples, is on the troubleshooting page. Terms used here are spelled out in the glossary.
Engagement
How I run an automation programme, from first workshop to steady state
Automation changes how a team works as much as the tools it uses. I start with the people and the changes they make most, prove the data, build a small pipeline that earns trust, then grow it from there.
Swipe sideways to see the whole diagram.
I start with the changes the team makes most often and the ones that cause outages. Each becomes a possible use case with a measure, such as lead time, failed changes or hours saved. Security and change management join from day one, because the pipeline will hold privileged access.
Functional requirements say which changes the pipeline makes and what it checks. Non functional ones set the bar: how long a change may take, how fast a rollback must be, what evidence an auditor needs, and how the platform itself stays available.
I find out what the network really is: platforms, software versions and which devices have APIs. I measure how accurate the inventory and IPAM data are, because automation on bad data makes bad changes faster. I also map today's change process, step by step.
The design covers the data model in the source of truth, the pipeline gates, where credentials come from, how runners reach devices, and what happens when the pipeline itself is down. The design authority reviews it like any other critical platform.
I build one use case end to end, often a low risk change such as VLAN or NTP updates, and run it in the lab and then in production with full evidence. Results measured against the baseline earn the trust to automate riskier changes next.
The pipeline becomes a production service with its own monitoring, on call cover and runbooks. Engineers learn to read job logs, write tests and review code, so the team can run and extend it without me.
Ansible collections, Python libraries, controller APIs and device software all change, often several times a year. I pin versions, test every upgrade in the lab, watch vendor API deprecations, and retire one off scripts as the pipeline replaces them.
Functional requirements
What it must do.
- Make the agreed changes from data in the source of truth, without anyone logging in to devices.
- Check every change before and after it runs, and roll back on its own if a check fails.
- Link every run to a change record and attach the evidence.
- Report drift between intent and the running network every day.
- Let engineers run approved jobs without ever seeing the credentials.
- Offer an emergency route that still leaves a full trail.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Lead time
- A standard change goes from merge to live in under an hour, inside its window.
- Coverage
- Every production change goes through the pipeline, except a recorded break glass.
- Rollback
- An automatic rollback of one site or pod completes within five minutes.
- Auditability
- Every pushed line can be traced to a commit, an approval and a person.
- Availability
- The pipeline survives the loss of a runner or a site, with a documented manual fallback.
- Security
- Short lived credentials, least privilege, signed commits and no secrets in code.
- Scale
- Hundreds of devices per run, within a concurrency limit agreed for each platform.
These are typical starting targets for a first pipeline. The team sets its own numbers in the first workshop, and the pipeline's own tests prove each one.
Who I talk to
- Network engineers, and the people who approve their changes
- Security, identity and privileged access teams
- Owners of NetBox, IPAM, the CMDB and the monitoring tools
- Change managers and auditors
- Vendors for each controller and platform
How I find out
- Change history analysis: which changes are frequent, slow or risky
- Shadowing engineers through a real change
- Data quality checks that compare NetBox and IPAM with the live network
- An API and software version survey across every platform
- Workshops to agree the first use cases and how to measure them
Discovery tools
| Tool or source | What it tells me |
|---|---|
| ServiceNow change history | Which changes happen most, take longest and fail most often. |
| NetBox reports and scripts | Where the source of truth is incomplete or wrong. |
| A read only Nornir or Netmiko audit | Software versions, configuration standards and which devices have APIs. |
| Controller inventories in Catalyst Center, Nexus Dashboard and SD-WAN Manager | What each controller already manages and can automate. |
| Git history of existing scripts | Which one off scripts exist, who wrote them and what they touch. |
| Batfish on today's configurations | Hidden problems in the current configuration, before automation repeats them. |
I troubleshoot automation in two halves. First the pipeline: did the job get its code, data, credentials and a path to the device? Then the network: did the change do what the data said it should?
My approach
- Read the first failure. Open the job log at the first failing step. Later errors are usually knock on effects.
- Check identity and reach. Can the runner get its credentials from the vault and reach the controller or the device API?
- Check the inventory. Is the device in the inventory, with the right group, platform and address?
- Check the data. Does the rendered configuration match what the source of truth says? Run the schema checks again.
- Check the device. Compare the snapshots from before and after. Did the change land, and did the device accept it?
- Fix and harden. Fix the cause, then add a check or a test so the same failure is caught earlier next time.
Faults I see most
- Expired or rotated credentials, or a certificate the runner does not trust
- A device left out of the inventory by a filter or by missing data
- Tasks that are not idempotent and report a change on every run
- Post checks failing after a software upgrade changed command output
- Changes made outside the pipeline that leave controllers out of sync
- Only one member of a pair changed, leaving it inconsistent
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Reachability of the runner and devices, management interfaces and out of band paths. |
| 2 · Data link | Management VLANs and the path from the runner subnet to the devices. |
| 3 · Network | Routes and firewall rules to the controller, with ping and traceroute from the runner. |
| 4 · Transport | SSH on 22, NETCONF on 830 and HTTPS on 443, and TLS certificates checked with curl -v or openssl s_client. |
| 5 to 7 · Application | Authentication, API status codes, payloads checked against the schema, and the device's reply to each call. |
My general method, with worked faults, is on the troubleshooting page.
Once a team relies on the pipeline, an outage of the pipeline is an outage of change. I set it up as a supported service.
Service ownership
A named platform owner, with support hours and escalation agreed with the network operations team.
Monitoring
Alerts on failed jobs, runner health, certificate expiry and secrets access, plus a daily smoke job across every platform.
Runbooks
Guides for the common pipeline failures, and a tested manual fallback for urgent changes.
Change control
Standard changes run through the pipeline with lighter approval, because the checks are built in.
Training
Engineers learn to read job logs, write checks and review pull requests, so the skills stay in the team.
Problem management
Every failed run gets a cause, and a new check or test so it is caught earlier next time.
Automation tools change faster than network hardware. I plan for that from the start.
- Pin. Collections, libraries and container images are pinned to versions, so a run behaves the same next month.
- Test upgrades. New versions run in the lab against the full test suite before production uses them.
- Watch APIs. Vendor API changes and deprecations are tracked, with time to adapt before an old version is removed.
- Review. Quarterly reviews of pipeline use, failure rates and lead time, against the original measures.
- Retire. One off scripts are retired as the pipeline covers their jobs, so there is one way to make each change.
- Refresh. Controllers and runners follow the same patching and end of life rules as any production platform.
Projects
Where I have applied it
My automation work sits inside larger network programmes: designing the architecture, the standards and the controllers that make automation safe.
Network automation architecture
Led discovery, due diligence and architecture for network infrastructure automation aligned to TM Forum and MEF standards, in support of Vodafone's digital customer platforms.
Autonomous network roadmap
Produced high level designs for Virgin Media O2 across RAN (radio access network), core, transport, OSS and BSS (operations and business support systems) using TM Forum frameworks, and fed architecture into its autonomous network and service orchestration roadmap.
Controller led networks
Designed and built LAN and WAN on Catalyst SD-WAN, SD-Access and Catalyst Center, the controller platforms whose APIs appear on this page, and set the design authority standards every regional delivery team adopted.
How I build it
Python, Terraform, Ansible and ServiceNow are part of my core skills, used alongside the controllers and APIs described here.
My CCNP (Cisco Certified Network Professional) Enterprise covers automation and programmability, including Python, REST APIs, NETCONF and RESTCONF. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.