Keep the clock honest
The clock starts when the incident is logged. I accept it fast and give a real first update. It only pauses in states the contract allows, such as waiting for the customer, and I never resolve and reopen to beat a target.
Support and service management
Running a network people can rely on
Building a network is the start. Keeping it healthy for years takes the practices of ITIL, the IT Infrastructure Library, done well: incidents restored fast, causes removed for good, every change controlled, and every device patched and replaced before it becomes a risk.
Practices
Restore fast, then remove the cause
ITIL is the common language of IT support. Version 4 describes 34 practices, and Version 5, which began rolling out in 2026, also has 34. A handful of them decide whether a network is pleasant or painful to run, and I build every handover around them. The figure follows one fault from the first alert to a permanent fix.
Swipe sideways to see the whole diagram.
Monitoring raises an event, such as an SNMP (Simple Network Management Protocol) trap: something changed state. Not every event is an incident. One uplink of a pair failing is worth an incident, often a priority 3, because the site has lost its resilience even though users have not noticed.
The service desk logs one incident, links any duplicates, and sets the priority from impact and urgency. The priority starts the SLA (service level agreement) clocks: time to respond and time to resolve.
The goal of incident management is to restore service, not to find the root cause. A workaround such as a failover or a swapped part is fine. If it needs a configuration change, it goes through an emergency change. The resolve clock stops when the user confirms.
When the same incident comes back, or after any major incident, I raise a problem record. Problem management removes the cause, so the incident stops happening. I never do root cause analysis while users are still down.
Once the cause is found, the problem becomes a known error. I write the symptoms, the cause and the workaround into the known error database, and a knowledge article so second line can fix the next one without escalating.
The permanent fix, such as a software release with the bug fixed, goes through change enablement and is deployed in a window. I watch for repeats for an agreed period before closing the problem with a specific, testable root cause.
The records, with network examples
| Record | What it is | A network example | The goal |
|---|---|---|---|
| Event | A change of state that monitoring sees | An SNMP trap: one uplink of a redundant pair is down | Detect |
| Incident | An unplanned interruption, or a loss of quality or resilience | A floor cannot log in | Restore service |
| Problem | The cause of one or more incidents | A switch stack drops every few weeks | Remove the cause |
| Known error | A problem with a known cause or workaround | A software bug, with a reload as the workaround | Reduce the impact until it is fixed |
| Change | Adding, modifying or removing anything that affects service | An NX-OS upgrade on a vPC (virtual port channel) pair | Change safely |
| Service request | A routine, expected request | Put a user's port in an existing VLAN (virtual local area network) | Fulfil it quickly |
Incidents
Priority sets the clock, so I set it carefully
Priority comes from two things. Impact is how many people and how important the service. Urgency is how fast the damage grows. Pick a cell to see the example targets that go with it. The real figures are always in the contract.
| Impact | Urgency high | Urgency medium | Urgency low |
|---|---|---|---|
| Higha site, many users or a critical service | P1 | P2 | P3 |
| Mediuma team, or resilience lost | P2 | P3 | P4 |
| Lowone user, with a workaround | P3 | P4 | P4 |
Example targets
| Priority | Respond | Update | Resolve |
|---|---|---|---|
| P1 | 15 minutes | Every 30 minutes | 4 hours |
| P2 | 30 minutes | Every 1 to 2 hours | 8 hours |
| P3 | 4 business hours | Daily | 3 business days |
| P4 | 1 business day | When it changes | 5 business days |
A security impact should raise the priority, whatever the user count. If the desk's priority looks wrong, I challenge it early.
Escalate at half time
If half the resolve time has gone and I have no clear cause, I escalate and warn the incident manager. A breach that was flagged early is a managed risk. A surprise breach is a failure.
Two P1s at once
When two priority 1 incidents compete for the same people, I do not choose on my own. The incident manager and the service owners decide the order on business impact, and I work the plan.
Resolved is not closed
Resolved means service is back and the user agrees. Closed means the record is complete: category, cause, linked changes and a problem record if it will happen again.
Major incidents
When it is big, one bridge and one timeline
A major incident is any incident big enough to need its own process: many users, a critical service or several suppliers. The roles matter as much as the technology. Each step below adds the people and the messages that keep it under control.
Swipe sideways to see the whole diagram.
The service desk or monitoring sees the scale and the incident manager declares a major incident early. Declaring one that turns out small costs little; declaring late costs a lot. The technical lead is named straight away.
One bridge, one conversation. The incident manager runs the call and the decisions; the technical lead runs the diagnosis; a scribe keeps the timeline. Engineers take one line of enquiry each, and suppliers join with cases already raised at the right severity.
Updates go to stakeholders and the service desk on a fixed rhythm, in plain English: what is affected, what we are doing, what users should do, and when the next update is. Silence is what makes senior people start ringing engineers.
Restoring service comes before the root cause. The workaround, such as backing out a change or failing over, is announced on the bridge with its rollback, agreed by the incident manager and recorded as an emergency change. Nobody changes anything off the bridge.
Service is restored and confirmed, and monitoring watches it. A post incident review follows within days. It is blameless, it uses the timeline, and every action has an owner and a date. The problem record carries the root cause work.
Several suppliers
When a fault crosses a carrier, a vendor and my own network, I find the seam with evidence: end to end flows, timestamps and correlated data from each side. Each supplier is held to its contract, on the same call.
Time in one zone
Every log and every update uses one agreed time zone, and every device takes its time from NTP (Network Time Protocol). Without that, a timeline across suppliers is guesswork.
Hand over long incidents
A tired engineer makes worse decisions. On a long incident I plan a handover with the timeline, the ruled out causes and the current hypothesis.
Problems
Root cause analysis that names a real cause
"Human error" is never a root cause. It is where the analysis stopped. I choose the technique that suits the fault, and I keep going until the answer is a process, design or software cause I can fix.
Swipe sideways to see the whole diagram.
Five whys. I ask why until the answer is something I can fix in the process. Here the trunk command switchport trunk allowed vlan 30 replaced the whole list, where switchport trunk allowed vlan add 30 was meant. The fix is a standard template and an automatic before and after check of allowed VLANs, not "be more careful". Each answer must be proved, not assumed.
Fishbone. An Ishikawa diagram is good for a group review. It sorts possible causes, such as a missing QoS (quality of service) standard, under people, process, technology, software, environment and suppliers, so nobody fixes on their favourite theory too early. Each candidate is then tested against the evidence.
Fault tree. For resilience failures I draw the top event and the gates beneath it. Two circuits are an AND: both must fail. A single edge router is an OR on its own, and so is a power cut that outlasts the UPS (uninterruptible power supply). Here both circuits failed together because they shared one duct, a common cause the design missed.
Is and is not. The Kepner Tregoe method compares what is affected with what is not, for what, where, when and how much. It is the best tool for intermittent or partial faults. The difference points straight at the cause: a firewall idle timeout shorter than the voice keepalive.
How I know it is the root cause
- It explains every symptom, including the cases that kept working.
- The timeline fits: the cause came before the first symptom.
- Removing it stops the fault, and putting it back brings it back, in a lab if not in production.
- The fix holds: the error counters stay at zero, not just quiet for a day.
- For software, the vendor confirms it with a bug reference and a fixed release.
Known errors and knowledge
- A known error records the symptoms, the affected items, the cause, the workaround and the fix status.
- A knowledge article is written for second line: how to confirm it, the workaround steps and when to escalate.
- I measure success as fewer escalations and a higher first time fix rate.
- Every article has an owner and a review date, and is archived once the permanent fix is proved.
Working the cause with vendors
I bring evidence and, where I can, a lab reproduction. I ask in writing for the root cause, the bug reference, the affected and fixed releases and the workaround, then check the fixed release against our standards and other advisories. I avoid one off engineering builds on critical kit.
Several suppliers, one call
When the cause could sit with more than one supplier, they join one call with shared evidence and actions with owners. Carriers book intrusive tests and loopbacks in a window, and every result goes into the problem record with a timestamp.
When the cause is never found
I record what was ruled out and why, add monitoring that would catch it next time, and keep the problem open. Closing it with a guess only hides the next outage.
Change
Changing a live network without breaking it
Uncontrolled change causes more outages than hardware does. ITIL 4 calls this practice change enablement: the aim is to let safe changes flow quickly and give risky ones the scrutiny they need, not to slow everything down.
Swipe sideways to see the whole diagram.
A request for change (RFC) says what will change, why, when and who will do it. For network work I include the exact commands, the test plan and the backout plan. A plan that says "configure the trunk" is not a plan.
I assess the risk as likelihood times impact, check the change calendar for clashes, and get a peer to review the commands. I confirm the redundancy is healthy, because a change on one side of a pair is only safe if the other side works.
Standard changes are pre approved because they are low risk and done the same way every time. Normal changes are assessed, often at a CAB (change advisory board). Emergency changes go to a smaller emergency CAB (ECAB), which can meet in minutes. Tested, automated changes can earn standard status.
The change is booked into a window agreed with the people affected, outside any change freeze. I set the time by which we must decide to roll back, so the decision is not made under pressure at the end of the window.
I save pre checks, make the change exactly as written, then run the same checks and compare them. If something looks odd near the end of the window, I back out rather than improvise. Unplanned extra changes are how outages start.
The change is closed with its outcome. Incidents it caused are linked, the CMDB (configuration management database) and drawings are updated, and anything learned goes into the templates.
Scoring the risk
I score likelihood and impact from 1 to 3 each and multiply them. A score of 1 or 2 is low, 3 or 4 is medium, and 6 or 9 is high and goes to the CAB with a senior reviewer. A change on a pair with one side already down is never low risk.
A backout plan that works
The backout has exact commands, was tested with the change, and has a decision time. The person who decides is named before the window opens, so nobody argues about it at 3am.
Change freezes
During a freeze, such as year end or a major launch, only emergency changes go ahead, each approved by the ECAB and the service owner. I plan big work around the freeze calendar months ahead.
Unauthorised change
Configuration compares against the last approved backup find changes nobody approved. Each one is investigated as an incident, and as a possible security event, until it is explained.
Service levels
Every promise needs a promise underneath it
A service level agreement is only as strong as the agreements that support it. If a carrier takes eight hours to fix a circuit, nobody can promise a four hour fix for the site. I check the chain adds up before the contract is signed.
Swipe sideways to see the whole diagram.
The service level agreement is the promise to the customer: availability, response and resolve times, support hours and how they are measured. It should describe a service the business recognises, not a list of devices.
Operational level agreements (OLAs) bind the internal teams that make the service work: network, servers and security. Their response times must fit inside the SLA, with time left for the handoffs between them.
Underpinning contracts (UCs) bind the external suppliers: the carrier's fix time, the vendor's hardware replacement and the support desk's hours. They are usually the longest times in the chain.
Here the carrier needs eight hours to fix a circuit, but the site is promised a four hour fix. No amount of effort closes that gap. The answer is design: a second circuit on a diverse path, so a single circuit failure is not a priority 1 at all.
The measures I report
| Measure | What it tells me |
|---|---|
| Availability against target | Whether the service met its promise, measured the way the SLA defines it. |
| MTTA and MTTR | Mean time to acknowledge and to restore. Rising times point at staffing, skills or tooling. |
| First time fix rate | How often the first team fixes it. Good knowledge articles push it up. |
| Repeat incidents | Faults that come back. Each one should have a problem record. |
| Change success rate | Changes completed without an incident or a back out. |
| Incidents caused by change | The most honest measure of how well change is controlled. |
| SLA breaches and near misses | Near misses show where the next breach will come from. |
| Backlog age | Old tickets hide unhappy users and unfixed risks. |
| Capacity headroom | Peak use of links, firewalls and licences against their limits. Capacity and performance management is its own ITIL practice: I forecast from the trend and plan upgrades before the peak, not after it. |
Monitoring
Knowing before the users ring the desk
Good monitoring finds a fault before anyone reports it, and tells me where to look. Each source answers a different question, and the alerts that reach a person must be worth waking up for.
Swipe sideways to see the whole diagram.
SNMP polling. The NMS (network management system) asks each device for counters on UDP (User Datagram Protocol) port 161 every few minutes: interface use, errors, processor load, memory, up or down. It gives trends and baselines. I use SNMPv3 with authentication and encryption, not community strings.
SNMP traps. The device sends a message on UDP 162 the moment something happens, such as a link going down. Traps are fast but can be lost, so polling stays as the safety net.
Syslog. Every device logs to a central server and the SIEM (security information and event management) platform, on UDP 514 or over TLS (Transport Layer Security) on 6514. Severities run from 0, emergency, to 7, debug. I collect 0 to 5 from everything and keep the timestamps aligned with NTP.
Flow records. NetFlow or IPFIX (IP Flow Information Export) records who talked to whom, on which port, and how much. Polling tells me a link is full; flows tell me why. Sampling reduces the load but also the accuracy.
Streaming telemetry. Modern devices push data every few seconds over gNMI (gRPC Network Management Interface), using YANG (Yet Another Next Generation) data models. It catches short bursts that a five minute poll averages away.
Synthetic tests. IP SLA probes or test agents send real traffic on a schedule and measure loss, delay and jitter. They show what users feel, even when every device says it is healthy.
Alerts worth waking for
Every alert that pages a person must need action. I tune thresholds during hypercare, group related alerts into one, and suppress alerts below a device that is already known to be down.
Baselines first
A number only means something next to normal. I record the normal processor load, errors and traffic for each device class, so a change in behaviour stands out.
Watch the watchers
A log source that goes quiet is an alert in itself. So is a certificate or a licence about to expire.
Patching
Not every vulnerability is urgent. Some cannot wait.
A CVE (Common Vulnerabilities and Exposures) entry and its CVSS (Common Vulnerability Scoring System) score do not set the timescale on their own. I ask four questions about my own estate, then decide. Try it below.
How I decide
| Situation | What I do |
|---|---|
| Version not affected, or the feature is off | Record the evidence and the decision. The fix arrives with the next routine release. |
| Being exploited | If an attacker can reach it, an emergency: mitigate today, check for compromise, patch through an emergency change within days. If not, mitigate now and patch within days. |
| Reachable from the internet | Within 5 days, the NCSC timescale for internet facing services. |
| Critical or high (7.0 and above) on an internal device, or any severity on a boundary or core device | Within 14 days, the Cyber Essentials limit, sooner for critical or high on a boundary or core device. |
| Medium or low, internal | The next planned maintenance window, with the exception recorded. |
Update by default
The NCSC's default is to apply every update: internet facing services within five days, operating systems and applications within a week, and internal or isolated systems within about two weeks, faster when a flaw is being exploited. On network kit every upgrade needs a window and a reload, so I follow that default for anything exposed and schedule the rest by risk, with each exception recorded.
Know what I run
Patching starts with an accurate inventory: every device, its model and its software, from the CMDB checked against discovery. I keep an approved "gold" release per platform.
Watch the right sources
Vendor advisories come from the Cisco and Fortinet PSIRTs (product security incident response teams) and from Broadcom for VMware. I also follow NCSC (National Cyber Security Centre) alerts and the KEV (Known Exploited Vulnerabilities) catalogue from CISA (the US Cybersecurity and Infrastructure Security Agency). Cisco publishes IOS and IOS XE advisories in bundles twice a year, and critical fixes out of cycle.
Prove the image
I download only from the vendor portal and compare the hash with the published value: verify /sha512 flash:image.bin on Cisco. On an isolated network the hash is checked again after every transfer. A mismatch means quarantine and report.
Test before production
A reference lab with the same models, software and a copy of the production configuration. I read the release notes for the upgrade path, open caveats and changed defaults, then test boot, neighbours, high availability (HA) failover, AAA (authentication, authorisation and accounting), SNMP and logging.
Stage the rollout
Least critical first, then waves, and never both halves of a redundant pair at once. Every device gets pre and post checks and a soak period before the next wave.
When a patch must wait
If a fix cannot go in yet, I add compensating controls: turn off the affected feature, restrict management access, watch harder. The named risk owner signs the decision with a review date, not the engineer.
Isolated networks
On a network with no internet path, images and signature updates arrive through an approved transfer route and are hash checked again on arrival. Tests take longer without vendor cloud services, and the accreditor or risk owner is involved from the start.
A critical advisory on a Friday
Friday: read the advisory, confirm exposure from the CMDB, mitigate under an emergency change and check for signs of compromise. Saturday: test the fixed image in the lab. Sunday: patch through the ECAB, standby units first. Monday: report, update the baseline and review. Engineers work in shifts so nobody patches tired.
Point patches and parts
A Cisco SMU (software maintenance upgrade) fixes one bug or advisory without a full upgrade, and still goes through change. A FRU (field replaceable unit), such as a power supply, fan, line card or optic, is swapped on site. I record part numbers and serials from show inventory for the return.
Prove compliance monthly
A monthly report compares the version on every device with the approved baseline. Anything off baseline gets a ticket or a recorded exception, so drift cannot hide.
Upgrades
Upgrading a pair without dropping traffic
Most network devices that matter come in pairs. The rule is the same on every platform: upgrade one half while the other carries the traffic, prove it, then swap. ISSU (in service software upgrade) automates this on platforms that support it.
Swipe sideways to see the whole diagram.
I back up the configuration off the box, copy the image to both units and verify its hash, and check the vendor's supported upgrade path. Some versions need an intermediate release first. I confirm the pair is healthy and in sync before starting.
The standby unit is upgraded and reloads while the active unit carries every session. Users see nothing.
The standby comes back on the new release. I check that HA (high availability) is in sync, the configuration matches and sessions are being copied across.
I fail over in a controlled way. The upgraded unit becomes active and existing sessions carry on, because their state was shared.
The old active unit, now standby, is upgraded the same way while the new active carries the traffic.
Both units run the new release and are in sync. I run the post checks, decide whether to fail back to the original roles, and close the change with the evidence.
Upgrade playbooks by platform
Cisco Nexus vPC pair
- Read the release notes and the ISSU support for the hardware. Copy the image to both switches and check the hash with
show file bootflash:image.bin sha512sum. - Check the impact first:
show install all impact nxos bootflash:image.binshows whether each module upgrades without disruption. - Capture the state:
show vpc brief,show vpc consistency-parameters global,show vpc orphan-portsandcheckpointthe configuration. - Upgrade one peer at a time with
install all nxos bootflash:image.bin. Wait for the peer link, keepalive and every vPC to come back before the second peer. - Single attached devices on orphan ports lose service when their switch reloads, so I move or warn them first.
Catalyst 9300 stack
- Check the boot mode with
show version. Install mode is recommended on IOS XE; bundle mode boots the image file directly. - The whole stack runs one control plane, so the whole stack reloads. I book a window and tell the users on those floors.
install add file flash:image.bin, theninstall activate auto-abort-timer 60, so the stack rolls back by itself if I cannot reach it.- Run the post checks, then
install committo keep the new release.show install summaryconfirms the state, andinstall remove inactivefrees the space once the release is proven.
Catalyst VSS (Virtual Switching System) pair with eFSU
- Check that the two releases are compatible for eFSU (enhanced fast software upgrade). If not, the upgrade is a full reload in a window. Have console access to both supervisors and know the abort command.
issu loadversionreloads the standby chassis on the new image.issu runversionswitches over, so the upgraded chassis becomes active.issu acceptversionstops the rollback timer, thenissu commitversionreloads the old active on the new image.show issu state detailconfirms each stage.- Devices dual homed with MEC (multichassis EtherChannel) stay up throughout. Single attached devices lose service while their chassis reloads.
A remote router with no engineer on site
- A single router means an outage, so I confirm an out of band path first: a console server, a cellular link or a second circuit, or a person on site with a console cable.
- Back up the configuration off the box, check flash space, copy the image and verify it with
verify /sha512 bootflash:image.bin. - Keep the old image as a fallback:
no boot system, thenboot system bootflash:new.bin, thenboot system bootflash:old.bin, thenwrite memory. - Reload, and if it does not return within the agreed time, the out of band path and the fallback image recover it.
Cisco ASA (Adaptive Security Appliance) failover pair
show failovermust show one Active and one Standby Ready on the same version.- Copy and verify the image on both units. Set the new image first with
boot system, keep the old one as a second entry, thenwrite memoryon the active, which saves on both. failover reload-standbyupgrades the standby. Wait for Standby Ready.- Fail over with
no failover activeon the active unit, so the upgraded unit takes over. - Reload the old active, now standby, with
failover reload-standbyfrom the new active, and checkshow failoveragain.
FortiGate HA cluster
- Follow Fortinet's upgrade path tool exactly, including any intermediate releases, and upgrade FortiManager first.
- Confirm the cluster is in sync with
diagnose sys ha checksum clusterand back up the configuration. - Under
config system ha,set uninterruptible-upgrade enableandset session-pickup enable. - Load the firmware on the primary. The cluster upgrades the secondary units first, then fails over and upgrades the primary.
- Check the checksums match afterwards. If they differ, I compare them per table rather than forcing a sync blindly.
Palo Alto HA pair
- Upgrade Panorama first, to the same or a later release. Check the upgrade path and the minimum content version for App-ID and threat updates.
- Save and export the configuration, and turn off preemption so roles do not flip unexpectedly.
- On the passive unit:
request high-availability state suspend, install the software,request restart system, thenrequest high-availability state functional. Check sync. - Suspend the active unit to fail over, then upgrade it the same way. Turn preemption back on and check sessions, HA and logs.
Juniper SRX chassis cluster
- Redundancy group 0 holds the control plane and group 1 and above the data plane. The method depends on the model and release, so I read Juniper's notes first.
- On platforms that support it, an in service upgrade keeps traffic flowing:
request system software in-service-upgrade. - Otherwise I use the minimum downtime method: upgrade the secondary node, fail the redundancy groups over to it, then upgrade the original primary.
- Check
show chassis cluster statusand the session counts before and after each step.
Forcepoint next generation firewall (NGFW) and older Sidewinder firewalls
- Upgrade the SMC (Security Management Center) first, after backing up the management server and the configuration.
- Upgrade the engines from the SMC, one cluster node at a time, on a supported upgrade path. Check traffic after each node before moving on.
- Reinstall and verify the policy, then check
sg-clusterandsg-versionon each engine. - Legacy Sidewinder firewalls are upgraded through their own management console. I confirm the support route and end of life dates with the vendor, and record them in the obsolescence register.
VMware NSX
- Check the interoperability matrix for NSX, vCenter and ESXi, and take a verified NSX Manager backup.
- Upload the bundle and upgrade the upgrade coordinator. The pre checks must pass before anything else.
- Upgrade the Edges in groups, so traffic moves to the Edges not being upgraded.
- Upgrade the hosts, each going into maintenance mode while its virtual machines move away, then upgrade the NSX Managers last.
End of life
Retiring kit before it retires itself
Every device has a published end of life. I track those dates in a register and plan refreshes years ahead, because on a secure network the real deadline comes well before the last date of support.
Swipe sideways to see the whole diagram.
The vendor announces end of life and publishes every milestone date for each product ID. I match the bulletin to the CMDB and add every affected device to the obsolescence register, two to three years ahead.
The last day to order. If a refresh will take years, as it often does where design, approvals and procurement take years, I buy tested spares now.
No more bug fixes. I settle on the most stable release and keep the refresh budget moving.
No more security fixes. For anything at a boundary or in the core, this is the date that matters. Past it, every new advisory is a risk I cannot patch.
No vendor support, no replacement parts. Kit still running now needs a signed risk acceptance, compensating controls, a locked down management plane, and a pool of spares pre loaded with the approved image and configuration. Third party maintenance can cover hardware, but not security fixes.
Where some common platforms stand
| Platform | End of sale | Last date of support | What it means now |
|---|---|---|---|
| Catalyst 3850, copper models | 30 Oct 2020 | 31 Oct 2025 | Past support. Replace, or hold a signed risk with controls. |
| Catalyst 3850, fibre models | 30 Apr 2022 | 30 Apr 2027 | Months left. The refresh should already be in build. |
| Nexus 7004 chassis | 28 Feb 2022 | 28 Feb 2027 | Months left. Move to Nexus 9000, and redesign virtual device contexts, OTV (Overlay Transport Virtualization) and FabricPath on the way. |
| ISR (Integrated Services Router) 4200, 4300 and most 4400 models | 7 Nov 2023 | 30 Nov 2028 | Bug fixes ended in August 2025. Security fixes continue until the last date. |
| NSX for vSphere | No longer sold | 16 Jan 2023 | General support ended in January 2022 and technical guidance a year later. Migrate to NSX (formerly NSX-T). |
What my obsolescence register holds
- The configuration item, model and quantity, and where it is
- Every milestone date for that product
- How critical it is and how exposed it is
- The refresh plan, its budget and its owner
- A link to the risk register if a date will be missed
I sort the refresh by criticality and exposure: boundary and core first, then distribution, then access.
Planned preventive maintenance (PPM)
- Daily. Alarms, critical logs and the configuration backup job.
- Weekly. Interface errors and discards, utilisation, processor and memory trends, HA and stack health.
- Monthly. Software against advisories, licences and certificates, firewall rule hit counts.
- Quarterly. Restore a backup, test failover, review capacity and the end of life register.
- Yearly. Rule recertification, a site walk of power, cooling and cabling, and a design review.
I record the values, not just ticks, so I know what normal looks like.
Records and handover
If it is not recorded, it cannot be supported
The CMDB links every service to the devices and suppliers under it. That link is what turns "the core switch is down" into "payroll and the contact centre are down", and what tells a change board who to warn.
Swipe sideways to see the whole diagram.
Each configuration item is a service, an application, a server, a device or a circuit, and the CMDB records what depends on what. Discovery tools keep it honest: a CMDB nobody checks against the network is fiction within months.
When the core switch fails, the relationships show the impact at once: payroll and the contact centre. The incident gets the right priority from the first minute, and the right owners hear about it.
The same relationships tell the change board that a firewall change touches the contact centre only. One owner is warned and the window is agreed with them, instead of everyone or no one.
Configuration backups
Every device and controller is backed up on every change and every day, versioned, stored outside its own failure domain, and restored in a test now and then. Backups hold secrets, so they are protected like secrets.
Handover to operations
Before a service goes live I hand over the high and low level designs, as built drawings, runbooks, monitoring, the support model and the known errors, then agree operational acceptance.
Managing suppliers
Contracts that match the SLA, regular service reviews with real data, clear escalation contacts, and a design change when a supplier cannot meet what the service needs. Where many suppliers run one service, SIAM (service integration and management) gives one integrator ownership of the end to end service and the handoffs between them.
The tools
I run these practices in ServiceNow. They work the same in other tools, such as BMC Helix (formerly Remedy) or Jira Service Management, as long as incidents, problems, changes and configuration items are linked.
Continual improvement
Lessons from incidents and reviews go onto an improvement list with owners. A recurring fault becomes a design change, a monitoring gap becomes new telemetry, and a supplier weakness becomes a contract change.
Documentation people use
Runbooks written for the engineer at 2am: the symptom, how to confirm it, the steps, when to escalate, and a review date so they do not go stale. They live in one shared place, such as a wiki like Confluence, linked from the CMDB record.
Projects
Where I have run it this way
Change governance, supplier management and handover have been part of every programme I have worked on, in banking, industrial software, publishing and managed services.
Change governance in a regulated bank
Advised a blended team of 5 engineers and 4 vendors on change governance and architecture standards. We replaced end of life equipment across Europe, the Middle East and Africa with minimal impact on global operations, and kept the estate compliant with MiFID II (the Markets in Financial Instruments Directive) with no service outages.
Uptime through transition
Safeguarded over £200M of transactional services through data centre transitions at 99.99% uptime, leading 7 engineers and architects and coordinating 5 vendors across design, deployment and operations.
Discovery to operational handover
Lead architect on network, cloud and data centre modernisation programmes from discovery to operational handover. I planned and ran remote site migrations across regulated environments with no critical incidents.
Support costs after a merger
Delivered £700k a month in operational savings after the merger by rationalising infrastructure, support and technology overheads.
Managed infrastructure
A Tier 3 data centre and managed infrastructure for over 300 customer sites, on Cisco ACI (Application Centric Infrastructure) and Nexus.
ITIL and PRINCE2
ITIL certified and a PRINCE2 (Projects in Controlled Environments) Practitioner, so service practices and controlled project delivery share one vocabulary in my work.
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.