Start from the business
I ask which services would cause real harm if they stopped, and for how long that harm is tolerable. The technology target follows from that answer.
Resilience, risk and cost
Resilience you can measure, at a cost you can defend
Availability is a business decision with a price. I turn it into numbers: downtime per year, recovery times, the shared services everything depends on, the rules a regulated firm must meet, and what each unit of capacity really costs.
Overview
What fails, and what then happens
Every resilience review I run comes back to that question. A service is a chain of parts. Any part that stands alone can stop the whole service, and pairs only help if the two halves cannot fail for the same reason. Each extra layer of resilience also costs money, so the business needs to see the price before it picks a target.
Swipe sideways to see the whole diagram.
Users look up the name in DNS (the domain name system), then traffic passes a front door, an application, a database and storage. Every part stands alone, so any one of them can stop the service.
The database fails. The other parts are healthy, but the service is down, because a chain is only as strong as its weakest link.
A second application, database and storage are added. Traffic goes round the failed database, and users notice nothing more than a short pause while the replica takes over.
Both halves still use the same DNS. When it fails, users cannot find the service at all. Pairs only help if the two halves cannot fail for the same reason.
Everything runs again, but every pair has a price: hardware, licences, power, space and the people to run it. That is why the business sets the target with the cost in front of it.
Tier, do not gold plate
Not everything needs five nines. I tier services by criticality, so money goes where an outage would hurt most.
Show the price
Each option comes with its availability, recovery time and cost, including the cost of doing nothing. The business then chooses knowingly.
Availability
What each nine really allows
Availability is the share of time a service works. Each extra nine cuts the allowed downtime by ten, and each one usually needs a bigger step in design: more redundancy, a second site, automated failover and changes made without downtime. Pick a target to see what it allows.
Swipe sideways to see the whole diagram.
| Down per year | 52.6 minutes |
|---|---|
| Per 30 day month | 4.32 minutes |
| Per week | 1.01 minutes |
| What it usually takes | Two sites running at once, automated failover, and most changes made without downtime. |
A year is taken as 365 days and a month as 30 days. The design notes are typical patterns, not rules: the real answer depends on the service and its dependencies.
| Target | Down per year | Per 30 day month | Per week | What it usually takes |
|---|---|---|---|---|
| 99% | 3.65 days | 7.2 hours | 1.68 hours | One site, single parts, support in office hours. |
| 99.9% | 8.76 hours | 43.2 minutes | 10.1 minutes | Redundant power, network and servers in one site, with support out of hours. |
| 99.95% | 4.38 hours | 21.6 minutes | 5.0 minutes | N+1 at every layer, tested failover and monitoring that pages someone. |
| 99.99% | 52.6 minutes | 4.3 minutes | 1.0 minute | Two sites running at once, automated failover, and most changes made without downtime. |
| 99.999% | 5.26 minutes | 26 seconds | 6 seconds | Several sites, no shared single dependency, and every change and patch done live. |
From need to target
I turn the business need into a service level objective (SLO), a measured target such as 99.95% a month. The service level agreement (SLA) with customers then sits a little below it, so there is room for error.
Tier the estate
Across a large estate I use a few site tiers. Hub sites get two diverse circuits and paired kit. Small sites get one circuit with 4G or broadband backup. Each tier has its own target and is measured against it.
Measure what users see
Availability is measured at the service, from the user's side, not by adding up green lights on devices. A network can be up while the application is not.
Series and parallel
How parts combine into a service
Parts in a chain multiply: three parts at 99.9% each give about 99.7%, so the service is worse than any one part. Parts in parallel do the opposite: a pair of 99.9% parts is down only when both are, which is about 99.9999%. That maths assumes the two halves fail independently, and that is the assumption I test hardest.
Swipe sideways to see the whole diagram.
| Service | 99.70% |
|---|---|
| Down per year | about 26.25 hours |
| Sum | A × A × A |
In series. Three tiers, one of each. The service only works when all three do, so the availabilities multiply and the result is lower than any single part.
In pairs. Each tier has two members and either will do. A tier is only down when both members are down at once, so each tier gets much better, and so does the service.
Shared dependency. The pairs are still there, but both halves need the same DNS. That single service sits in series with everything, and drags the result back down to its own level.
Shared fault. Both application servers run the same firmware and hit the same bug on the same day. The pair behaves like one part, and the gain from pairing disappears. I stagger firmware and changes across pairs for this reason.
Active and active
Both halves carry live traffic, so failover is near instant and each half is proven every day. Each half must be able to carry everything: two sites at 70% load cannot absorb each other.
Active and standby
Simpler for stateful systems such as some firewalls and databases, and often cheaper to license. The standby must be tested on a schedule, or it fails when it is finally needed.
Faults that hit both halves
The same firmware bug, the same bad change pushed everywhere, or certificates that all expire on one day. I stagger upgrades, never change both halves together and keep a certificate inventory.
Detect and repair
Time between failures, and time to recover
Two numbers drive availability: how often something fails, the mean time between failures (MTBF), and how long it takes to put right, the mean time to restore (MTTR). Inside the repair time sits the mean time to detect (MTTD), which is often the easiest part to shorten.
Swipe sideways to see the whole diagram.
The service runs normally until a part fails. From this moment the clock is running, whether anyone knows about it or not.
Detection is the gap between the failure and the first alert a person acts on. Good monitoring, tuned alerts and synthetic tests that act like a user make this gap short.
The fault is diagnosed and fixed, then the service is checked and restored. The whole outage, from failure to restore, is the mean time to restore. The service then runs until the next failure.
Availability is MTBF divided by MTBF plus MTTR. With failures every 2,000 hours and four hours to restore, the service reaches 99.80%. Cutting the restore time to one hour lifts it to 99.95%, with no new hardware. Here MTBF is the uptime from one restore to the next failure, as ITIL defines it.
Shorten detection
Alerts that page a person, synthetic checks that behave like a user, and dashboards per service rather than per device. Faster detection is often the cheapest nine.
Shorten repair
Runbooks, spares on site, out of band access and automated failover. A fix that needs one specialist who is asleep is not a fix.
Raise the gap between failures
Better parts, redundancy, patching and fewer risky changes. This usually costs the most, so I check the cheaper levers first.
Cold start
Bringing everything back in the right order
After a full power loss, everything has to start again in order. Circular dependencies hide until that day: the key service that runs on the encrypted cluster that needs its keys, or the DNS server behind a firewall whose management needs DNS. I document the start order, put the foundations on their own small platform, and prove it with a timed cold start.
Swipe sideways to see the whole diagram.
Power and cooling come first, then the out of band (OOB) management network that lets engineers reach every device, then time. Everything after this needs accurate clocks.
DNS starts next, then identity: the directory, the authentication servers for network devices, TACACS+ (Terminal Access Controller Access Control System Plus) and RADIUS (Remote Authentication Dial In User Service), and privileged access management (PAM).
Key management starts, so encrypted systems can start, then the network fabric comes up and builds its routes.
Here the key service runs as a virtual machine on the encrypted cluster that needs its keys. The cluster waits for the keys, and the keys wait for the cluster. Neither starts. Loops like this only show up in a real cold start.
The fix is to move time, DNS, identity and keys onto their own small platform, often called pod zero, which depends on nothing above it. The loop is gone, and platforms and workloads start in order. I prove the whole sequence with a timed cold start.
Regulation
The rules a regulated firm designs around
In financial services, resilience is not optional. The UK and EU rules below shape the infrastructure I design for regulated firms: which services must survive, how fast incidents are reported, how suppliers are managed and how often defences are tested. I treat each rule as a design input with evidence behind it. The governance side is on my design authority page.
Swipe sideways to see the whole diagram.
The Financial Conduct Authority (FCA) and Prudential Regulation Authority (PRA) ask firms to name their important business services, set an impact tolerance for each, map what each depends on and test it. The deadline to stay within tolerance was 31 March 2025.
The PRA's supervisory statement SS2/21 covers outsourcing and third party risk: due diligence, contract rights, business continuity and exit plans, including what happens if a supplier fails suddenly.
The EU Digital Operational Resilience Act (DORA) covers risk management for information and communication technology (ICT), incident reporting, resilience testing, third party risk and information sharing. It has applied since 17 January 2025, and it lets EU supervisors oversee critical ICT third party providers directly. The European supervisors named the first of these on 18 November 2025.
The Payment Card Industry Data Security Standard (PCI DSS) version 4.0.1 sets the controls for any system that stores, processes or sends card data. Its future dated requirements, such as multi factor authentication (MFA) for all access into card systems, are now mandatory.
Since 1 January 2025 the UK can bring a key supplier to the whole financial sector under direct oversight. The first four, all cloud and technology providers, took effect on 13 July 2026. Each firm still owns its own resilience and exit plans.
New UK rules from 18 March 2027 standardise operational incident reports and require notice of material third party arrangements. I design monitoring and logging so the facts for a report are ready within hours, not days.
Swipe sideways to see the whole diagram.
Monitoring raises the incident and the firm classifies it against DORA's criteria, such as clients affected, duration, data lost and services hit. Classification must be quick, so the facts need to be on hand.
Within four hours of classifying it as major, and no later than 24 hours after becoming aware of it.
Within 72 hours of the initial notification, and again whenever the situation changes materially.
Within one month of the latest intermediate report, with the root cause, the impact and the fixes. A UK firm may also owe reports on UK timelines for the same incident.
Swipe sideways to see the whole diagram.
Within tolerance. The impact tolerance is a business measure, always a length of time. The IT recovery time objective (RTO) sits well inside it, leaving room to detect the problem, decide and restore the full service.
Breached. Detection was slow, and recovery waited on the slowest dependency, such as a restore from backup. The service came back after the tolerance line. An RTO that equals the tolerance leaves no margin at all.
Impact tolerance is not an RTO
A tolerance is the most disruption a service can take before harm is intolerable to customers or the market. It covers people and process too. I set technical recovery targets well inside it.
Concentration and exit
Too much on one provider, site or technology is a concentration risk, and being unable to leave is an exit risk. DORA and SS2/21 both ask for exit plans. I use open standards at the boundaries so leaving is possible.
A regulated high level design
For a regulated firm, the high level design maps each important business service to its pods, sites and suppliers, states its impact tolerance and recovery time objective, and names the evidence each control will leave.
Hidden concentration
Two suppliers can still share one weak point: the same firmware, the same optics maker, the same building or the same fibre route. I check for these during discovery, not after an outage.
PCI and clocks
Card data scope, and clocks you can trust
Two rules shape the network more than most. PCI DSS draws a boundary around the cardholder data environment (CDE), and everything that can reach it joins the scope, so segmentation decides the cost of compliance. The Markets in Financial Instruments Directive (MiFID II) sets how closely trading clocks must follow Coordinated Universal Time (UTC), which drives time design down to the network card.
Swipe sideways to see the whole diagram.
Flat network. Any system can reach the card database, so every system is in scope for the assessment. That makes every control, scan and review apply to the whole estate.
Segmented. Only the payment systems sit in the CDE. Systems that connect to it or can affect its security, such as the directory, jump hosts, logging and backup, are in scope too. The office, email and development are out of scope. PCI DSS does not demand segmentation, but it is the biggest lever on scope.
Shared tool. Office PCs can reach the jump host that also manages the CDE. That one path pulls the office back into scope. Shared management tooling is the most common way scope leaks, so I give the CDE its own management path.
From rule to control to evidence
| Rule | Design control | Evidence I keep |
|---|---|---|
| DORA Article 8, PCI DSS 12.5.1 | Every asset recorded in the configuration management database (CMDB) from the factory, tagged with the services it supports. | CMDB reports reconciled with discovery scans. |
| FCA and PRA impact tolerances | Important services placed across two sites, with recovery targets inside each tolerance. | Scenario test results against each tolerance. |
| DORA Article 9, PCI DSS 1.3 and 1.4 | A routing table (VRF) per zone, deny by default, and only named flows allowed. | Firewall rule reviews at least every six months (PCI DSS 1.2.7) and flow tests. |
| PCI DSS 11.4.5 and 11.4.6 | Segmentation that can be tested automatically. | Segmentation tests at least every 12 months and after changes, or every six months for service providers. |
| DORA Article 10, PCI DSS 10 | Central logging to the security information and event management system (SIEM), with logs that cannot be altered. | Logs kept 12 months, with three months on hand (PCI DSS 10.5.1). |
| PCI DSS 10.6.1, MiFID II regulatory technical standard (RTS) 25 | Redundant time sources, with the Precision Time Protocol (PTP) for trading systems. | Time offset monitoring and alarms. |
| DORA Articles 11 and 12 | Backups isolated physically and logically, and immutable. | Scheduled restore tests per service. |
| PCI DSS 8.4.2 | Multi factor authentication for all access into the CDE. | Access and authentication reports. |
| PCI DSS 6.3.3 | Critical security patches applied within one month of release. | Patch compliance reports. |
| DORA Article 28, PRA SS2/21 | A register of third parties, with exit plans for the material ones. | The register, and tested exit plans. |
| DORA ICT risk standards, Delegated Regulation (EU) 2024/1774 | A full key lifecycle, and a register of certificates and the devices that hold them. | Key and certificate records. |
Swipe sideways to see the whole diagram.
Trading time. MiFID II and its clock standard, RTS 25 (Delegated Regulation (EU) 2017/574, kept in UK law), require business clocks traceable to UTC. For high frequency algorithmic trading that means within 100 microseconds, with timestamps to 1 microsecond. Other algorithmic trading gets 1 millisecond.
The Precision Time Protocol (PTP, IEEE 1588) is the usual way to get there reliably. Every switch on the path must take part as a boundary or transparent clock, and the paths must be symmetric, or the delay maths is wrong. Satellite signals can be jammed or spoofed, so I add a terrestrial source and clocks that hold time on their own. General servers only need NTP.
Testing
Proving recovery one rung at a time
A resilience design is only a claim until it is tested. I build a ladder of tests, from pulling one cable in the factory to an attack simulated by a specialist team. Every test produces evidence: the plan, the expected result, what really happened, the issues and the fixes. The evidence pack for each pod is described on my design authority page.
Swipe sideways to see the whole diagram.
In the factory, before anything ships, I pull power feeds, links, switches and disks, and check that every redundant part takes over as designed.
The whole pod is powered off and on again, and each stage is timed. This is the only test that finds circular dependencies, so I repeat it in production on a schedule.
Each service is failed over to its second site and back, by the people who would do it for real, to prove the recovery time objective (RTO) and recovery point objective (RPO).
Severe but plausible scenarios, such as a cyber attack, the loss of a site or a key supplier failing, are tested against each impact tolerance, including the people and processes, not only the technology.
Threat led penetration testing (TLPT) uses intelligence about real attackers. DORA requires it at least every three years for firms in scope, and the UK's versions are CBEST and STAR-FS (simulated targeted attack and response for financial services). It runs on live systems, so they must be safe to test and log every step.
Risk and debt
Risks scored against appetite, and debt with an owner
I keep an architecture risk register alongside every design: end of life kit, single points of failure, capacity ceilings, security exposure, supplier concentration and regulatory deadlines. Each risk is scored for likelihood and impact, compared with the organisation's risk appetite, then treated or knowingly accepted by the right person.
Swipe sideways to see the whole diagram.
Assess. End of life switches get no more security fixes once support ends, and they carry the core. Likelihood 4 times impact 5 gives 20 on a five by five scale. End of support is the textbook risk you can see coming years ahead.
Compare with appetite. The organisation decides how much risk it will carry. Here nothing above 9 is acceptable on critical services, so R1 must be treated.
Treat. A refresh is funded and planned, and meanwhile extended support and tighter management access reduce the exposure. The risk falls to 8, within appetite, with an owner and a date.
Accept. Not every risk is worth fixing. One circuit at a small site with 4G backup scores 6. The owner accepts it in writing, with a review date, so the decision is visible rather than forgotten.
A technical debt register
Technical debt is any shortcut or ageing part that will cost more later. The goal is not zero debt. It is debt that is known, justified, owned and on a path to closing. These rows are illustrative.
| Item | Why it exists | Risk if left | Fix and cost | Owner and review |
|---|---|---|---|---|
| Core switches past end of sale | Refresh deferred last year to fund a migration. | No fixes after end of support, spares harder to find. | Refresh in two waves, budgeted next year. | Head of infrastructure, quarterly. |
| Firewall rules with no owner | Rules added during incidents and never removed. | Wider access than needed, failed audit. | Rule review and clean up, about ten days' effort. | Security lead, every six months. |
| Manual site builds | No templates when the estate was small. | Drift between sites, slower fault finding. | Templates and a source of truth, one quarter. | Network lead, monthly. |
| Time exception on a legacy system | Vendor does not support NTP authentication. | Spoofed time could corrupt logs. | Isolate it, then replace at contract end. | Service owner, at contract end. |
Unit cost
What one unit of capacity really costs
A unit cost is the full yearly cost of a platform divided by the units it can sell at its target utilisation: a standard virtual machine (VM), a terabyte, a graphics processing unit (GPU) hour. I divide by what can be sold, not by what is installed, and I state what is included. After a merger, I build one unit cost per platform across both estates, so duplicate platforms show up as idle, expensive units. Try the model below. The figures are indicative, not a quote.
Swipe sideways to see the whole diagram.
VM pod
CPU ratio
GPU cluster
Assumptions: 64 cores and 1 TB of memory per host, and a standard VM of 4 virtual central processing units (CPUs) and 16 GB. Two hosts in every cluster of 16 are held in reserve. Each host costs £60k with its network share and draws about 1 kW. A rack holds 12 hosts and costs £15k a year. Licences are priced per core per year. Power is £0.20 a kWh with a power usage effectiveness (PUE) of 1.4, and the operations team share is a fixed £200k. Indicative only.
Swipe sideways to see the whole diagram.
Balanced. Processor, memory, power and space are used at about the same rate, so the pod fills evenly and every pound spent can be sold.
Memory runs out. The hosts have processor cores to spare, but no memory, so no more virtual machines fit. The spare cores are stranded: paid for but unusable. Raising the CPU ratio from 4:1 to 6:1 in the model above shows the same effect.
Power runs out. In a dense GPU hall the power budget is used up long before the floor is full. Rack space is stranded by design, which is why GPU capacity is often costed per kilowatt as well as per GPU hour.
TCO
Total cost of ownership, including what gets missed
Total cost of ownership (TCO) is build plus run plus lifecycle plus exit, usually over five years. The purchase price is rarely the biggest part. Licences, people, power and the cost of leaving often are, and they are the lines most often missed.
Swipe sideways to see the whole diagram.
Servers, switches and storage, plus optics and a share of the cabling. Optics are easy to leave out, and at 400G they are a real cost.
For a virtualisation pod, software licences are often the biggest line, especially where they are charged per processor core.
Vendor support and maintenance contracts, which grow once kit passes its standard support period.
Space, power and cooling. Power is the IT load multiplied by the power usage effectiveness (PUE): at a PUE of 1.4, each kilowatt of IT needs another 0.4 kW for cooling and losses.
The engineers who run it, and the monitoring and automation tools they need. This is the line most business cases forget.
Moving workloads in, running old and new side by side, then leaving one day: data return, exit help and certified destruction of drives. About two thirds of the total is running cost, not purchase price.
Capital or operating cost
Capital spend buys kit and depreciates it, which is cheapest over its life at high use. Operating spend covers subscriptions, support, power and people, and scales faster. Finance decides the treatment, and some as a service deals count as leases.
Pay as you use
Consumption models such as HPE GreenLake, Dell APEX and Pure Evergreen//One place spare capacity on site and bill what is used above a commitment. I buy for the steady base and consume for uncertain growth. In a bank, such a deal is also a third party arrangement to register.
Showback, then chargeback
Showback tells each team what it consumes without billing it. Chargeback bills the cost centre. Showback alone usually removes idle and oversized machines, so I start there, with accurate metering and clear ownership in the CMDB.
Capacity
When to order the next pod
New capacity takes months to arrive: ordering, delivery, factory build, burn in, installation and acceptance. So I order when free capacity falls to the demand expected during that whole lead time, plus a buffer for failover headroom and forecast error. Move the sliders to see the trigger and the order date.
Swipe sideways to see the whole diagram.
| Order trigger | 1,600 free units |
|---|---|
| Order in | week 23 |
| New pod arrives | week 43, with 400 units still free |
The pod starts with 3,000 free units and a new pod adds 2,688. Trigger = demand a week × (lead time + build time) + buffer. Indicative only.
Separate triggers for long lead parts
GPUs and high speed optics can take far longer than servers. I give them their own, earlier trigger, so one late part does not hold up a whole pod.
Track per pod type and site
A trigger for the whole estate hides a site that is about to run out. I track each pod type in each site against its own forecast.
A buffer for the unexpected
For sudden demand, I keep a buffer pod or a standing order that is approved in advance, so a spike does not wait for a full procurement cycle.
Business case
Options a finance team can decide on
My input to a business case is a short set of options, each with its architecture, cost to build and run, resilience, risk and lifespan, and always the cost of doing nothing. I lead with the decision needed, state the weak points first, and show which assumptions move the answer most.
| Option | Build cost | Run cost | Resilience | Main risk | Lifespan |
|---|---|---|---|---|---|
| Do nothing | None | Rising: extended support, field visits | Falls as kit ages | Unfixed security flaws, long outages waiting for spares | Ends with support |
| Like for like refresh | Medium | Similar to today | Same design, newer kit | Old design faults carried forward | About seven years |
| Standardise and automate | Higher | Lower: fewer site visits, faster changes | Better: standard patterns, tested failover | Change for the teams, skills to build | About seven years, easier to extend |
An illustrative comparison. Real options carry costed figures agreed with finance.
One page first
The recommendation, three reasons, the options rejected and why, the cost, the risk and the decision needed. Detail sits behind it for those who want it.
Weak points first
I say where the recommended option is weakest before anyone asks. It builds trust, and the decision is made with the full picture.
If I am overruled
I record the risk in the decision record, make sure the owner has accepted it, and then deliver the chosen option properly.
Engagement
How I run a resilience and cost review, from service map to evidence
A resilience or cost review follows the same path as any engagement. I start with the services that must not stop and the people who own them, map what each one really depends on, then design, prove and report against numbers the business agreed.
Swipe sideways to see the whole diagram.
I start with the business, not the kit. Which services would cause real harm if they stopped, who owns each one, and how much risk is the organisation willing to carry? I agree who decides what in a RACI (responsible, accountable, consulted, informed).
Each service gets numbers: how long it may be down, how much data it may lose, the availability it must reach and the cost per unit it should hit. Regulatory obligations, such as DORA or PCI DSS, are mapped to the services they touch.
I map every service down to people, process, technology, facilities and suppliers, then look for single points of failure and shared services nobody listed. With finance I build a cost baseline from invoices, depreciation and staff time.
I present two or three options, each with its availability, recovery time, cost and risk, including the cost of doing nothing. The business chooses with the price visible, and the decision goes into an architecture decision record.
Recovery is proven, not assumed. I test failover, a full cold start and severe but plausible scenarios against each impact tolerance. Results, gaps and fixes go into an evidence pack an auditor can follow.
Once live, I report what was promised: availability and recovery against target, cost per unit, and risks against appetite. Major incidents follow a timed reporting path so regulator deadlines are met even under pressure.
Targets, risks and costs drift as the business changes. I review them on a fixed cycle and after any big change, track capacity against the order trigger, and keep refresh and exit plans current.
Functional requirements
What it must do.
- Identify each important business service and the systems, people, suppliers and sites it depends on.
- Recover each service within its impact tolerance in a severe but plausible scenario.
- Report availability, recovery and cost per service in a form the business can read.
- Keep a register of third parties, with exit plans for the material ones.
- Give a unit cost per service unit, such as a VM, a terabyte or a GPU hour, for showback or chargeback.
- Raise an alert when free capacity falls below the order trigger.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Availability
- A target per service tier, for example 99.95% for tier 1 and 99.9% for tier 2.
- Recovery
- Recovery time objective (RTO) and recovery point objective (RPO) per service, set well inside the impact tolerance.
- Evidence
- Every control has an owner and dated evidence an auditor can follow.
- Reporting
- Major incidents classified and reported within each regulator's deadlines.
- Accuracy
- Unit cost within an agreed margin of finance's actual figures.
- Time
- Business clocks traceable to UTC, within the accuracy each rule sets.
- Cost
- Cost per unit and total cost of ownership reviewed every quarter.
These are typical starting points. Each organisation sets its own targets and tolerances, and each one is traced to a test.
Who I talk to
- Business service owners and product leads
- Operational risk, compliance and internal audit
- Finance business partners and procurement
- Platform, network and security teams
- Third party and vendor managers
How I find out
- Service mapping workshops covering people, process, technology, facilities and information
- Walking one real transaction end to end through every dependency
- Reviewing past incidents and their timelines
- Reading contracts for exit terms, support levels and renewal dates
- Building a cost baseline with finance from invoices, depreciation and staff time
Discovery tools
| Tool or source | What it tells me |
|---|---|
| CMDB and service maps (ServiceNow or similar) | Which configuration items each service depends on, as long as the data is kept honest. |
| Monitoring and incident history | Real availability, repeat faults, and how long detection and repair really took. |
| Flow data and dependency mapping tools | Hidden dependencies, such as a forgotten DNS server or file share. |
| Contracts and the third party register | Material suppliers, exit clauses and concentration on one provider. |
| Finance ledgers and asset registers | Depreciation, licences, support and power: the inputs to unit cost. |
| Capacity dashboards: vSphere, storage arrays and data centre infrastructure management (DCIM) | Utilisation, headroom and stranded capacity, resource by resource. |
When a service misses its target, I treat it like any fault: define it, find the weak link with data, fix it and stop it coming back. The weak link is rarely where people look first.
My approach
- Define. Which service missed which target, by how much, and for whom? Agree how it was measured first.
- Timeline. Build the incident timeline: when it failed, when it was detected and when it was fixed. Slow detection is a finding on its own.
- Dependencies. Walk the dependency map. Look for single points of failure and shared services that both halves rely on.
- Correlation. Check whether redundant parts failed together because of the same firmware, change, certificate or supplier.
- Cost. Price the options that close the gap, and compare them with the cost of the next outage.
- Fix and record. Fix through change, update the risk register and record the cause for problem management.
Faults I see most
- Both halves of a pair taken down by the same change or firmware
- A shared service such as DNS, identity or time that nobody listed
- Recovery that works in a test but takes too long for real
- Failover that needs a person who is not on call
- Capacity bought in time but stranded, because power or ports ran out first
- Unit cost rising because utilisation fell, not because prices rose
Layer by layer
| Layer | What I check |
|---|---|
| Facilities | Power and cooling redundancy, generator tests, single feeds and maintenance windows. |
| Network | Diverse paths, shared ducts, routing failover times and carrier diversity. |
| Platforms | Cluster headroom (N+1), admission control, storage replication and tested restores. |
| Shared services | DNS, identity, time and key management: where they run and what they depend on. |
| Application | Health checks, session handling, retries, and whether the service fails over cleanly. |
My general method, with worked faults, is on the troubleshooting page.
Resilience and cost both decay unless someone owns them, so I set up the reporting and reviews before handover.
Service levels
Availability, RTO and RPO reported per service against the agreed targets, every month.
Major incidents
A clear classification and a timed reporting path, so regulator deadlines are met under pressure.
Risk register
Risks scored against appetite, with owners, actions and dates, reviewed every month.
Showback
Cost per unit shown to each team that consumes it, so idle and oversized resources get noticed.
Capacity
Free capacity tracked against the order trigger for each pod type and site.
Evidence
Test results, approvals and configuration baselines kept together, ready for audit.
Resilience targets, risks and costs change as the business changes. I review them on a fixed cycle and after any big change.
- Every year. A self assessment against impact tolerances, refreshed service maps and a new test plan.
- Every quarter. Unit cost, utilisation and capacity triggers reviewed with finance.
- Every month. The risk register, exceptions and technical debt reviewed by age and score.
- On change. Any new supplier, site or platform is checked for concentration and exit risk.
- Refresh. Kit is replaced before end of support, with the refresh costed in the model from day one.
- Exit. Exit plans for material suppliers are tested, so leaving is possible without disruption.
Projects
Where I have kept services up and costs down
Availability targets, regulated estates and cost savings run through my career, from banks to manufacturers.
99.99% through a merger
Safeguarded over £200M of transactional services through data centre transitions during AVEVA's reverse merger, holding 99.99% uptime throughout.
Regulated refresh, no outages
Delivered the network refresh for Europe, the Middle East and Africa (EMEA), replacing end of life equipment with minimal impact on global operations, keeping the estate compliant with MiFID II throughout with no service outages.
£700k a month saved
Post merger infrastructure integration across a multi site estate, rationalising infrastructure, support and technology overheads to save £700k a month.
Payback in the first year
Local and wide area networks, voice, virtualisation and security across the group, saving over £250k a year with first year payback.
Capacity for dense compute
Designed AI ready data centre infrastructure for high density compute clusters, covering network, power, cooling and structured cabling, with the rack layouts the build teams worked from.
How I run it
Technical design authority, high and low level designs and vendor and third party management, in MiFID II and PCI DSS environments, delivered with PRINCE2 and ITIL practice.
Backed by PRINCE2 Practitioner and ITIL. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.