Built in repeatable pods
Each pod has the same racks, fabric, compute, storage and tools. Growth means adding another tested pod.
Private cloud and storage
Private cloud and storage, built to recover
I design the layers that sit on a data centre pod: virtualisation, containers, storage and the copies that bring it all back after a failure or an attack. This page shows how each layer works, with diagrams you can step through, and where I have used it.
Private cloud stack
Five layers, one pod
A private cloud is a stack of layers on a data centre pod. Each layer depends on the one below it. Select a layer in the diagram, or use the buttons, to see what it holds and what flows through it.
Swipe sideways to see the whole diagram.
Facility. Racks, structured cabling, A and B power and cooling sized for the load in each rack. An out of band network lets me reach every device even when the pod itself is down. The detail is on the data centre page.
Infrastructure. Servers, storage arrays and a leaf and spine fabric, with firewalls at the edge of the pod. Every pod is built to the same tested design, so the second one is a copy of the first.
Platforms. Three ways to run work on the same kit: virtual machines (VMs) on VMware Cloud Foundation, containers on OpenShift, and bare metal servers for databases, big data and GPU (graphics processing unit) jobs that want the whole machine.
Automation. This is what turns a virtualised data centre into a cloud. Teams order from a catalogue or an application programming interface (API), and code builds the same thing every time. The US National Institute of Standards and Technology (NIST) lists five traits in NIST SP 800-145: on demand self service, broad network access, pooled resources, rapid elasticity and measured use.
Workloads. Applications, databases and AI jobs, each placed on the platform that suits it. Users see a service with an address. The servers, disks and switches stay out of sight.
The right platform for each job
VMs for most applications, containers for software built to scale out, and bare metal where a job needs the whole server.
Recovery is designed in
Every layer has a way back: spare capacity for failures, copies in other places, and a clean copy that an attacker cannot reach.
Pod lifecycle
A pod's life, from design to disposal
A private cloud grows one pod at a time, and every pod follows the same life. It is designed once, built and proven in a factory, accepted on site, joined to the fabric, patched in waves, then retired safely. The governance around it is on the design authority page.
Swipe sideways to see the whole diagram.
Each pod starts as a standard design: racks, fabric, addressing, compute, storage and a fixed bill of materials, with the test plan written up front. It is reviewed and signed at design authority before anything is ordered.
Racks are built off site, cabled and labelled to the design, and brought to the agreed firmware. They run under load, typically for one to three days, so weak parts fail in the factory. Results go into an evidence pack, and the rack ships whole. This is often called rack and roll.
In the hall the rack is powered from the real feeds and cabled to the leaves. Every cable is checked against the cabling plan, and every link is tested at line rate. GPU pods also run NCCL (NVIDIA Collective Communications Library) tests to prove the bandwidth between GPUs. I have trained in NVIDIA's Cable Validation Tool (CVT), which checks fabric cabling against the planned topology.
The pod's leaves peer with the spines, and the automation pipeline builds their configuration from the source of truth. Monitoring starts at once, and the new capacity appears in the self service catalogue.
Firmware and patches are proven in the lab, then go to a canary pod and soak there. They then roll out in waves, with health checks between them. The two pods that carry the same service are never in the same wave.
Hardware is refreshed on a planned cycle. Old drives are sanitised to NIST SP 800-88, the US guide to media sanitisation: clear, purge or destroy, with cryptographic erase where drives encrypt themselves. Each drive gets a certificate tied to its serial number, and lessons from the old pod feed the next design.
Moving workloads to a new pod
| Method | When I use it | What I watch |
|---|---|---|
| vMotion between clusters | Same vCenter, compatible processors or Enhanced vMotion Compatibility (EVC), the same networks on both pods. | The maximum transmission unit (MTU) and the vMotion VMkernel path between pods. |
| Cross vCenter vMotion | A new pod with its own vCenter. | Networks must exist on both sides, or the VMs given new addresses. |
| VMware HCX (Hybrid Cloud Extension) | Different versions or sites, bulk or live moves, with networks extended for the move. | Licensing, and removing extended networks once the move is done. |
| Storage replication | Very large VMs, or data that must move ahead of the compute. | Time, bandwidth and the cutover window. |
| Redeploy | Stateless and container workloads. | The pipeline must be able to rebuild everything. |
Blockers I look for early: raw device mappings, Universal Serial Bus (USB) licence keys, VMs pinned to a host, very old virtual hardware, and software licensed to a media access control (MAC) address or processor.
All flash platforms and subscriptions
| Platform | Shape | Where it fits |
|---|---|---|
| Pure Storage FlashArray | Block, with file services. ActiveCluster for synchronous replication. | VM and database pods. |
| Pure Storage FlashBlade | Scale out file and object. | Backup landing zones, analytics and AI data. |
| VAST Data | Disaggregated and shared everything: stateless servers reach shared flash enclosures over NVMe fabrics. | Large file and object estates, and AI training data. |
| NetApp AFF and ASA | Unified block and file with ONTAP (AFF), or block only (ASA). | Mixed estates with large file shares. |
| Dell PowerStore and PowerMax | Midrange unified storage (PowerStore), high end block (PowerMax). | Existing Dell estates, and mainframe storage on PowerMax. |
Subscriptions such as Pure's Evergreen replace controllers in place on a schedule, so the array refreshes without moving data. I match the subscription term to the pod's life and check what happens to capacity and price at renewal.
VMware
VMware Cloud Foundation, piece by piece
VMware Cloud Foundation (VCF) bundles the hypervisor, storage, networking and management tools into one tested stack. Step through how the parts fit together.
Swipe sideways to see the whole diagram.
ESXi is the hypervisor. It runs on each server and shares the processors, memory and network cards between many virtual machines, each with its own operating system.
vCenter groups hosts into clusters and manages them as one. If a host fails, HA (high availability) restarts its VMs on the others. DRS (Distributed Resource Scheduler) moves running VMs to balance the load.
vSAN pools the local disks in each host into one shared datastore. With a mirroring policy, each write is kept on two hosts, so one host can fail without losing data. External arrays are the other choice.
NSX runs switching, routing and the firewall inside the hypervisor on every host. Networks travel in tunnels over the physical fabric, which only has to move IP between hosts. A Tier 0 gateway peers with the fabric using BGP (Border Gateway Protocol).
VCF keeps its management tools in a management domain. Each workload domain gets its own vCenter, which runs in the management domain. A busy or broken workload domain cannot take the tools down with it.
VCF upgrades the whole stack in a tested order, so versions always match. Each host goes into maintenance mode in turn: vMotion moves its VMs away, the host is patched, then it rejoins the cluster.
Licensing and platform choice
Since Broadcom bought VMware in late 2023, VMware has been sold only as a subscription, priced per processor core, in a small set of bundles led by VCF. Licences already bought keep working, but no new perpetual licences are sold. Many customers have publicly reported higher renewal costs, so for many clients the licence bill now drives the platform choice. I compare VCF with three alternatives on cost, skills and the effort to move off later.
| Platform | What it is | Cost | Skills | Effort to move off later |
|---|---|---|---|---|
| VMware VCF | vSphere, vSAN, NSX and automation sold as one stack | Per core subscription, sold as a bundle | Widely available | High once NSX, vSAN and VCF Automation are in daily use |
| Nutanix | The AHV hypervisor and storage on the same nodes, managed through Prism | Per core term licences | Close to VMware for day to day work | Medium. Tools convert the VMs, but operations are built around the platform |
| OpenStack | Open source cloud software: Nova for compute, Neutron for networks, Cinder for block storage | No licence fee. Support and a skilled team are the cost | Scarce, and it needs a larger team to run | Low vendor lock in, but you own all the integration |
| OpenShift Virtualization | VMs run as Kubernetes workloads beside containers, based on KubeVirt | Red Hat subscription | Needs Kubernetes skills in operations | Medium. A migration toolkit moves VMs, but ways of working change the most |
Based on public reporting up to October 2026. Prices, bundles and minimum orders change often, so I check current terms for each client.
Live migration with vMotion
vMotion moves a running virtual machine from one host to another without stopping it. I use it in my home lab to move running VMs between my ESXi hosts, and it is what makes patching without downtime possible.
Swipe sideways to see the whole diagram.
A VM runs on host A and serves users. Its disks sit on shared storage that both hosts can reach, so only its memory and running state need to move.
vMotion copies the VM's memory to host B over its own VMkernel network while the VM keeps running. Some pages change during the copy.
The changed pages are sent again. Each pass is smaller than the last, until what is left can move in a moment.
The VM usually pauses for under a second while the last pages and the processor state move. It resumes on host B, which tells the switches where the VM's hardware (MAC) address now lives.
Sessions stay up and users notice nothing. Host A is now free for patching. For this to work, both hosts need the same networks and compatible processors.
A packet walk through NSX
NSX moves the switch and the firewall into the hypervisor. I have not run NSX in production. The ideas are the ones I have used on VXLAN (Virtual Extensible LAN) fabrics with Cisco Nexus and ACI (Application Centric Infrastructure): tunnel endpoints, an overlay, and policy set centrally.
Swipe sideways to see the whole diagram.
A web VM on host A talks to a database VM on host B. Both sit on the same NSX segment, a layer 2 network that exists only in software.
The distributed firewall (DFW) checks the packet at the web VM's virtual network card, before it reaches the switch. A rule allows web to database on the database port.
Host A's tunnel endpoint (TEP) wraps the frame in a Geneve header inside User Datagram Protocol (UDP) port 6081 and sends it to host B's TEP. The fabric only sees IP traffic between two hosts.
Host B removes the wrapper. The firewall checks the packet again at the database VM's virtual card, then delivers it.
The test VM tries to reach the database. No rule allows it, so the firewall drops the packet at the test VM's own card. It never leaves host A or touches the fabric.
The wrapper adds bytes to every packet, so the fabric needs a maximum transmission unit (MTU), the largest packet it carries, of at least 1,600 bytes. 1,700 or more is recommended. A mismatch shows up as small packets working while large transfers stall.
Containers
Kubernetes and OpenShift, in plain terms
OpenShift is Red Hat's Kubernetes platform, and it runs on bare metal or on VMs in the pod. I have not delivered a container platform as a project, so I show this as knowledge: how it works, and what the network and storage under it must provide.
Swipe sideways to see the whole diagram.
A cluster has control plane nodes, which hold the API and the cluster database (etcd), and worker nodes, which run the applications. With three control nodes, the cluster keeps a majority if one fails.
A pod is one or more containers that share an address. The scheduler places pods on workers with spare capacity. Here the app runs as three copies, one on each worker.
Every pod gets its own IP address on the pod network. In OpenShift the default network plugin, built on OVN (Open Virtual Network), carries traffic between nodes in Geneve tunnels, much like NSX.
Pods come and go, so their addresses change. A Service gives the app one stable name and address, and spreads requests across the healthy pods.
Traffic from outside reaches a load balancer, then the router, which is OpenShift's ingress. The router matches the host name, looks up the healthy pods behind the Service and sends the request straight to one of them. This one lands on worker 3.
If a pod or a whole worker fails, the cluster starts a replacement on another worker, and the Service stops sending it traffic. The app stays up while copies run on more than one node.
What the pod must give a cluster
- Address space planned up front. Pod and Service ranges must not overlap the rest of the network, and they are hard to change once the cluster is built.
- MTU with room for the tunnel. The Geneve wrapper in OpenShift's default plugin uses about 100 bytes, so the pod MTU sits below the fabric MTU.
- Persistent storage through a driver. Containers are short lived but their data is not. Volumes come from the arrays through a storage driver that follows the Container Storage Interface (CSI).
- A load balancer in front. Two or more routers behind a load balancer, so ingress survives a worker failure.
The networking pieces, one by one
| Piece | What it does | What I watch |
|---|---|---|
| Service types | ClusterIP is reachable only inside the cluster. NodePort opens the same port on every node. LoadBalancer asks for an outside address. | On premises, LoadBalancer needs something to hand out those addresses. |
| MetalLB | Gives LoadBalancer Services an address on bare metal, announced with address resolution protocol (ARP) in layer 2 mode, or with Border Gateway Protocol (BGP) to the leaves. | In BGP mode it peers with the leaf pair, and the leaves filter what it may announce. |
| NetworkPolicy | Rules, chosen by labels, for which pods may talk to which. With no policy at all, every pod can reach every other pod. | I start each namespace with a default deny, then allow named flows. |
| Ingress and Gateway API | Route web traffic from outside to Services. Gateway API is the newer, role based successor to Ingress. | Who owns the gateway, its certificates and its host names. |
| Egress IP | Gives a namespace's outbound traffic a fixed source address. | Lets firewalls outside the cluster tell one application from another. |
| Multus | Gives a pod a second network interface, for example a VLAN for storage or for a virtual machine. | Each extra network bypasses the cluster's own policy, so it needs its own controls. |
| Service mesh | Sidecars or a proxy on each node add mutual transport layer security (TLS), retries and telemetry between services. | More latency and more moving parts. Only where the security or visibility need is real. |
OpenShift defaults to plan around
| Range | Default | Why it matters |
|---|---|---|
| Cluster network (pods) | 10.128.0.0/14, a /23 per node | Must not overlap anything the pods need to reach. |
| Service network | 172.30.0.0/16 | Internal only, but a clash still breaks routing. |
| OVN join subnet | 100.64.0.0/16 | Used inside OVN. If the wider network already uses it, change it at install or later through the cluster network operator. |
| OVN transit switch subnet | 100.88.0.0/16 | Same rule: check it against the address plan, and change it at install or later through the cluster network operator. |
| Masquerade subnet | 169.254.0.0/17 | Link local, used for traffic between the host and OVN. New clusters from 4.17 use this. Upgraded clusters keep 169.254.169.0/29. |
From Red Hat's OpenShift networking documentation. Defaults can change between releases, so I check the notes for the version being installed.
OpenShift Virtualization
Virtual machines run inside pods, using KubeVirt. Multus connects them to existing VLANs, and the Migration Toolkit for Virtualization moves them from vSphere. Live migration needs shared storage that many nodes can mount at once (ReadWriteMany). Before a pilot I agree criteria for backup, disaster recovery and day to day operations, because those differ most from vSphere.
Bare metal or virtual machines
On bare metal, containers get the full server and no hypervisor licence is needed, but each node is bigger, so a failure takes more pods with it and rebuilds are slower. On virtual machines, nodes are small and easy to replace, at the cost of licences and some overhead.
Bare metal as a service
Servers are driven through Redfish, the standard API on the baseboard management controller (BMC): power, boot order and firmware. They boot an installer over the network with the preboot execution environment (PXE) or iPXE. Metal3 brings the Ironic provisioning service into Kubernetes, so a server is declared like any other resource. OpenShift uses it for bare metal installs.
How each platform joins the fabric
| Platform | How it attaches | How tenants are kept apart |
|---|---|---|
| VMware with NSX | Hosts dual homed to the leaf pair. Overlay tunnel endpoints on one VLAN. Tier 0 gateways peer over BGP with the border or service leaves. | NSX segments and Tier 1 gateways per tenant, each mapped to a routing instance (VRF) on the fabric. |
| VMware without NSX | Port groups on VLANs trunked to the leaf pair. The gateway is the fabric's anycast gateway. | One VRF per zone on the fabric, with the firewall between zones. |
| OpenShift | Nodes on a routed VLAN, Geneve between nodes, MetalLB or an external load balancer for ingress. | Namespaces, NetworkPolicy and egress IPs, with a VRF per cluster or zone. |
| Bare metal | Servers dual homed with link aggregation (LACP) to the leaf pair, or routed to each leaf. | A VRF and VLAN per tenant on the fabric. |
Whatever the platform, I keep one tenant model: a zone is a VRF on the fabric, and each platform maps its own constructs onto it. NSX in depth is on the VMware NSX page.
Storage tiers
Block, file and object, and when to use each
All three put data on disks across a network. They differ in how the data is addressed, how it travels and what it is good at. Switch between them, or pick something to store and see where it belongs.
Swipe sideways to see the whole diagram.
Block. The array hands a server a raw volume, a logical unit number (LUN), and the server puts its own file system on it. Two separate paths, A and B, with multipathing on the host, so one failure never cuts the server off.
File. A file server, or network attached storage (NAS), shares folders over the network. NFS (Network File System) is the Linux and Unix standard, SMB (Server Message Block) the Windows one. Many clients open the same files by name.
Object. Data is stored as objects in buckets, each with a key and metadata, read and written over the S3 (Simple Storage Service) API. It scales to billions of objects, and Object Lock can make copies unchangeable.
What are you storing?
Block. Ideally NVMe (non volatile memory express) over Fabrics. A database wants low, steady latency on a volume of its own.
Block, usually. VMFS, the VMware file system, on an array volume. An NFS datastore and vSAN, built from the hosts' own disks, are the other options.
File, over SMB. Add frequent snapshots, because anything users can write, ransomware can encrypt.
File, usually scale out NFS. Many GPU servers read the same training data at once. Object storage often holds the raw data lake behind it.
Object, with Object Lock in compliance mode. A locked backup cannot be changed or deleted before its retention period ends, even by an administrator.
Object. Cost per TB matters more than speed, and lifecycle rules can move old data to cheaper capacity.
| Type | Protocols and ports | Good for | Watch out for |
|---|---|---|---|
| Block | Fibre Channel; iSCSI (SCSI over IP) on TCP 3260; NVMe over Fabrics, such as NVMe/TCP on port 4420 or NVMe over RoCE (remote direct memory access over Converged Ethernet) | Databases, VM datastores, container volumes | Zoning and masking decide which server sees which volume |
| File | NFS on TCP 2049; SMB on TCP 445 | Shared files, home drives, AI training data | Users can write to it, so ransomware can too |
| Object | S3 API over HTTPS on TCP 443 | Backups, archives, logs, data lakes | Not for transactional databases; check which S3 features a product supports |
How storage should connect to the pod
- Its own VRF (virtual routing and forwarding table). Storage traffic sits in its own routing domain, so a change made for applications cannot touch a storage path.
- Two paths that share nothing. Path A and path B on separate switches, or Fibre Channel fabrics A and B, with multipathing on the host instead of bundled links.
- Jumbo frames, tested end to end. MTU 9000 on hosts and arrays and 9216 on switch ports, checked before anything goes live.
- No firewall in the data path. The VRF and access lists control storage traffic. A firewall in line adds delay and one more thing to fail.
- NVMe/TCP as the default. It gives most of the speed of NVMe on the normal fabric. RoCE works best on a lossless class, with priority flow control (PFC) and explicit congestion notification (ECN) tuned end to end, so it belongs on a separate network and only where a workload proves the need.
- Management out of band. Array management sits on the out of band network, with admin access through privileged access controls.
Storage networks
How data reaches the disks
Block storage needs two separate paths from every host to every array, so that one failure never stops the data. The protocol decides how those paths are built. Underneath, RAID or erasure coding decides what survives a failed disk.
Swipe sideways to see the whole diagram.
Fibre Channel. A dedicated block network at 32G or 64G, on Cisco MDS or Brocade switches. Each server has a host bus adapter (HBA) on each fabric, and fabrics A and B are never joined, so one can fail completely. Buffer to buffer credits make it lossless.
Zoning and masking. Devices are known by their worldwide port names (WWPNs). Zoning on the switch decides which ports may talk, ideally one initiator and one target per zone. LUN masking on the array decides which volumes each host sees. The host runs multipathing across both fabrics.
iSCSI. SCSI over TCP on port 3260, so no separate network is needed. I use two subnets or VLANs, path A and path B, with multipathing on the host, never LACP, and keep it off routed firewall paths. Hosts are named by IQNs (iSCSI qualified names) and authenticate with CHAP (Challenge Handshake Authentication Protocol).
NVMe over TCP. NVMe has up to 65,535 queues, each up to 65,536 commands deep, where SCSI had one. NVMe/TCP carries that over any IP network on port 4420, with discovery on 8009, and needs no lossless Ethernet. It is my default for new pods.
Fibre Channel over Ethernet. FCoE carries Fibre Channel frames in Ethernet, so it needs data centre bridging: priority flow control (PFC) and enhanced transmission selection (ETS). The two storage area networks, SAN A and SAN B, still stay apart, each on its own switch and FCoE VLAN. It was popular on converged access switches but is fading in favour of NVMe/TCP.
| Protocol | Transport and ports | Names | Where I use it |
|---|---|---|---|
| Fibre Channel | Its own network at 16, 32 or 64GFC, lossless by buffer to buffer credits | WWPN, zoning on the switch | Existing estates and the most critical databases |
| NVMe/FC | NVMe over the same Fibre Channel fabric, with recent Gen 6 or later adapters, often 32G | NQN (NVMe qualified name) | Getting more from an existing FC investment |
| iSCSI | TCP 3260 on Ethernet, two subnets for the two paths | IQN, with CHAP | Smaller sites and older arrays |
| NVMe/TCP | TCP 4420, discovery on 8009, any routed IP network | NQN | My default for new pods |
| NVMe over RoCE (RDMA over Converged Ethernet) | RDMA on UDP 4791, needs PFC and ECN (explicit congestion notification) tuned | NQN | The lowest latency, run by specialists |
| FCoE | Ethernet with data centre bridging | WWPN | Older converged designs only |
Other ports I keep in mind: NFS on TCP 2049, SMB on TCP 445, S3 over HTTPS on 443, and the Key Management Interoperability Protocol (KMIP) for encryption keys on TCP 5696.
RAID and erasure coding
Swipe sideways to see the whole diagram.
RAID 1, mirroring. Every block is written to two disks. Half the raw capacity is usable, reads can come from either disk, and one disk can fail. Rebuild is a straight copy, so it is quick.
RAID 5, single parity. One parity block per stripe, rotated across the disks. Three quarters of four disks are usable and one disk can fail. With today's large drives a rebuild takes many hours, and a second failure or a read error during it loses data, so I avoid it for important data.
RAID 6, double parity. Two parity blocks, P and Q, per stripe, so any two disks can fail. It suits capacity tiers with large drives. Writes cost more, because both parity blocks must be updated.
RAID 10, mirrors striped. Data is striped across mirrored pairs. It gives the best write performance and fast rebuilds, so I use it for write heavy, latency sensitive work such as databases. It survives one disk in each pair.
Erasure coding. Data is split into k pieces with m parity pieces spread across drives or nodes, and any k pieces rebuild it. 8 + 2 survives two failures with 25 percent overhead, against 200 percent for three full copies. Across nodes it survives a node or rack loss, but rebuilds use CPU and network, as in vSAN and scale out storage.
vSAN or an external array
vSAN builds storage from the hosts' own drives, so compute and storage grow together, per VM policies set how many failures to tolerate, and it is included in VMware Cloud Foundation. An external array grows separately, offers array snapshots and replication, and serves VMware, containers and bare metal alike. A standard vSAN cluster needs three hosts, a two node cluster needs a witness, and four or more leave room to rebuild during maintenance.
Hyperconverged or disaggregated
Hyperconverged pods grow in host sized steps with one stack and one lifecycle, and suit general virtual machine pods and edge sites. Disaggregated pods scale compute and storage separately, with a storage leaf pair or Fibre Channel, and suit storage heavy, shared, AI and big data work. I decide on the expected ratio of compute to storage growth, licence cost and the data protection design.
Trusting data reduction
Deduplication, compression and thin provisioning stretch capacity. Virtual machines and desktops reduce well. Encrypted, compressed or media data barely reduces, and encryption inside a VM defeats deduplication. I plan on ratios measured from our own data, and ask for them as a guarantee in the contract.
Terms I use
| Term | Meaning |
|---|---|
| WWPN, NPIV, FLOGI | A Fibre Channel port's worldwide name, several virtual logins on one port, and the login a port makes to the fabric. |
| IQN and NQN | The names of iSCSI and NVMe hosts and targets. |
| CHAP and iSER | iSCSI authentication, and iSCSI over RDMA. |
| FTT | Failures to tolerate: in vSAN, how many failures a VM's data can survive. |
| vVols | One array volume per virtual disk, so array features work per VM. |
| RS(6,3) | A Reed Solomon erasure code with 6 data and 3 parity pieces. |
| ALUA | Asymmetric logical unit access: the array tells the host which paths are active and optimised, and which are standby, so multipathing prefers the right controller. |
| VSAN (Cisco MDS) | A virtual SAN: separate fabric services and zoning on shared Fibre Channel switches. Not the same thing as VMware vSAN. |
| Queue depth | How many commands a host, adapter or volume may have outstanding. Too low starves fast arrays, too high moves the queue into the array. |
| Boot from SAN | Hosts boot from an array volume instead of a local disk, so a failed server is replaced by pointing a new one at the same volume. |
Data protection
Copies that survive a failure or an attack
Each kind of copy protects against a different problem. Watch the copies flow out from production, then see which ones are still clean after a ransomware attack.
Swipe sideways to see the whole diagram.
The array takes point in time snapshots, often every hour. They are almost instant and give a fast rollback after a mistake. They live on the same array, so on their own they are not a backup.
A write is confirmed only once the metro array has it too, so a site failure loses no data. Every write waits for the round trip, which in practice keeps the two sites within about 100 km. A witness in a third location decides which side keeps running if the link breaks.
Changes ship to another region a few seconds or minutes behind. It works over any distance and covers a regional event, at the cost of the last moments of data.
A backup is a separate copy in a different system, kept for weeks or years. It protects against corruption and deletion, which replication would copy straight across.
With object lock in compliance mode, or locked snapshots, nobody can change or delete a copy before its retention ends, not even an administrator. Governance mode can be overridden, so this copy uses compliance mode, and the time source it relies on is protected too. Attackers go after backups first, so this matters most.
The vault sits outside the production trust domain, with its own identity and admin accounts. The link opens for a short window so the vault can pull a copy, then closes again.
Attackers encrypt production. Replication does its job and copies the encrypted data to the metro and disaster recovery (DR) arrays. The immutable copy and the vault are untouched.
Recovery starts in an isolated clean room. Identity comes back first, then data from the last clean copy is restored and scanned. Only then do systems return to production.
The 3, 2, 1, 1, 0 rule
- 3copies of the data, counting production
- 2different storage types or systems
- 1copy in another place
- 1copy offline or immutable
- 0errors, proven by regular restore tests
Two targets for every workload
I set two targets for every workload: the recovery point objective (RPO), how much data it can afford to lose, and the recovery time objective (RTO), how long it can be down. Each kind of copy gives a different answer to both.
| Snapshot | Replication | Backup | |
|---|---|---|---|
| What it is | A point in time copy on the same system | A running copy on another system or site | A separate copy in another system, kept for a set time |
| Protects against | Mistakes, with a quick rollback | Losing an array or a site | Corruption, deletion, ransomware, long term retention |
| Copies corruption? | Not once it is taken | Yes, straight away | Not into older restore points |
| Data you could lose (RPO) | Back to the last snapshot, often up to an hour | None if synchronous. Seconds to minutes if asynchronous | Back to the last backup, often up to a day |
| Time to recover (RTO) | Seconds to minutes | Minutes, by failing over | Hours, depending on size and speed |
Recovery classes
I group workloads into a few recovery classes, so each one gets the protection it needs and pays for no more. Switch between the classes to see how the two targets change.
Swipe sideways to see the whole diagram.
Gold. Synchronous replication across the metro, frequent snapshots, and immutable and vault copies. A site failure loses no data, and service returns in minutes. For the few services that cannot stop.
Silver. Asynchronous replication to another region, snapshots and an immutable backup. Seconds to minutes of data are at risk, and service is back in minutes to hours.
Bronze. A daily backup with an immutable copy. Up to a day of data is at risk, and a restore takes hours to days. Right for systems the business can do without for a while.
Every class keeps an immutable copy, because an attacker can reach every other kind of copy.
Recovery after an attack takes longer than these targets suggest. The last clean copy may be days old, and identity has to be rebuilt before anything else, so I plan and test the two separately. Public cloud and disaster recovery patterns are on the cloud page.
Disaster recovery patterns
| Pattern | How it works | Data loss (RPO) | Recovery time (RTO) |
|---|---|---|---|
| Metro active and active | Synchronous replication between sites, a witness in a third site | Near zero | Near zero to minutes |
| Warm standby | Asynchronous replication, infrastructure running at the second site | Seconds to minutes | Minutes to hours |
| Asynchronous to another region | Replication to a distant site, started on demand | Minutes | Hours |
| Backup and restore | Restore from backup onto rebuilt or spare infrastructure | Hours | Hours to days |
| Cyber recovery vault | Restore the last clean copy into an isolated environment first | The last clean copy | Days |
Indicative. For active and active I prefer the application to replicate its own data, rather than stretching layer 2 between sites.
Distance limits for stretched designs
| Design | Round trip limit between sites | Notes |
|---|---|---|
| vSAN stretched cluster | 5 ms between the data sites | The witness can sit much further away, up to about 200 ms, in a third site or the cloud. |
| vSphere Metro Storage Cluster | Under 10 ms for the network | Synchronous storage usually needs 5 ms or less. The array vendor's statement applies. |
| Cisco ACI Multi-Pod | Up to 50 ms | One Application Policy Infrastructure Controller (APIC) cluster across pods. |
These are support ceilings, not targets. I check the current vendor guidance and test the real application at the real latency.
The backup platform itself
| Part | What it does | How I protect it |
|---|---|---|
| Control server | Holds the catalogue, schedules, plans and reports. | Its own disaster recovery plan, with a copy in the vault. Without it, restores are slow or impossible. |
| Media servers or proxies | Move and deduplicate the data, with the deduplication database on fast flash. | Enough of them to meet restore times, not just backup windows. |
| Targets | Disk, object storage, appliances, cloud storage or tape. | An immutable copy with object lock in compliance mode, and an air gapped copy. |
| Agents and snapshots | Agents for databases and file systems, hypervisor APIs for VMs, and array snapshot orchestration. | Credentials kept separate from production identity. |
Choosing a cyber recovery platform
I judge it on recovery within tolerance. That means immutability with retention that administrators cannot shorten, an isolated recovery environment with its own identity, and more than one person to approve destructive actions. It also needs anomaly detection, and restore at scale, hundreds of VMs and large databases, timed in a proof of concept.
Media sanitisation
NIST SP 800-88 Revision 2 replaced the 2014 version. Clear overwrites, purge uses block erase, cryptographic erase or degaussing, and destroy shreds. Each device gets a certificate with its serial, method, tool and who verified it. Cryptographic erase only counts if the data was always encrypted and every copy of the key, including any escrowed copy, is destroyed.
IT asset disposition
Hardware is redeployed, resold or returned through a certified partner, or recycled under the UK Waste Electrical and Electronic Equipment (WEEE) regulations, with a chain of custody for every serial number. The CMDB and the financial asset register are reconciled, and disposal partners are managed as third parties for operational resilience.
Engagement
How I run a private cloud programme, from service catalogue to refresh
A private cloud is a product that application teams consume, not just a set of hosts. I run every programme the same way: agree the service, size it from real data, design the platform and its protection, then hand over something the platform team can run, patch and grow.
Swipe sideways to see the whole diagram.
I start by agreeing what the private cloud must offer: virtual machines, containers, bare metal and storage tiers, with the recovery each tier promises. Licensing and cost come into the first conversation, because they now shape the platform choice as much as features do.
Functional requirements say what the platform must run and how teams request it. Non functional ones give numbers: overcommit per tier, recovery point and time objectives, storage performance, isolation between zones and how fast a request is fulfilled.
I collect inventory and performance history from the current estate with tools such as RVTools and LiveOptics, measure real data reduction, map dependencies between applications and test whether today's backups really meet their recovery targets.
I design the pod: compute, the hypervisor or container platform, storage and its network, management domains, recovery classes and the backup and vault design. The design authority approves it, with the platform choice recorded as a decision.
Pods are built from templates and firmware baselines, then failure tested: a host, a switch, a storage controller and a full restore of sample workloads. Workloads move in waves through change control, each with a rollback point.
Before handover the platform team has runbooks, capacity and performance alerts, a schedule of restore tests and a self service catalogue for application teams. Hypercare ends only when the agreed exit criteria are met and the first restore test has passed.
Hypervisor, firmware, storage and backup software each follow a release plan, patched in waves. Licences and support renewals are tracked, the next pod is ordered on a capacity trigger, and old pods are drained, sanitised and retired.
Functional requirements
What it must do.
- Offer virtual machines, containers and bare metal from one catalogue.
- Provide block, file and object storage, with tiers by performance.
- Keep tenants and zones apart, with policy that follows the workload.
- Back up every workload, with an immutable copy and an isolated vault copy.
- Recover a workload, a cluster or a whole pod within its recovery class.
- Let application teams request and change services through an API or portal.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Recovery
- RPO and RTO per recovery class, for example near zero for gold and 24 hours for bronze.
- Capacity
- N+1 hosts in every cluster, with overcommit set per tier.
- Performance
- Storage latency targets per tier, measured from the guest.
- Availability
- No single host, switch, path or controller failure stops a workload.
- Security
- Encryption at rest, separate backup identity, and destructive actions approved by two people.
- Operability
- Changes from templates and pipelines, patching without downtime, drift reported.
- Cost
- Cost per VM, per TB and per pod tracked, including licences.
These are typical starting targets. Each organisation sets its own numbers for each recovery class, and proves them in a restore or failover test.
Who I talk to
- Application owners for the critical services
- Virtualisation, storage and backup teams
- Security, risk and compliance
- Finance and the licensing team
- Operations and the service desk
How I find out
- Workshops by workload type and recovery class
- Inventory and performance collection from the current estate
- Dependency mapping between applications
- Restore tests against current recovery targets
- Licence and support contract review
Discovery tools
| Tool or source | What it tells me |
|---|---|
| RVTools | Every VM, host and datastore in a vSphere estate, with configuration and sizing. |
| LiveOptics | Performance history, peaks and data reduction to size the new platform. |
| Array and vSAN performance reports | Real latency, IOPS and capacity growth. |
| Backup reports and restore tests | Which workloads are protected, and how long restores really take. |
| Flow data or dependency mapping tools | Which applications must move together. |
| Licence entitlement reports | What is owned, what is subscribed and when it renews. |
Private cloud faults often look like application problems. I separate compute, storage and network quickly, using the platform's own counters before I touch anything.
My approach
- Define. Which VMs, hosts, clusters or datastores are affected, since when, and what changed?
- Compute. Host health, CPU ready time, memory ballooning and swapping, and HA events.
- Storage. Latency at the guest, the host and the array, path state and multipathing.
- Network. Uplinks, VLANs and MTU on the vMotion, storage and overlay networks.
- Platform. vCenter, NSX or Kubernetes control plane health, certificates and time.
- Fix and record. Fix through change, prove it with the same counters, and add the cause to the platform's known errors.
Faults I see most
- High CPU ready time from overcommit
- Storage latency from a failed path or a busy array
- An MTU mismatch breaking vMotion or the overlay
- A full datastore or thin provisioning running out
- Certificates expiring on management components
- Backups failing silently after a change
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Host hardware health, network card and adapter errors, cabling to the leaf pair. |
| 2 · Data link | VLANs on the distributed switch, NIC teaming, MAC learning on the leaves. |
| 3 · Network | VMkernel addresses and routes, vmkping with large packets and the do not fragment bit. |
| Storage | esxcli storage core path list, array latency and controller state. |
| Platform and application | vCenter and cluster events, HA and DRS decisions, the guest's own view. |
My general method, with worked faults, is on the troubleshooting page.
I agree the operating model before the first workload moves, so the platform team and application teams know where one ends and the other begins.
Service model
Who owns hosts, storage, backup, the catalogue and the applications, with response times.
Monitoring
Capacity, performance and health alerts, with thresholds set from real baselines.
Restore testing
Scheduled restores of sample workloads, with times recorded as evidence.
Runbooks
Steps for a failed host, a full datastore, a path failure and a restore from the vault.
Self service
A catalogue with quotas, so routine requests never need a ticket.
Problem management
Repeat faults, such as a host that keeps losing a storage path, get a root cause, a known error record and a fix in the pod standard.
A private cloud is never finished. I plan its life from the start, using the vendors' release and end of support dates.
- Baseline. Approved versions of hypervisor, firmware, storage and backup software, tested together. vSphere 8, vSAN 8 and VCF 5.2 reach end of general support on 11 October 2027, so the move to VCF 9 is planned now.
- Patch. Hosts patched in waves through maintenance mode, with no downtime for workloads.
- Licences. Subscriptions and support renewals tracked, with platform choices reviewed before renewal.
- Capacity. Quarterly reviews of CPU, memory and storage, with a trigger to order the next pod.
- Refresh. A new pod beside the old, workloads moved by live migration, the old pod drained.
- Dispose. Media sanitised to NIST SP 800-88 with certificates, and assets reused or recycled.
Projects
Where I have worked with it
My virtualisation and data centre work has always sat alongside network and security design, rather than apart from it.
VMware Certified Professional
Data Centre Virtualisation. The version I hold is VCP5, for vSphere 5. VMware has changed a lot since then, and I keep current through my lab and my reading.
Data centre transitions at 99.99 percent
Safeguarded over £200 million of transactional services through data centre transitions, holding 99.99 percent uptime throughout.
Cloud and disaster recovery
Cloud, software defined networking (SDN), disaster recovery and security architecture for enterprise clients. On my last engagement I directed architecture and delivery across AWS, Google Cloud, Azure and disaster recovery.
Enterprise data centre
Post merger infrastructure integration across a multi site estate, with an enterprise data centre on Cisco, Palo Alto and VMware.
Virtualisation across the group
Local and wide area networks, voice over IP, virtualisation and security across the group, including the systems at the food depots.
ESXi, vCenter and vMotion
Three HP servers run VMware ESXi, managed through vCenter, with vMotion to move running VMs between hosts. Virtual firewalls, Cisco Identity Services Engine (ISE) and a wireless controller run on them. See the lab
Containers and Kubernetes appear on this page as knowledge, not as delivered projects. The racks, power and cooling under the pod are on the data centre page, and public cloud on the cloud page.
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.