Keep the GPUs busy
A GPU waiting for the network or for storage is expensive and idle. Every layer of the design exists to keep them fed.
AI infrastructure
Infrastructure that keeps every GPU busy
I design the networks, power, cooling and data paths that AI workloads depend on. GPUs (graphics processing units) are expensive, so the whole design is about keeping them fed and busy. This page shows how a GPU cluster is built, the decisions that matter, and where I have done this work.
Overview
Why AI changes the infrastructure
AI workloads come in two shapes, and they stress a data centre in very different ways. Getting the mix right is the first design decision.
Swipe sideways to see the whole diagram.
Training. One job runs across many GPUs. After every step all GPUs swap their results at the same moment, so every link in the back end fills at once. The step only finishes when the slowest exchange arrives.
Inference. A trained model answers requests. Most traffic runs between users and a model on one GPU or a few servers, so the front end network and response time matter most. For most organisations, inference outgrows training once models are in use, so I size for the real mix.
Density changes the building
Most enterprise racks draw under 15 kW. Air cooled GPU racks typically run 30 to 60 kW, and liquid cooled rack scale systems such as GB200 NVL72 well over 100 kW. These are indicative figures, and they make this a facilities decision before it is a network one.
The slowest flow sets the pace
Training steps wait for the last message to arrive, so tail latency, not average latency, decides how long a job takes.
GPU pod networks
Four networks, one GPU pod
A GPU pod grows in scalable units: a fixed block of servers, switches and cables, built and tested as one. Each unit has four separate networks, so a storage burst or a surge of users does not slow a training run.
Swipe sideways to see the whole diagram.
Compute back end. Each GPU has its own network interface card (NIC). NIC 1 of every server goes to rail 1, NIC 2 to rail 2, and so on, so matching GPUs swap gradients one switch hop apart. The dark cables are rail 1. Each rail switch has eight links down to the servers and eight up, two to each spine, so the fabric is non blocking. It has no route to anything else, and NVLink joins the eight GPUs inside each server.
Front end. Users, inference APIs, job submission and data ingest arrive on an ordinary Ethernet leaf pair, often through a data processing unit (DPU), a NIC with its own processors, in each server. It joins the rest of the estate through a firewall. It can be oversubscribed, because its traffic is north south and bursty.
Storage. Data sets stream in from a parallel file system and checkpoints burst back out, shown in orange. The storage network has its own leaf pair, so a checkpoint never slows the gradient exchange. GPUDirect Storage moves data straight between storage and GPU memory, without a copy through the CPU.
Out of band. A separate 1G network reaches every baseboard management controller (BMC), switch management port, power distribution unit (PDU) and coolant distribution unit. It has its own virtual routing and forwarding table (VRF), firewall and jump hosts, so the pod can still be managed, rebuilt and power cycled when the production fabrics are down.
| Bandwidth | Figure | Working |
|---|---|---|
| Each GPU to the back end | 400 Gb/s | One NIC per GPU, about 50 GB/s. DGX B300 doubles it to 800 Gb/s. |
| Each GPU over NVLink | 1.8 TB/s | About 36 times the NIC, which is why traffic changes rail inside the server |
| Down into the rails | 102.4 Tb/s | 32 systems × 8 GPUs × 400 Gb/s |
| Up to the spines | 102.4 Tb/s | 8 rail switches × 32 links × 400 Gb/s, so one to one |
Eight servers keep the cabling readable in the drawing. NVIDIA's DGX B200 design uses 32 per unit, shown below, and its rail switches have 32 ports of 400 Gb/s down and 32 up. Each unit follows the same build, test and refresh cycle as any other pod, set out in the pod lifecycle on my private cloud page. For how the servers sit in racks, see the GPU rack elevations on my data centre facilities page.
Rail optimised
Cabling that keeps traffic one hop away
In a training job, GPUs mostly exchange data with the GPU in the same position in other servers. Rail optimised cabling puts those GPUs on the same switch.
Swipe sideways to see the whole diagram.
Rail optimised. GPU 1 in every server connects to leaf 1, GPU 2 to leaf 2, and so on. In an all reduce, the step where every GPU combines its results with every other GPU, matching GPUs swap data one switch hop apart. Traffic between different positions moves over NVLink inside the server first.
Top of rack. If each server's GPUs land on that server's own leaf switches, the same exchange climbs to the spine and back. That adds hops and puts every flow on the spine at once. When two big flows land on the same spine link, that link sets the pace of the whole job.
| For the back end | InfiniBand | RoCEv2 Ethernet |
|---|---|---|
| Behaviour | Lossless by design, central subnet manager | Close to InfiniBand with priority flow control (PFC), explicit congestion notification (ECN), DCQCN (Data Center Quantized Congestion Notification) rate control and adaptive routing |
| Vendors | Mainly NVIDIA | NVIDIA, Cisco, Arista and others |
| Skills and tools | Specialist | The same as the rest of the network |
| Segmentation | Partition keys | VRFs and Ethernet virtual private networks (EVPN), like the rest of the estate |
| Standards | InfiniBand Trade Association | RoCEv2 today. The Ultra Ethernet Consortium 1.0 specification, published in 2025, adds a transport built for AI |
| Where I lean | Very large training clusters where every percent of job time counts | Enterprise clusters, where one operating model and more than one vendor matter |
Whichever fabric is chosen, the cabling has to be exact. One cable on the wrong rail quietly slows every job, so the cabling plan is part of the design and is checked against the intended topology before handover. NVIDIA's Cable Validation Tool, which I have trained on, does that check.
Lossless Ethernet
Congestion handled before anything drops
Remote direct memory access (RDMA) lets one server write straight into another's memory. Over Ethernet it is called RoCEv2, and it copes badly with lost packets. So the fabric is tuned to slow senders down early and pause only as a last resort.
Swipe sideways to see the whole diagram.
In a collective, many GPUs send to one at the same instant. Every sender is fast, so the queue on the switch port facing the receiver fills in microseconds. This is called incast.
As the queue passes a threshold, the switch marks packets instead of dropping them. Nothing is lost yet, but the receiver now knows there is congestion on the path.
With DCQCN, the receiver sends congestion notifications back, and each sender's NIC lowers its rate for a moment, then ramps up again. Done well, the queue drains before anything is paused.
If the buffer still fills, the switch sends pause frames upstream for that one traffic class, so nothing in it is dropped. PFC is the safety net. It should trigger rarely, because pauses can spread back through a fabric.
A well tuned fabric keeps queues short, so the slowest flow stays close to the average. That tail sets the job completion time, so it is the number to measure, not link utilisation.
In configuration terms, RoCEv2 runs on UDP port 4791. Its DSCP (Differentiated Services Code Point) value, a marking in the IP header, maps it to one lossless priority, and the congestion notifications get their own high priority queue. Every switch and NIC needs the same settings.
NVIDIA reference design
Built to grow one unit at a time
NVIDIA publishes tested designs for its DGX SuperPOD. They use the same ideas as the pod above, at a much larger scale. The right starting point is the closest published design, adapted to the client's security zones and operating model, with each change recorded and tested.
Swipe sideways to see the whole diagram.
NVIDIA builds its DGX SuperPOD from scalable units. With DGX B200, one unit is 32 systems of eight GPUs. The unit is rail aligned, so traffic on a rail is one hop from every other system in the unit.
Traffic between rails, or between units, goes up to the spine layer. One unit uses 8 leaf and 4 spine switches on 400 Gb/s NDR (next data rate) InfiniBand. NVIDIA Unified Fabric Manager (UFM) provisions and monitors that fabric.
The cluster grows by whole units, not single servers. Four units give about a thousand GPUs on 32 leaf and 16 spine switches. Each unit has the same cabling, so it can be built and checked the same way.
From 16 units upwards, the published design adds a core layer above the spines. NVIDIA describes the design scaling to 64 units and beyond.
The compute fabric is kept for GPU traffic. Storage has its own fabric, on InfiniBand or Ethernet. In band management carries Base Command Manager, Slurm, Run:ai and user access. Out of band reaches every management port and rack PDU.
Generations move quickly
DGX B300 and GB300 NVL72, on Blackwell Ultra, now ship with ConnectX-8 SuperNICs at 800 Gb/s per GPU, on Quantum-X800 InfiniBand or Spectrum-X Ethernet. NVIDIA's DGX B300 design uses units of 72 systems, four to a rack at about 56 kW. Vera Rubin NVL72 is next, with partner systems from the second half of 2026. The ideas stay the same, but the counts change, so each design starts from the current published version.
Rack scale changes the unit
In GB200 and GB300 NVL72, each rack is one NVLink domain of 72 GPUs and 36 Grace CPUs, and the GPUs and CPUs are liquid cooled. NVIDIA's GB200 design uses eight racks per unit, or 576 GPUs.
Spectrum-X for Ethernet
NVIDIA's Ethernet option, Spectrum-X, pairs Spectrum-X switches with BlueField-3 or ConnectX-8 SuperNICs. The switches spread packets across every path and report congestion. The NICs put packets back in order and set each sender's rate.
Enterprise reference designs
For clusters of 32 to 256 GPUs, NVIDIA's Enterprise Reference Architectures use certified partner servers on Ethernet. Each is named by CPUs, GPUs and NICs per server and bandwidth per GPU, so 2-8-9-400 means two CPUs, eight GPUs, nine NICs and 400 Gb/s.
Figures from NVIDIA's public DGX SuperPOD reference architectures for DGX B200, DGX B300 and DGX GB200, and the Enterprise Reference Architectures. Switch counts are for the compute fabric.
GPU servers
Three kinds of GPU server, three different rooms
The kind of GPU server decides almost everything around it: the rack, the cooling, the network and the cabling. I pin this down first, because a room built for one kind rarely suits another.
| Kind | What it is | Typical use | Rack and cooling |
|---|---|---|---|
| PCIe (Peripheral Component Interconnect Express) GPU servers | Two to eight GPU cards, such as L40S, H100 NVL or RTX PRO 6000 Blackwell, in a 2U or 4U server. | Inference, fine tuning, graphics and lighter training. | Air cooled at 15 to 30 kW a rack, which many existing halls can reach with containment. |
| HGX eight GPU systems | Eight GPUs on one board joined by NVLink and NVSwitch, such as H100, H200, B200 or B300, in 6U to 10U. | Training and large inference, scaled out over a rail optimised fabric. | About 10 to 15 kW per server. Air cooled at four per rack, or liquid cooled for more. |
| Rack scale systems | A whole rack acting as one machine, such as GB200 or GB300 NVL72, with 72 GPUs on one NVLink domain. | The largest training and reasoning models. | About 120 to 140 kW a rack, direct liquid cooled, with heavy floor loads. |
Indicative figures. I always size from the vendor's current reference design for the exact system.
Where the fabric switches go
Each GPU server connects to several rail switches, and the cables must stay within copper or active optical cable reach. So the back end switches often sit in a network rack in the middle of the row, following the vendor's reference layout.
Sharing GPUs
Multi Instance GPU (MIG) splits one large GPU into up to seven isolated slices for small jobs. Time slicing shares a GPU between containers without isolation. Whole GPUs and whole servers go to big training jobs.
Scheduling the work
Slurm suits research and HPC (high performance computing), with gang scheduling and topology awareness. Kubernetes suits services, with the NVIDIA GPU Operator for drivers and monitoring and the Network Operator for RDMA NICs. Run:ai adds fair sharing on top.
Power and cooling
Density decides the cooling
Before any switch is chosen, I check whether the hall can power and cool the racks. Try a few configurations and see where air stops being enough.
Swipe sideways to see the whole diagram.
GPU power
Servers
GPUs each
| Rack load | about 59 kW |
|---|---|
| Cooling | Rear door heat exchanger |
| Air needed | about 4.1 cubic metres a second |
| Power feed | about 90 A per side, A and B |
Indicative figures. GPUs are taken as about 55 percent of server draw, close to NVIDIA's figures of about 14 kW for eight GPU DGX B200 and B300 systems. DGX B200 is air cooled and 10U tall, and NVIDIA fits four DGX B300 in a rack at about 56 kW. Racks with more than four eight GPU servers assume 4U liquid cooled servers. Feeds are 400 V three phase, A and B. Air warms by 12 degrees, and liquid takes about 80 percent of the heat with a 10 degree rise. Real designs use the vendor's measured figures.
Air has a limit
Air needed rises with load: a 15 kW rack needs about one cubic metre of air a second, a 100 kW rack nearly seven. That is why dense AI goes liquid.
Zone by density
I group liquid cooled racks together, with their coolant units and pipework, rather than scattering 100 kW racks through an air cooled aisle.
Liquid still leaves heat
Direct to chip captures most of the heat, but not all. The residual air load still has to be cooled, so I plan both.
Data pipeline
Feeding GPUs at the speed they work
Storage has two hard jobs in an AI cluster: feeding data to many GPUs at once, and absorbing checkpoints when a job saves its state.
Swipe sideways to see the whole diagram.
Raw data from applications, plants and logs lands in object storage. It is cheap, durable and scales without practical limits, so it is the right home for the full data set.
Curated training sets are staged onto a parallel file system, where many storage nodes serve one namespace at the same time.
During training every GPU reads at once. Parallel reads, a local NVMe cache and direct paths into GPU memory stop the GPUs from waiting on storage.
Jobs save their state at intervals. Checkpoints arrive as large write bursts, so I size the storage and its network for the burst, not the average.
Models, checkpoints and training data are valuable and often confidential. They stay inside the AI zone, encrypted, with access controlled and logged.
AI technology stack
From the building to the model
An AI platform is a stack of layers, and a weak layer anywhere leaves GPUs idle. Choose a layer to see what it holds, what I bring to it and the training behind it.
Swipe sideways to see the whole diagram.
What I bring
Facility. I design AI ready data centre infrastructure for high density compute: network, power, cooling and structured cabling, with the rack layouts the build teams work from. I also delivered the £25M Network Rail data centre from an empty shell.
Related training (specialist training, trained): Power and Electrical; Heating, Ventilation and Air Conditioning; Fibre and Copper Cabling.
GPU servers. I have prepared data centre space for high performance workloads: rack allocation, structured cabling pathways and deployment coordination.
Related training (NVIDIA, course completed): Cable Validation Tool (CVT) Fundamentals.
Networks. This is the layer I know best: more than twenty years of network design, from Cisco Nexus and ACI data centres to VXLAN, BGP and segmented fabrics.
Related training (NVIDIA, course completed): InfiniBand Essentials; Cumulus Linux Essentials; SONiC Essentials by NVIDIA; NetQ Deployment and Installation; Getting Started with NVIDIA DSX Air.
Storage. I size the storage network for the checkpoint burst, keep it apart from the back end, and keep training data inside the AI zone with encryption and access control.
More on this page: the data pipeline section.
OS and drivers. NCCL, the NVIDIA Collective Communications Library, runs collectives such as all reduce over NVLink, InfiniBand or RoCE. The network has to give it what it expects: rails, a lossless class and each NIC close to its GPU. I automate builds with Python and Ansible.
Related training (NVIDIA, course completed): NVIDIA License System (NLS), which serves licences for NVIDIA enterprise software.
Cluster management. Base Command Manager provisions nodes, runs the workload manager and monitors the cluster. I plan the management networks and the provisioning path it depends on, and I have trained on running it.
Related training (NVIDIA, course completed): Base Command Manager Administration.
Frameworks. I do not tune models. My part is the platform under them: data paths, storage throughput and a fabric with a short tail, so every GPU stays busy.
Related training (NVIDIA, course completed): AI for All: From Basics to GenAI Practice.
MLOps (machine learning operations) and models. For most organisations, inference outgrows training once models are in use, so I size GPUs, network and facility for the real workload, not the biggest reference design. Models also need context. In my operational technology (OT) work I structure plant data in a Unified Namespace named to ISA-95.
Related training (OpenAI, course completed): Scope AI Solutions; Optimize AI Application Performance.
Governance. I start with ownership, risk and the rules for data and models before a platform is chosen. My consultancy, GoNexL.AI, provides AI strategy, governance and fractional Chief AI Officer services.
Related training (course completed): Google: Responsible AI: Applying AI Principles with Google Cloud; Google: Security in the World of AI; OpenAI: AI Leadership; Anthropic: AI Fluency: Framework and foundations.
HPC and big data
Not every cluster talks the same way
Before I size a network for a cluster, I ask how its nodes talk. Some jobs need every node in constant conversation. Others never let nodes speak at all. That choice changes the fabric, the storage and the cost by a large factor.
Swipe sideways to see the whole diagram.
Tightly coupled. A simulation or a training job splits one problem across many nodes, which swap results constantly through MPI (Message Passing Interface) or NCCL. Every node waits for the slowest message, so low latency and a lossless fabric decide the speed.
Loosely coupled grid. A risk or pricing grid, such as a Monte Carlo run at a bank, splits into thousands of independent tasks. Nodes never talk to each other, so ordinary servers and networks scale it out cheaply.
Big data. Hadoop and Spark move the compute to the data: each task runs where a copy of its block already sits, and HDFS (the Hadoop Distributed File System) keeps three copies by default. The heavy network moment is the shuffle between stages. Kafka feeds events in, with partitions spread across brokers and usually three replicas.
InfiniBand in brief
InfiniBand was built for this. A subnet manager, such as NVIDIA UFM or OpenSM, discovers the fabric and works out every route centrally. Credit based flow control means a sender only transmits when the receiver has room, so nothing is dropped. Current generations run at 400 Gb/s (NDR) and 800 Gb/s (XDR, extreme data rate).
Work done inside the switch
InfiniBand switches can run SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), which adds up results from many GPUs inside the network itself, so less data crosses each link during a collective operation.
Ultra Ethernet
The Ultra Ethernet Consortium, under the Linux Foundation, published its 1.0 specification in June 2025. It adds packet spraying, new congestion control and RDMA style transport to Ethernet, aimed at the same jobs.
Cost
What a GPU hour really costs
Owning GPUs only pays if they stay busy. I model the full cost of one eight GPU server per year, then divide by the hours the GPUs are actually used. Change the inputs to see how much utilisation matters. Power is taken at full draw, so the figure is slightly pessimistic at low use. All figures are illustrative.
| Item | Per year |
|---|---|
| Server, spread over its life | £70,000 |
| Network and storage share | £20,000 |
| Power, including cooling | £22,776 |
| Software, support and staff | £20,000 |
| Total | £132,776 |
| Per GPU hour, if always busy | £1.89 |
| Per GPU hour, at real use | £2.71 |
At 70% busy, each used GPU hour costs £2.71. Compare that with a cloud quote for the same GPU and term.
Four rack scale systems in a 15 kW hall
Each rack draws about 120 to 140 kW and needs liquid cooling. I check power at the right point, A and B feeds through busway and panel ratings, then the facility water for coolant units, floor loading and lead times. Often the answer is a liquid ready zone built in phases, or colocation in the meantime.
An AI pod in a regulated estate
The pod is its own pod type and security zone. The GPU back end has no route anywhere else. The front end joins the estate through border leaves and firewalls, and the usual logging, identity and change controls apply.
What I ask a specialist first
The workload mix of training, fine tuning and inference, the largest single job in GPUs and the three year growth. Then the system type, its reference design and scalable unit, and the storage throughput the job really needs.
Engagement
How I run an AI infrastructure engagement, from workload to running pod
AI infrastructure is expensive, power hungry and quick to age, so I start from the workload, not the hardware. Then I make sure the building, the network and the operations team are ready before the first rack arrives.
Swipe sideways to see the whole diagram.
I start with the work: training, fine tuning or inference, the largest single job, the data it needs and how fast demand will grow. That decides whether to build, colocate or rent, and how big the first step should be.
Functional requirements set the GPUs, the networks and the storage each workload needs. Non functional ones put numbers on fabric bandwidth, storage throughput, power per rack, cooling, job completion time, availability and how the data is protected.
I check the facts on site: spare power at the transformer, switchgear, uninterruptible power supply (UPS) and busway, cooling capacity and facility water, floor loading, cable routes and lead times. Most AI plans are limited by the building long before the budget.
I start from the vendor's reference architecture for the exact system and scalable unit, then fit it to the site: rail optimised fabric, storage, management, power and liquid cooling. Departures from the reference are recorded with their reasons.
Racks are built and burned in at the factory, then every cable is validated on site and bandwidth is proved with NCCL tests across the whole pod. Only then is it accepted and handed to the platform team.
I agree who watches what: GPU health and errors, fabric congestion, cooling and power, and the scheduler. Faulty GPUs and links are drained from the scheduler automatically, so one failure does not stall a large job.
GPU generations change every year or two. I keep firmware and drivers current in waves, track utilisation and the cost of each GPU hour, plan capacity quarterly and decide early whether older systems move to inference or retire.
Functional requirements
What it must do.
- Provide GPU capacity for the agreed mix of training, fine tuning and inference.
- Join GPUs through a non blocking back end fabric, separate from the front end and storage networks.
- Feed the GPUs with storage fast enough for data loading and checkpoints.
- Schedule and share GPUs fairly between teams, with quotas.
- Keep the GPU network and training data isolated in their own security zone.
- Monitor GPUs, fabric, power and cooling from one place.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Throughput
- Measured bus bandwidth close to the reference figure for the system, proved with NCCL tests.
- Job time
- Training time per epoch or step agreed, and the slowest flow tracked rather than the average.
- Power
- Rack power and the whole power chain sized with headroom, A and B fed.
- Cooling
- Liquid cooling sized to the rack, with the remaining air load also covered.
- Availability
- Failed GPUs and links taken out of service automatically, so jobs restart from the last checkpoint.
- Utilisation
- A target for how busy the GPUs are, reported monthly, because idle GPUs are the biggest cost.
- Security
- Back end fabric with no route out, data encrypted, and access through the usual identity controls.
Typical starting targets. The workload and the vendor's reference design set the real numbers.
Who I talk to
- Data science, research and AI platform teams
- Facilities, electrical and mechanical engineers
- Network, storage and security teams
- Finance, for build against rent decisions
- The GPU vendor and integrator
How I find out
- Workload workshops: models, data sizes and growth
- Site survey of the power chain, cooling, water and floor loading
- Review of current GPU use and queues, if any exist
- Reference architecture review with the vendor
- Cost model for build, colocation and cloud
Discovery tools
| Tool or source | What it tells me |
|---|---|
| Data centre infrastructure management (DCIM) and power monitoring | Real headroom at each point in the power chain, not the nameplate figure. |
| Building management system (BMS) and cooling data | Cooling capacity, water temperatures and how close the hall runs to its limits. |
| NVIDIA DCGM (Data Center GPU Manager) | GPU use, memory, power and errors on any existing GPUs. |
| Scheduler reports (Slurm or Kubernetes) | Queue times and how busy existing GPUs really are. |
| Storage performance tests | Whether the storage can feed the GPUs and absorb checkpoints. |
| Cable validation and NCCL tests | That the fabric delivers the bandwidth the design promises. |
AI clusters fail in a particular way: one slow GPU, link or cable can hold back a whole job. So I look for the outlier, not the average.
My approach
- Define. Which job, how many GPUs, and is it slow, stuck or failing? Since when, and what changed: driver, firmware, cable or job size?
- Find the outlier. Compare every GPU and link. One node with errors, or one rail with congestion, usually explains a slow job.
- Check the GPU. DCGM diagnostics, temperatures, ECC (error correcting code) memory errors and Xid errors in the kernel log.
- Check the fabric. Link errors, PFC pause counters, ECN marks and congestion per rail. On InfiniBand, check the subnet manager's view.
- Check storage and power. Storage throughput during data loading and checkpoints, and any power or thermal throttling.
- Fix and record. Drain the faulty node or link, fix it, test with NCCL, then return it to the scheduler and record the cause.
Faults I see most
- A single bad cable or optic slowing an entire collective
- PFC pause storms spreading through the fabric
- GPUs throttling because of heat or power limits
- Driver and firmware versions out of step across nodes
- Storage too slow for checkpoints
- A job placed across rails or racks badly by the scheduler
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Optics, cables and link errors, checked with the cable validation tool and switch counters. On InfiniBand, ibstat and ibdiagnet. |
| 2 · Data link | PFC (priority flow control) pause counters, priority settings and lossless queue configuration on every hop. |
| 3 · Network | Rail routing, ECMP (equal cost multi path) balance and, on InfiniBand, routes from the subnet manager. |
| 4 · Transport | RoCEv2 congestion: ECN (explicit congestion notification) marks, DCQCN behaviour and retransmissions. |
| 5 to 7 · Application | GPU health with dcgmi diag -r 3 and nvidia-smi -q -d ECC, bandwidth with all_reduce_perf -b 8 -e 8G -f 2 -g 8 from the nccl-tests suite, job logs, and what the scheduler placed where. |
My general method is on the troubleshooting page.
An AI pod needs the platform, the fabric and the building to be supported together, because a fault in any one of them stops the jobs.
Who runs what
The AI platform team runs the scheduler and software, the network team the fabrics, and data centre operations the power and cooling.
Health checks
Automated checks drain a bad GPU or link from the scheduler before the next job lands on it.
Monitoring
GPU health, fabric congestion, storage throughput, power and coolant temperatures on one dashboard.
Spares
Spare GPUs, optics, cables and fans on site, sized to the failure rates seen in burn in.
Vendor support
Clear escalation for hardware faults, with the evidence the vendor will ask for collected automatically.
Capacity
Queue times and utilisation reviewed monthly, so growth is planned rather than discovered.
GPU platforms age quickly. I plan their whole life, including what happens when the next generation arrives.
- Firmware. Drivers, firmware and the software stack updated in waves, the same version across the pod.
- Utilisation. Monthly reports of how busy the GPUs are, and what each used GPU hour costs.
- Capacity. A quarterly capacity plan tied to the workload roadmap and the building's limits.
- Facilities. Power and cooling headroom tracked, with the next liquid ready phase planned before it is needed.
- Reuse. Older systems move from training to inference or development before they retire.
- Disposal. Drives and GPUs with memory are wiped to a recorded standard, such as NIST SP 800-88 from the US National Institute of Standards and Technology, before they leave.
Projects
Where I have done this work
My AI infrastructure work builds on twenty years of data centre, network and security delivery, backed by focused training on AI fabrics and tools.
AI ready data centre design
Designed AI ready data centre infrastructure for high density compute clusters, covering network, power, cooling and structured cabling, with the rack layouts and deployment documentation the build teams worked from.
High performance workloads
Prepared data centre infrastructure for high performance workloads: rack allocation, structured cabling pathways and deployment coordination.
AI ready plant data
Shaped a phased IT/OT roadmap for Dyson that includes a Unified Namespace for plant data. Supported secure MQTT (a lightweight messaging protocol) data paths from production lines to manufacturing execution systems (MES) and cloud data platforms.
AI consultancy
I founded GoNexL.AI, which provides AI consulting and fractional Chief AI Officer services to organisations adopting AI.
AI networking and cluster tools
InfiniBand Essentials, Cumulus Linux Essentials, SONiC Essentials by NVIDIA, NetQ Deployment and Installation, Base Command Manager Administration, Getting Started with NVIDIA DSX Air, and Cable Validation Tool (CVT) Fundamentals.
Responsible and applied AI
Responsible AI: Applying AI Principles with Google Cloud, Security in the World of AI, Introduction to Large Language Models, AI Leadership from OpenAI, and AI Fluency: Framework and foundations from Anthropic, among others.
See the data centre page for the physical design side, and all my credentials for every course and certification.
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.