IT load
How many kW the IT draws today and at its peak. It sets the size of the power and cooling plant, and the grid supply behind them.
Data centre facilities
The building under the network
I plan the space, power and cooling that a data centre network depends on. That means how big a hall needs to be, how racks are laid out, how power, heat and light move through it, and the building management system (BMS) that watches it all. Every diagram can be stepped through at your own pace.
Overview
What sets the size and shape of a hall
Four numbers shape a data hall before anyone draws a rack: how much power the IT needs, how densely it is packed, how resilient it must be, and how fast it will grow. The same load can look very different on the floor.
Swipe sideways to see the whole diagram.
10 kW a rack. Typical enterprise and virtualisation racks. One megawatt of IT fills about 100 racks across four rows, and air with contained aisles cools it comfortably. Floor space is usually the limit.
40 kW a rack. Air cooled GPU servers. The same megawatt fits in 25 racks, but air alone cannot carry the heat away. Rear door coolers take it into chilled water at every rack, so pipework reaches the rows.
120 kW a rack. Rack scale AI systems. Nine racks carry the whole megawatt. They need direct liquid cooling through coolant distribution units (CDUs), very large power feeds and a floor rated for racks of well over a tonne. Some of the heat still leaves in air, so the row keeps a little air cooling. Power and cooling are now the limit, not space.
Density
How that load is spread across racks. It decides air or liquid cooling, the size of each feed and how much the floor has to carry.
Resilience
N+1 or 2N for power and cooling. 2N roughly doubles the electrical plant, so I match it to what the services in the hall need.
Growth
How fast the load rises. Halls are fitted out in phases, so I plan when the next block of power and cooling is needed, before it is needed.
Capacity planning
Plan the hall before it fills
Set the racks, the density, the growth, the design PUE (power usage effectiveness) and the resilience, and see what the hall needs today and the year it runs out. This is the first conversation I have with the facility team, before any design is drawn.
Redundancy
IT load today960 kW
IT load in year 51.55 MW
Facility power today1.3 MW
Heat to remove960 kW
UPS plant2 × 2 MW
Hall fullYear 6
Show the numbers
| Year | IT load | Facility power | Headroom |
|---|
Indicative only. It assumes steady growth every year, a PUE that holds at the design figure and a power factor of 0.95. Rack feeds are loaded to no more than 80 percent, and uninterruptible power supply (UPS) modules are 250 or 500 kW. A real plan checks every stage of the chain, from the grid supply to the rack, not just the hall total.
Data hall layout
A hall planned before it is built
TIA-942, the US Telecommunications Industry Association standard, gives data centre cabling a clear structure, from where carriers arrive to the racks themselves. In the UK and Europe, BS EN 50600 covers the same ground. I map the network design onto those spaces, and lay the cooling and pathways around them, so the builders and the network team mean the same thing.
Swipe sideways to see the whole diagram.
Carriers arrive in two entrance rooms on physically separate routes. This is where their equipment meets mine, so colocation sites often call it the meet me room. Two carriers in the same duct are one failure, not two.
The main distribution area holds the core, the super spine and the border. TIA-942 places it inside the computer room, as central as the layout allows, so backbone runs to every row stay short.
Rack fronts face a cold aisle and backs face a contained hot aisle. Fan walls flood the room with cold air, and the hot exhaust returns to them through the ceiling without mixing.
Pre terminated fibre trunks run overhead from the main distribution area to a network rack at each end of every pod: the horizontal distribution areas. Trays then carry the links along each row.
Two rows facing a shared hot aisle make a natural pod: one block of racks, power, cooling and network that is built, tested and run as a unit.
- Carrier links
- Cold air
- Hot air
- Fibre trunks
The spaces, by name
| Space | What it holds | Notes |
|---|---|---|
| Entrance room (ER) | Carrier equipment and the demarcation point | Two, on diverse routes. The meet me room in colocation. |
| Main distribution area (MDA) | Core, super spine, border and main cross connects | Central, so backbone runs stay short. |
| Intermediate distribution area (IDA) | A second level of distribution | Optional, for very large or multi floor sites. |
| Horizontal distribution area (HDA) | The network rack for a row or a pod | Where trunks from the MDA land. |
| Zone distribution area (ZDA) | A consolidation point within a row | Optional. Useful where racks change often. |
| Equipment distribution area (EDA) | The racks themselves | Patch panels at the top of each rack. |
The cabling standards family: TIA-942 and BS EN 50600 for the facility, BS EN 50173-5 and ISO/IEC 11801-5 for data centre cabling, BS EN 50174 for installation, and TIA-606 for labelling.
White space and grey space
White space holds the IT racks. Grey space holds the plant that keeps them running: switchgear, UPS, batteries, chillers, pumps and generators. Grey space is often larger, and it is where most maintenance happens.
Raised floor or slab
A raised floor carries cold air underneath and suits older, lower density halls. New halls are usually built on a slab, with overhead cabling and fan walls, because a slab takes heavy racks and liquid pipework more easily.
Rack elevations
Three racks, three different jobs
I standardise a small set of rack types and draw every one before it is built. Pick a rack to see its front, its airflow or coolant, and its rear with the cabling and power.
Swipe sideways to see the whole diagram.
Pod network rack. One of two per pod, at opposite ends of the row. 800 mm wide for cable managers and bend radius. Two of the pod's spines, fibre panels, out of band aggregation and a console server, about 3 kW. The empty space is closed with blanking panels, so air only passes through the switches.
Air cooled compute rack. A leaf pair and an out of band switch at the top, then 16 servers. About 15 kW, so each of the A and B feeds is a 32 A three phase supply. Cold air in at the front, hot air out of the back.
Liquid cooled GPU rack. Eight GPU servers at around 100 kW. Cold plates take most of the heat into coolant, which the manifolds carry to a coolant distribution unit. Each GPU has its own network port and its own link to a rail leaf, so eight links leave every server. Racks like this can weigh well over a tonne.
Select a deviceHover over or tap any device in the front view to see what it does.
How I build every rack
- Sealed, not just blanked. Blanking panels in every empty U, side panels between racks, and brush strips or grommets wherever cables pass, so air has no short cut.
- Sized for the kit. 600 mm wide for compute, 800 mm for network racks, and 1200 mm deep for GPU servers and their rear manifolds.
- A and B, colour coded. Two rack power distribution units (PDUs) in different colours, with every dual power supply split across them.
- Management survives a lost feed. The out of band switch takes power from both PDUs, and single supply kit goes on a small rack transfer switch.
- Weight checked. Heavy kit at the bottom, and racks of over a tonne checked against the floor rating and the delivery route.
- Two network racks per pod. Spines and out of band aggregation are split between both ends of the row, so one rack problem cannot take out every spine.
Rack sizes, weight and floors
| Item | Typical figure | Why it matters |
|---|---|---|
| One rack unit (1U) | 1.75 inches, 44.45 mm, on 19 inch (482.6 mm) rails to EIA-310 | Every elevation is planned in U. |
| Rack height | 42U is about 2 m, 48U about 2.2 to 2.3 m, 52U in new builds | Check the loaded height against doors and lifts. |
| Width | 600 mm for compute, 800 mm for network | TIA-942 recommends wider cabinets, typically 800 mm, where switches need side cable management. |
| Depth | 1,000 to 1,200 mm | GPU servers need the full 1,200 mm. |
| Open Rack (OCP) | 21 inch wide, 48 mm OpenU, a DC busbar at the rear | Used for some hyperscale and AI rack scale systems. |
| Compute rack, loaded | About 700 kg on 0.72 m², about 970 kg/m² or 9.5 kN/m² | Within most data hall floor ratings. |
| Rack scale AI system | About 1,360 kg on a 600 × 1,068 mm footprint, about 2,100 kg/m² or 20.8 kN/m², 340 kg on each foot | Needs a structural check and often spreader plates. |
Indicative figures. A structural engineer confirms uniform and point loads for the whole delivery route, not just the final position.
Density bands
| Band | kW a rack | How I cool it |
|---|---|---|
| Legacy | 3 to 8 | Room air |
| Mainstream virtualised | 8 to 15 | Aisle containment |
| High density | 15 to 30 | Containment plus in row cooling or rear doors |
| Air cooled GPU | 30 to 50 | Rear door heat exchangers, three or four 8 GPU servers at about 10 kW |
| AI rack scale | 40 to 130 and rising | Direct liquid cooling is mandatory |
I plan per rack and per row, never on the hall average.
What servers really draw
A two socket general purpose server draws 600 to 1,000 W in real use, even with power supplies rated 1,100 to 1,800 W each. An 8 GPU HGX class server draws 10 to 15 kW. A GB200 or GB300 NVL72 class rack is well over 100 kW. Power supplies draw an inrush current when switched on, so switched rack PDUs power outlets up in sequence. Power capping through the baseboard management controller holds a server or a rack under a set limit, at some cost in performance.
Switch airflow direction
A top of rack switch with its ports facing the rear, towards the server network cards, needs port side exhaust fans so it still draws cold air from the front. The wrong airflow kit is one of the most common factory mistakes.
When the hall allows 20 kW and the servers need more
I spread the load, for example two GPU servers a rack across more racks, checking cable and optic reach. Where power exists but cooling is short, rear door heat exchangers help. It is a pod type decision with a cost, so it goes in the decision matrix.
The rack build pack
- The test. A build team can install the rack without calling the designer.
- Elevations. Every device by U position, front and rear.
- Port map and cable schedule. Port to port, cable type and length, labels at both ends to TIA-606.
- Power schedule. Every device by PDU and outlet, with phase balance.
- Floor plan, BOM and sequence. Rack position, bill of materials, install order, and the test and acceptance criteria.
- One source. All of it generated from the source of truth, and updated as built.
Power chain
Every rack fed twice, from separate paths
A fabric is only as resilient as the power under it. I follow the chain from the grid to the power supply in the server, and look for anything both sides quietly share.
Swipe sideways to see the whole diagram.
Every rack gets an A and a B side, fed from separate utility supplies where the site has them. The red flag I look for is A and B sharing a board, an uninterruptible power supply (UPS) bypass or a cable route upstream.
A transformer on each side steps the high voltage supply down to 400 V three phase for that side's switchboard.
Each side has generators on an automatic transfer switch, with enough fuel on site to run for at least 12 hours.
A double conversion UPS runs the load from its inverter all the time, with the batteries already on its DC bus. A mains failure causes no break at all.
Busway or floor power distribution units (PDUs) carry each side along the rows to rack PDUs A and B. Every server has two power supply units (PSUs), one plugged into each side.
Utility A drops. The UPS on side A carries on from its batteries with no break, and side B carries on as before.
Within about 10 to 15 seconds the generator is running and the transfer switch moves side A onto it. The batteries recharge, and the server never noticed.
Swipe sideways to see the whole diagram.
N+1, a module fails. Three UPS modules where two are enough. One fails and the other two share the load. N+1 protects against a unit failing.
N+1, the path fails. The distribution board after the UPS fails. Both rack PDUs hang off it, so the server loses both supplies. N+1 covers a unit failing, not a shared path. A second distribution path, or a full 2N design, covers this.
2N, a side fails. Two complete systems, each able to carry the whole load. Side A is lost from the switchboard down, and side B carries everything. This is why each side is planned to run at no more than 40 percent in normal running, so the survivor sits at no more than 80 percent.
How I check a power design
- Trace both sides end to end. From the utility feed to the server inlet, A and B never share a board, a bypass or a cable route.
- Every stage has a limit. A rack can have spare power while the UPS block feeding its row is full. I check each stage, not just the rack.
- Budget on measured power. A server label might say 1.4 kW and the server peak at 0.85 kW. Budgeting on the label strands power nobody can use.
- Single corded kit needs care. A device with one power supply needs a transfer switch, or it becomes the single point of failure.
- Cooling stays up in a power cut. Fans and pumps on UPS, or a dense hall overheats in minutes while the generators start.
- Prove it under load. I ask for acceptance tests that pull a feed with the hall running, rather than trusting the drawings.
Tiers, ratings and classes
The Uptime Institute Tiers describe resilience. Tier I has no spare capacity. Tier II adds spare units. Tier III can be maintained without a shutdown. Tier IV keeps running through any single fault. TIA-942 uses Rated 1 to 4 for the same idea, and BS EN 50600, the European standard, uses availability classes 1 to 4. I design to the level the services need, and only call a site certified when it is.
Power maths I use every week
- kVA is what the cables carry. Volts times amps. kW is what the equipment turns into work and heat. kW = kVA × power factor, which is 0.95 to 0.99 for modern power supplies.
- Single phase in the UK. 230 V, so kVA = 230 × amps ÷ 1,000. A 32 A feed gives 7.4 kVA.
- Three phase in the UK. 400 V between lines, so kVA = 1.732 × 400 × amps ÷ 1,000. A 32 A feed gives 22.2 kVA.
- Plan to 80 percent. I plan continuous load to 80 percent of a breaker's rating. With A and B feeds, each normally carries 40 percent or less, so either can take the whole rack.
- Balance the phases. Load spread evenly across L1, L2 and L3, checked on the rack PDU readings at commissioning.
- Measured, not nameplate. I budget on measured peak draw. GPU servers are the exception: training runs every GPU hard together, so I plan near the vendor maximum.
| Each feed | kVA | At 80% | kW at 0.95 | Typical use |
|---|---|---|---|---|
| 16 A single phase | 3.7 | 2.9 | 2.8 | Light racks and network cabinets |
| 32 A single phase | 7.4 | 5.9 | 5.6 | Older compute racks |
| 16 A three phase | 11.1 | 8.9 | 8.4 | Mainstream racks up to about 8 kW |
| 32 A three phase | 22.2 | 17.7 | 16.9 | The 15 kW class compute rack |
| 63 A three phase | 43.6 | 34.9 | 33.2 | 30 kW plus, air cooled GPU racks |
| 125 A three phase | 86.6 | 69.3 | 65.8 | Busway tap offs for dense AI racks |
Planning figures for UK supplies. Each rack has the same supply on its A side and its B side.
Size a rack yourself
Swipe sideways to see the whole diagram.
Each feed
| Apparent power | 15.8 kVA |
|---|---|
| One feed gives, planned at 80% | 22.2 kVA, 17.7 kVA usable |
| Feeds on each side | 1 A and 1 B |
| Each feed with both sides up | 36% of its rating |
| Air: heat and flow | 15 kW, 1.04 m³/s (3,750 m³/h, 2,207 CFM) |
| Liquid: heat and flow | none |
Each side can carry the whole rack on its own, and neither runs above half its rating in normal use.
CFM means cubic feet per minute, and BTU/h British thermal units an hour, the units many cooling data sheets use.
Redundancy shorthand
| Scheme | For 1 MW with 500 kW UPS units | What it survives |
|---|---|---|
| N | 2 units | Nothing. Any failure or maintenance drops load. |
| N+1 | 3 units on one path | One unit failing. Not a failure in the shared path. |
| 2N | 2 + 2 units on two separate paths | A whole path failing, or being maintained. |
| 2(N+1) | 3 + 3 units on two paths | A whole path, plus a unit on the other. |
| Distributed redundant | For example 3 units sharing to make 2, each at about 67% or less | One unit or path. Efficient, but harder to reason about. |
| Block redundant | N blocks plus a reserve block switched in by a static transfer switch | One block. Common in large colocation sites. |
| 2N+1 | 2N plus one spare unit | A path and a unit at the same time. |
For an important banking service I want 2N all the way to the rack.
UPS and batteries
Data centre UPS units are double conversion: mains to DC to AC all the time, with the batteries on the DC bus, so there is no break at all. Valve regulated lead acid (VRLA) batteries are cheaper but heavy and heat sensitive. Lithium ion is smaller and lasts longer but needs its own battery management and fire risk assessment. Autonomy is usually 5 to 10 minutes, enough for the generators.
Generators and transfer switches
Generators, increasingly run on HVO (hydrotreated vegetable oil), take the load in about 10 to 15 seconds. Uptime asks for at least 12 hours of fuel. Banks often ask for 24 to 72 hours plus priority refuelling. An automatic transfer switch (ATS) moves between mains and generator. A static transfer switch (STS) moves single corded loads between two live sources in a few milliseconds.
The last few metres
Floor PDUs, remote power panels (RPPs) with breakers near the rows, or overhead busway with tap offs feed the racks. Rack PDUs are basic, metered, monitored or switched, and outlet metering supports chargeback. IEC 60320 C13 outlets carry 10 A and C19 outlets 16 A. Commando plugs to IEC 60309 are blue for 230 V and red for three phase.
Stranded power, worked through: 16 servers with 1,400 W nameplate power supplies look like 22.4 kW. Measured at 850 W each they draw 13.6 kW, so a rack sized on nameplate strands 8.8 kW, about 140 kW across a 16 rack pod.
Cooling chain
From the chip to the outside air
Every watt of IT power becomes heat that has to go somewhere. I design the chain as a whole, from the chip to the plant, because the weakest link sets the density the hall can take.
Swipe sideways to see the whole diagram.
Almost every watt the IT draws ends up as heat at the chips. The cooling chain moves it from there to the outside air, and every stage needs the same redundancy thinking as power.
Server fans push air over the heat sinks into the contained hot aisle. Fan walls or computer room air handlers (CRAH) pass it over chilled water coils, and cold air goes back into the room. Older rooms use computer room air conditioners (CRAC), which have their own compressors.
Cold plates sit on the GPUs and CPUs. A coolant distribution unit (CDU) keeps the clean server loop apart from the building water. Most of the heat leaves in the liquid, and the rest still goes to air.
Pumped water carries the heat from the coils and CDUs to the plant. Cold plates can run on much warmer water than air coils, so many sites run two water temperatures.
Chillers use compressors to make cold water. When the outside air is cool enough, dry coolers reject the heat without them. In the UK that is possible for much of the year.
Warm return water can heat nearby buildings or feed a local heat network, often through a heat pump that raises its temperature first.
Swipe sideways to see the whole diagram.
Containment. Rack fronts face the cold aisles and rack backs face a sealed hot aisle. Hot exhaust rises through a chimney into the return plenum and never mixes with the cold supply. ASHRAE recommends 18 to 27 °C at the server inlet. Blanking panels in every empty U keep the air on the right side.
Cooling options by density
| Option | How it works | Typical rack density |
|---|---|---|
| Computer room air conditioner (CRAC) | A room unit with its own compressor. Common in smaller rooms. | Up to about 10 kW |
| Computer room air handler (CRAH) | Fans blow over coils fed with chilled water from central chillers. | About 10 to 20 kW with containment |
| Fan wall | A wall of fans and coils in a gallery beside the hall floods it with cold air. | About 15 to 30 kW with containment |
| In row cooler | A cooling unit sits between racks, so the air path is short. | About 20 to 40 kW |
| Rear door cooler | A water coil in the back door cools the exhaust before it leaves the rack. | About 20 to 50 kW, more with fans in the door |
| Direct to chip | Cold plates on the GPUs and CPUs, fed by a CDU. | About 50 kW to well over 100 kW |
| Immersion | Whole servers sit in tanks of fluid that does not conduct electricity. | Very high, often over 100 kW a tank |
All figures are indicative. The real limit depends on the hall, the containment and the water temperatures.
How the cooling is controlled
- Supply air to a set point. The heating, ventilation and air conditioning (HVAC) units hold the supply air at a set point inside the ASHRAE recommended inlet range of 18 to 27 °C, rather than chilling the whole room.
- Fans follow pressure. Fan wall speed follows the pressure difference across the containment, so the cold side always has slightly more air than the servers draw.
- Humidity by dew point. Dry enough to avoid condensation, damp enough to limit static. As an indicative range, a dew point of minus 9 to 15 °C and no more than about 60 to 70 percent relative humidity.
- N+1 plant. At least one spare fan wall, chiller and pump, so any one unit can fail or be serviced with the hall at full load.
- Alarms that matter. The building management system (BMS) alarms on high inlet temperature, humidity out of range, lost pressure difference, a unit or pump failing, high water temperature and any leak.
- Measured at the rack. Sensors sit at the rack inlets, top, middle and bottom, because that is the air the servers actually breathe.
PUE, simply
A watt of overhead for every IT watt
Power usage effectiveness is total facility power divided by IT power. At 2.0, cooling and losses use as much as the IT. Typical of older rooms.
Roughly the industry average
Industry surveys have put the average close to 1.5 for some years. Containment, warmer supply air and free cooling are the usual ways down.
What new builds aim for
Warm water liquid cooling and long free cooling hours make this possible. I use annual figures, not a cold day snapshot. All figures here are indicative.
Cooling maths
- Air flow. m³ a second = kW ÷ (1.2 × temperature rise in K). 15 kW at a 12 K rise needs about 1.0 m³/s (3,750 m³/h, about 2,200 cubic feet per minute). 100 kW needs about 6.9 m³/s.
- Water flow. Litres a minute = kW × 60 ÷ (4.18 × temperature rise in K). 80 kW at a 10 K rise needs about 115 L/min.
- Heat in other units. 1 kW is about 3,412 BTU an hour, the figure many cooling data sheets use.
- Inlet temperature. ASHRAE recommends 18 to 27 °C at the server inlet. Class A1 allows 15 to 32 °C. Servers usually raise the air 10 to 20 K.
A worked pod with mixed densities
Swipe sideways to see the whole diagram.
Sixteen racks at 15 kW and four at 100 kW make 640 kW of IT load. I never mix them in one air aisle: the AI racks get their own zone, with the pipework and floor strength they need.
Direct to chip cooling takes about 80 percent of the AI racks' heat, so 320 kW goes to water. At a 10 K rise that is about 460 litres a minute, through N+1 coolant distribution units (CDUs).
The air system still removes the 240 kW from the 15 kW racks and the 80 kW the cold plates miss: about 320 kW. At a 12 K rise that is about 22 m³ of air a second.
A 15 kW rack needs one 32 A three phase feed on each side. A 100 kW rack needs about 105 kVA on each side, which at 80 percent planning means four 63 A three phase feeds a side, or a busway tap off.
An AI rack weighs about 1.4 tonnes, so I check point loads along the whole delivery route. The UPS must also handle the fast load swings of AI training.
| Containment | How it works | Where I use it |
|---|---|---|
| Cold aisle | Doors and a roof over the cold aisle. The room runs hot. | Easy retrofit on a raised floor. |
| Hot aisle | The hot aisle is closed and returned to the cooling units through a ceiling plenum. The room stays cool. | Most new builds, one contained aisle per pod. |
| Chimney cabinets | Each rack has a duct from its rear to the ceiling return. | Single dense racks in mixed rooms. |
Containment only works with blanking panels and sealed cable entries. Fire detection and suppression must still reach inside it, and roof panels may drop on an alarm.
Direct to chip, in detail
Cold plates take about 70 to 85 percent of a server's heat. A CDU sits between the facility water system (FWS) and the technology cooling system (TCS) loop that runs through the racks. It controls temperature, pressure and flow, filters the coolant and isolates the two loops. Warm facility water, in the 30s °C, allows free cooling and heat reuse. Immersion, with servers in a tank of single or two phase fluid, removes nearly all the heat, but it changes how racks are serviced and supported, so I treat it as a niche choice.
Efficiency figures
Power usage effectiveness (PUE) is total facility energy divided by IT energy, so 1,300 over 1,000 is 1.3. Data centre infrastructure efficiency (DCiE) is the inverse, as a percentage. Partial PUE covers one zone or system. Water usage effectiveness (WUE) is litres per IT kWh. Evaporative cooling lowers PUE but raises WUE, and neither says how well the IT itself is used.
Reporting and sustainability
EU sites with 500 kW or more of IT must report energy and water figures each year under the Energy Efficiency Directive. I also look at embodied carbon: refreshing hardware too often adds to it, so I refresh on performance per watt and send old equipment for certified reuse.
Cabling
Cabling from the server to the super spine
Most physical layer faults come from people, not equipment: the wrong cable pulled, an unlabelled link, a tray too full to trace. Structured cabling is cheap insurance against all three.
Swipe sideways to see the whole diagram.
Servers connect their network cards (NICs) to the top of rack leaves with short direct attach copper (DAC). Inside a rack, short fixed cables are cheap, cool and reliable.
Leaf uplinks leave through a fibre panel on pre terminated MPO (multi fibre push on) trunks to the network rack at the end of the row. Multimode OM4 is fine for short runs, but its reach falls as speeds rise: 100G SR4 reaches about 100 m.
The spines reach the main distribution area on single mode OS2 trunks. Single mode reaches kilometres and carries every speed I can plan for, so I use it for anything that leaves a row.
Cat 6A copper links every server's baseboard management controller (BMC) and every rack PDU to the out of band switch. It is cheap, simple and fine at 1G.
Fibre in its own raceway, copper in its own tray and power on its own busway. BS EN 50174-2 sets the separation between data and power cabling.
Every cable carries the same label at both ends, matching the records. Every permanent link is loss tested, and every connector is inspected and cleaned before it is plugged in.
How I approach structured cabling
- Pre terminated trunks. Factory tested fibre trunks make installs faster and cleaner than terminating on site, and a new pod can be connected the day it arrives.
- Count the connectors. Every mated pair costs light. 100G SR4 on OM4 allows only about 1.9 dB, so I limit the panels in a multimode path.
- One polarity method. Fibre type, connector and polarity are chosen once for the whole estate, for the speeds two refreshes away.
- Label once, everywhere. One TIA-606 scheme on both ends of every cable, with the same names the asset and IP records use.
- Room to work. 800 mm network racks, vertical managers and bend radius respected, so a change does not mean unpicking a rack.
- Documented for the builders. Rack layouts, elevations and port maps the installers work from, signed off before anything is pulled.
Typical reach
| Link | Medium | Typical reach |
|---|---|---|
| 400G direct attach copper | Twinax copper | About 3 m, inside the rack |
| 100G SR4 | OM4 multimode, MPO | About 100 m |
| 400G SR8 | OM4 multimode, MPO | About 100 m |
| 400G DR4 | OS2 single mode, MPO | About 500 m |
| 400G FR4 | OS2 single mode, duplex LC | About 2 km |
| 1G and 10G management | Cat 6A copper | Up to 100 m |
Indicative figures. I check each optic's data sheet and the loss budget of the real path before relying on them.
Media and reach
| Medium | Jacket colour | Typical use and reach |
|---|---|---|
| Cat 6A copper | Varies | 10GBASE-T up to 100 m. Out of band switches and building systems. |
| OM3 multimode, 50 µm | Aqua | Legacy. 10G to 300 m, 100G SR4 to 70 m. |
| OM4 multimode, 50 µm | Aqua or violet | 10G to 400 m, 100G SR4 and 400G SR8 to 100 m. |
| OM5 multimode, wideband | Lime | Short wavelength multiplexing, such as 400G SR4.2, to 150 m. |
| OS2 single mode, 9 µm | Yellow | 500 m to 10 km and beyond. My default between racks and rows. |
Multimode reach shrinks as speed rises, which is why I standardise on single mode for anything leaving the rack.
Connectors and polarity
Duplex LC for two fibre links. MPO-12 for SR4 and DR4, which use 8 of the 12 fibres, MPO-16 for 400G SR8, and MPO-24 for trunks. MTP is a brand of MPO. Single mode parallel links use angled physical contact (APC, green) connectors. TIA-568 defines polarity methods A, B and C, and I use one method across the whole estate.
DAC, AOC or transceivers
Passive direct attach copper (DAC) reaches about 3 m at 400G and costs least in money and power. Active optical cables (AOC) reach tens of metres, but a failed end means a new cable. Transceivers with structured fibre reach 100 m to 10 km through patch panels. Breakouts, such as 400G to 4 × 100G, depend on the switch's port groups, so I confirm them in the low level design.
Testing every link
Tier 1 testing uses a light source and power meter, an optical loss test set (OLTS), to measure total loss. Tier 2 adds an optical time domain reflectometer (OTDR), which shows each connector and splice along the fibre. Every connector is inspected and cleaned to IEC 61300-3-35, because dirt is the most common optical fault.
Decode an optic and check its loss budget
Swipe sideways to see the whole diagram.
Loss a connection
| Speed and lanes | 100G over 4 lanes |
|---|---|
| Medium | Multimode fibre, 850 nm |
| Connector | MPO-12, 8 fibres used |
| Reach on this fibre | 100 m |
| Fibre loss | 0.27 dB |
| Connections and splices | 1 dB |
| Total loss | 1.27 dB of a 1.9 dB budget |
| Result | Passes with 0.63 dB to spare |
400G, baseband. D means 500 m on single mode fibre. R means 64B/66B style coding. 4 means four lanes, here four parallel fibre pairs.
Budgets are the IEEE channel insertion loss figures for each optic. IEEE counts the connections between the two transceivers, not the transceiver ports themselves. For fibre loss I use the TIA cable figures, 3.0 dB per km for multimode at 850 nm and 0.4 dB per km for single mode at 1310 nm. IEEE assumes 3.5 dB per km for multimode. Real cables vary, so the manufacturer's data and a tested link always win.
Top of rack, end of row or middle of row
| Layout | Strengths | Watch outs | Where I use it |
|---|---|---|---|
| Top of rack | Short copper or DAC to every server, one rack as the failure domain, built whole in a factory. | More switches, and ports can sit unused. | Leaf and spine pods. |
| End of row | Fewer switches, high port use. | Long horizontal runs, and the whole row shares a failure domain. | Low density rows and out of band copper. |
| Middle of row | Shorter runs than end of row. | Still a shared row switch. | Rail leaves in AI pods. |
BMS and monitoring
A hall that tells you first
The building runs on its own control systems. I treat them as operational technology: kept on a separate network, with data flowing out to the teams who need it. The same zoning ideas are on the operational technology page.
Swipe sideways to see the whole diagram.
The building management system (BMS) monitors and controls the cooling plant: fan walls, chillers and pumps. Their controllers talk BACnet/IP.
A separate electrical power monitoring system (EPMS) reads the UPS, switchgear, generators and meters, usually over Modbus TCP.
Leak detection runs under CDUs and pipework, with temperature and humidity sensors at the rack inlets. Their readings feed the BMS.
Fire detection and access control are life safety and security systems with their own panels. They send status out, and nothing in IT can control them.
All of it sits on its own OT network, never on the office LAN. A firewall lets data out to data centre infrastructure management (DCIM) and the operations centre, so capacity and alarms are visible without exposing the plant.
A leak sensor trips under a CDU. The BMS raises it, the operations centre sees it within seconds, and the on call engineer gets the exact location.
Build and readiness
From an empty shell to a live hall
With 2e2 I led an engineering team on a £25 million data centre build from an empty shell, so I know the order things must happen in, and what must be ready before the first pod arrives.
Swipe sideways to see the whole diagram.
I start from the IT load in kilowatts, now and in five years, and the resilience the business needs. That sets the tier, the hall layout, the pod size and the power and cooling designs, all signed off before ground is broken.
The floor or slab, containment, cable and pipe pathways, fire detection and suppression, and physical security go in before any plant arrives.
Switchgear, UPS, generators, chillers and the final distribution are installed, earthed and tested one system at a time.
The building management system (BMS) and power monitoring get a points list covering every sensor and alarm, on a controls network kept separate from IT.
Level 1 is witness testing at the factory, levels 2 to 4 check installation, start up and function, and level 5 is the integrated systems test (IST). Load banks simulate the IT load, and the mains is pulled to prove the UPS and generators take over.
Trunks are installed and certified, then the out of band network and the network racks land first, so automation can build everything that follows.
Pods roll in, pass site acceptance tests and go live. As built records and the data centre infrastructure management (DCIM) tool are updated before operations sign off.
Commissioning levels
| Level | What is tested | Where |
|---|---|---|
| 1 | Factory witness tests of major plant, such as UPS and generators | At the manufacturer |
| 2 | Installation checks: correct kit, correctly installed | On site |
| 3 | Start up of each piece of equipment | On site |
| 4 | Functional tests of each system on its own | On site |
| 5 | Integrated systems test: every system together, under simulated load, with failures | On site, with load banks |
Before a pod lands
| Area | Must be ready |
|---|---|
| Facility | Commissioned to level 5, IST passed, A and B feeds and cooling at every rack position. |
| Space | Floor rating confirmed, containment built, positions marked, delivery route surveyed. |
| Cabling | Trunks tested with OLTS and OTDR, polarity verified, labelled, as built records issued. |
| Network | Out of band live, super spine ports ready, IP address management (IPAM) allocations made, zero touch provisioning (ZTP) ready. |
| Shared services | Domain name system (DNS), Network Time Protocol (NTP), authentication, certificates, logging and telemetry reachable from the pod management network. |
| Security | Access zones, closed circuit television (CCTV) and, where card data is in scope, the Payment Card Industry Data Security Standard (PCI DSS) controls in place. |
Fire detection and suppression
Aspirating smoke detection, such as VESDA, samples the air for the earliest signs of smoke. Suppression is either a gas, inert gases such as IG-541 and IG-55 or a chemical clean agent, or water, through pre action sprinklers or water mist. Gas needs a sealed room, proven by a door fan integrity test, and pressure relief. Some chemical agents now face fluorinated gas (F gas) and per and polyfluoroalkyl substances (PFAS) restrictions, and lithium ion UPS batteries need their own fire risk assessment.
Physical security in layers
Fence and vehicle barriers, a staffed reception, mantraps, card and biometric access to the hall, cages or locked racks for sensitive kit, and CCTV with recordings kept to policy. Every layer is logged, so access can be traced to a person and a time.
Critical national infrastructure
In September 2024 the UK designated data centres as critical national infrastructure, the first new designation in nearly a decade. That brings support from the National Cyber Security Centre (NCSC) and the National Protective Security Authority, and raises expectations on physical security, power resilience, supply chains and tested recovery.
When I walk a hall we might use
- Power. kW per hall, row and rack, true A and B independence back to separate UPS and switchboards, and headroom on UPS, generators and the utility supply.
- Cooling. Air or chilled water, capacity per row, containment, facility water for liquid cooling and supply temperatures.
- Floor and space. Raised floor or slab, uniform and point loads, ceiling height and space to swing a pod.
- Delivery route. Doors, lifts, ramps and the loading bay, for a 1.4 tonne rack on a pallet.
- Connectivity. Two entrance rooms on diverse routes, the meet me room and the carriers present.
- Operations. Remote hands, spares storage, the staging room and the access process.
Concurrently maintainable or fault tolerant
Concurrently maintainable covers planned work: any part can be maintained without an outage. Fault tolerant covers an unplanned failure, and needs A and B in separate rooms and routes. For services a bank depends on I want at least Tier III equivalent halls, and fault tolerant power and cooling where the tolerance is tight. AI training on lower redundancy can be a recorded decision.
Tier availability figures
You will still see 99.982 percent quoted for Tier III and 99.995 percent for Tier IV. These are old marketing figures, and the Uptime Institute does not tie them to its Tiers today. I design to the topology and test it, rather than quote a number.
Certification and acceptance
Uptime certifies a design (TCDD), a constructed facility (TCCF) and its operations (TCOS). For equipment, factory acceptance tests (FAT) happen before shipping and site acceptance tests (SAT) after installation. The Information Technology Industry Council (ITIC) curve describes the power quality IT equipment should ride through.
Engagement
How I run a data hall programme, from site survey to refresh
Facilities work joins two worlds: mechanical and electrical engineering, and the IT that sits on top. I bring them together around one set of numbers, prove the site with surveys and data, and hand over a hall that operations can run and grow.
Swipe sideways to see the whole diagram.
I start with the people who own the building and the people who own the IT. We agree the load in kilowatts today and in five years, the resilience the business needs, and who decides what, before anyone talks about equipment.
Functional requirements cover what the hall must hold and feed. Non functional ones give numbers: kW per rack, redundancy such as 2N, inlet temperatures, water temperatures for liquid cooling, floor loading, fuel autonomy and how maintenance happens without an outage.
I walk the hall, read the single line diagrams, check A and B independence back to separate switchboards, and look at real readings from the power monitoring and BMS trend logs. A thermal survey shows where air is really going.
I lay out the hall in pods, each matched to a power and cooling block, with a cooling choice per density band, the cabling pathways and the out of band network. The design authority signs it off before anything is ordered.
Plant is commissioned level by level, ending with an integrated systems test under simulated load with the mains pulled. Cabling is certified, the network racks land first, and pods roll in only when the readiness checklist is complete.
Operations get one view of plant and IT through the BMS, power monitoring and DCIM, alarms routed to the right team, a planned maintenance schedule and runbooks for losing a feed, a cooling unit or a CDU.
UPS batteries, cooling plant and racks all have a life. I keep a quarterly capacity review against measured load, a plant replacement plan, efficiency reporting and a decommissioning plan that reuses or recycles equipment through certified partners.
Functional requirements
What it must do.
- Provide A and B power to every rack position, each able to carry the whole rack.
- Cool each density band in the right way: air, rear doors or direct liquid.
- Give every rack structured cabling back to its pod network rack, and out of band copper.
- Monitor power, temperature, humidity, leaks and doors, with alarms to the right team.
- Allow racks of up to the agreed weight to be delivered and installed safely.
- Control and log access to the hall, cages and racks.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Resilience
- 2N power to the rack for important services, N+1 cooling, concurrently maintainable as a minimum.
- Capacity
- kW per rack and per pod agreed, with headroom for the next generation of hardware.
- Environment
- Server inlet within ASHRAE's 18 to 27 °C recommended range.
- Autonomy
- UPS ride through of 5 to 10 minutes, and generator fuel for at least 12 hours, often 24 to 72 for banks.
- Efficiency
- A design PUE agreed up front, often 1.2 to 1.3 for new builds, and measured annually.
- Structure
- Floor rated for the heaviest rack along the whole delivery route.
- Security
- Layered physical security with logged access, meeting the expectations for critical national infrastructure.
These are typical starting targets. Each site agrees its own numbers, and each one is proven at commissioning or acceptance.
Who I talk to
- Facilities and estates managers
- Mechanical and electrical engineers
- The colocation provider or landlord
- IT platform, storage and AI teams
- Health and safety, and security
How I find out
- Site walk of every hall, plant room and delivery route
- Review of single line diagrams and cooling schematics
- Analysis of measured power and temperature history
- Thermal survey or airflow modelling
- Review of maintenance and test records
Discovery tools
| Tool or source | What it tells me |
|---|---|
| DCIM (data centre infrastructure management) | Space, power and ports per rack, and what is really free. |
| Power monitoring (EPMS) and rack PDU readings | Measured load per feed and phase, and true headroom. |
| BMS trend logs | Temperatures, humidity and plant behaviour over months. |
| Thermal camera or airflow modelling | Hot spots, bypass air and recirculation. |
| Load bank and IST records | Whether the plant has really been proven under load. |
| Single line diagrams | Whether A and B really are independent, all the way back to the source. |
Facility faults move fast and affect everything above them, so I work from the source of the problem towards the rack and keep the operations team informed throughout.
My approach
- Define. Which racks, rows or halls are affected, and is it power, temperature, humidity or a leak?
- Protect. Keep the IT running: check the other feed is healthy and loads are within its rating.
- Trace. Follow the chain from utility and generator through UPS and distribution to the rack PDU, or from chiller to CRAH to the aisle.
- Measure. Readings from the BMS, power monitoring and rack PDUs, not assumptions.
- Fix. Through the facilities change process, with the maintenance contractor where needed.
- Learn. Record the cause, update the runbook and feed it into the next design.
Faults I see most
- A tripped rack breaker after load was added to one feed
- A and B feeds that share an upstream breaker
- Hot spots from missing blanking panels or open cable cut outs
- A CDU alarm on flow or pressure
- Phase imbalance on a three phase rack PDU
- Humidity outside range after a cooling change
Layer by layer
| Layer | What I check |
|---|---|
| Utility and generator | Mains status, generator readiness, fuel level, last load test. |
| UPS and distribution | UPS mode and battery health, switchboard and floor PDU readings, breaker states. |
| Rack power | Rack PDU readings per feed and phase, outlet loads, server power supply status. |
| Cooling plant | Chillers, pumps and CRAH status, supply and return temperatures. |
| Airflow and liquid | Inlet temperatures, containment, blanking panels, CDU flow, pressure and leak sensors. |
| Monitoring | Alarm history in the BMS and DCIM, and whether the right team was told. |
My general method, with worked faults, is on the troubleshooting page.
I agree the support model for the hall before the first pod goes live, so IT and facilities teams know who acts on each alarm.
Service model
Who owns each layer: facilities, the colocation provider, maintenance contractors and the IT operations team, with response times.
Monitoring
One view of plant and IT through the BMS, power monitoring and DCIM, with alarms routed by type.
Planned maintenance
Schedules for UPS, generators, cooling and fire systems, each done without an outage.
Runbooks
Steps for losing a feed, a cooling unit, a CDU or the mains, tested in drills.
Access and change
Controlled access to the hall and a change process that covers facilities work as well as IT.
Capacity
Measured power and cooling per rack and pod, reviewed every quarter.
A data hall lasts for decades, but much of what is inside it does not. I plan the whole life from the start.
- Batteries. UPS batteries replaced on their rated life, sooner if testing shows decline.
- Plant. Cooling units, pumps and switchgear on a replacement plan tied to condition and vendor support.
- Racks and pods. Pods refreshed on a cycle, with a new pod built beside the old one.
- Efficiency. PUE and WUE measured each year, with improvements such as higher water temperatures.
- Capacity. Quarterly reviews against measured load, with triggers to order the next pod.
- Decommission. Equipment reused or recycled through certified partners, with records closed in DCIM.
Projects
Where I have built and planned them
From an empty shell to AI ready designs, with the building systems treated as part of the infrastructure, not someone else's problem.
A £25 million data centre from an empty shell
Led 3 engineers across 2e2 projects, including this build, which covered structured cabling, building management systems and mechanical and electrical integration, with building management and power monitoring on a segregated OT network.
AI ready data centre infrastructure
Designed infrastructure for high density compute clusters, covering network, power, cooling and structured cabling, with the rack layouts and deployment documentation the build teams worked from. More on the AI infrastructure page.
Ready for high performance workloads
Prepared data centre infrastructure for high performance workloads: rack allocation, structured cabling pathways and deployment coordination.
A Tier 3 data centre
Tier 3 data centre and managed infrastructure for over 300 customer sites, on Cisco ACI and Nexus. The fabric side is on the data centre networks page.
Building automation networks
LAN, WAN and cloud for multi site real estate and construction, with BACnet/IP building automation and IoT sensor networks.
Monitoring that protects stock
Warehouse, cold chain and temperature monitoring systems at the food depots, linked to the group's enterprise resource planning (ERP) system.
Backed by training in power and electrical systems, HVAC, building management systems, Honeywell building technologies and fibre and copper cabling. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.