Short and fixed inside the rack
Servers reach their leaf switches over short copper or active cables that never leave the rack. Nothing to patch, nothing to trace.
Data centre and switch cabling
Data centre cabling, from the server port to the spine
I plan the cabling of every data centre I design: what joins each server to its switch, how racks reach the network core, and how switches are stacked. This page covers DAC (direct attach copper), AEC (active electrical cables) and AOC (active optical cables), and MPO (multi fibre push on) trunks. It then covers breakouts, FEC (forward error correction), Catalyst stacking, Nexus optics, labels and schedules, testing, and how I find a cabling fault.
Overview
Every link is a cable decision
Switches and servers get the attention, but the cabling decides how fast a pod can be built, how cleanly it can change and what the next speed will cost. This page covers the choices I make inside racks and between them, and how I keep the records straight. The facilities view, with media types, reach tables and a loss budget calculator, is on the data centre facilities page.
Swipe sideways to see the whole diagram.
Structured between racks
Anything that leaves a rack goes through panels and factory terminated trunks, so a change is a patch cord, not a new cable pull.
One standard, written down
Media, connectors, polarity, colours and labels are chosen once for the whole estate and written into the design standard.
Records that match the floor
The patching schedule is the source of truth. Every change updates it first, and the switches prove it afterwards.
A link between two switches passes far more connection points than the drawing on a design suggests. I count them, label them and test them, because each one is a place where light is lost or a cable can be pulled by mistake.
Layouts
Where the switches sit decides how far every cable runs
The facilities page compares the three switch layouts by failure domain and cost. Here I look at the same choice from the cabling side: how many cables cross the row, how long they run and which medium they need. Standards such as TIA-942 then name the spaces each run starts and ends in.
Swipe sideways to see the whole diagram.
Top of rack. Each rack has its own leaf pair, so the 256 server links in this row (16 servers, two links each, in eight racks) stay inside their racks on short DAC (direct attach copper). Only a few fibre uplinks per rack cross the row overhead. This is my default for compute.
Middle of row. The network rack sits in the middle, so all 256 server links leave their racks, but none travels more than about half a row. Fewer switches, more copper in the trays, and every change means patching in the network rack.
End of row. All 256 server links run to one rack at the end. Switch ports are used well, but the tray carries the whole row's copper and the longest runs reach the far rack. I keep it for low density rows and out of band management.
Counts assume eight server racks of 16 servers with two links each, so 256 server links in the row. They are indicative, to show the shape of each layout.
Swipe sideways to see the whole diagram.
A patch cord runs straight from the switch port to the trunk panel. It is the shortest path, with one mated pair at the panel, but every change means handling cords plugged into live switch ports.
Every switch port is cabled once to its own panel and never touched again. Changes are made with short cords between the switch panel and the trunk panel.
A move or a new link is one patch cord between two panels. The switch cabling and the trunks stay as they were installed and tested.
The extra panel adds a mated pair, so it uses part of the loss budget, plus rack space and cost. I use cross connects in the main distribution area, where change is constant, and interconnects inside pods.
The spaces a cable runs between
| Space | What sits there | How I use it |
|---|---|---|
| ER (entrance room) | Where carrier and campus cables enter | Kept separate from the computer room where it can be |
| MDA (main distribution area) | Core and spine switches, the main cross connect | Cross connects, so changes never touch the core |
| IDA (intermediate distribution area) | A second level of distribution in large sites | Only in multi hall or very large buildings |
| HDA (horizontal distribution area) | Row or pod network racks | Fibre panels and the row switches |
| ZDA (zone distribution area) | An optional consolidation point in the floor or overhead | Rare in my designs; pre terminated trunks do the same job |
| EDA (equipment distribution area) | The server and storage racks | Top of rack switches and short in rack cables |
Space names from TIA-942. TIA-942-C, published in 2024, sets a minimum cabinet width of 800 mm in the MDA, IDA and HDA, and keeps LC (Lucent connector) and MPO as the connectors at the equipment outlet. In Europe, EN 50600-2-4 covers the same ground.
MPO trunks
MPO trunks, cassettes and the fibre count that fits
Between racks I use factory terminated trunks: tested before they arrive and plugged in on the day. What sits at each end depends on the optics. Parallel optics take the MPO (multi fibre push on) connector straight through, and duplex optics need a cassette that breaks the trunk out to LC ports. Optic names give the reach and the lanes. SR (short reach) runs about 100 m on OM4 (a multimode fibre grade, like OM3 and OM5); DR (500 m), FR (2 km) and LR (long reach, 10 km) run on OS2 (single mode fibre). A digit counts the lanes, so SR4 (four lanes) uses eight fibres.
Swipe sideways to see the whole diagram.
Base 8, base 12 or base 16
| System | Connector | What it suits | Watch out |
|---|---|---|---|
| Base 8 | MPO-12 connector, 8 fibres used | Trunks in multiples of 8 fibres. Every fibre carries light for 8 fibre optics such as SR4, DR4 and SR4.2, and for the 8 fibre 800G options. | Duplex links need base 8 cassettes. My default for new builds. |
| Base 12 | MPO-12 connector, all 12 fibres used | Older systems built around duplex breakouts: six duplex links per connector. | With 8 fibre optics, 4 fibres in every connector sit idle unless conversion modules are used. |
| Base 16 | MPO-16 connector, offset key | Built for 16 fibre optics. Among 400G Ethernet optics only 400GBASE-SR8 needs it. | MPO-16 does not mate with MPO-12, and the trunks are larger for the same fibre count. |
Base 16 trunks are typically 20 to 30% larger in diameter than base 8 trunks with the same fibre count (Siemon). Most 800G Ethernet options use 2 or 8 fibres, with LC or MPO-12.
Polarity, chosen once
TIA-568 polarity methods A, B and C each work, but mixing them breaks links. I pick one method for the whole estate. How each method maps the fibres is on the fibre optics page.
Angled or flat ends
Single mode parallel optics such as 400GBASE-DR4, and the multimode 400GBASE-SR8, need angled (APC, green) MPO connectors. 100GBASE-SR4 uses flat ended (UPC) multimode MPO. One wrong cord and the link stays dark.
Count for two refreshes
I size trunks for the speeds two refreshes away, with spare fibres in every trunk. Pulling a new trunk through a full tray costs far more than the spare fibres.
Fibre grades, MPO polarity methods and loss budgets for these trunks are explained on my fibre optics page.
In the rack
DAC, AEC, AOC or optics: what joins a server to its switch
Inside a rack, and to the next rack, I choose between four kinds of cable. All four plug into the same SFP (small form factor pluggable) or QSFP (quad small form factor pluggable) cage, so the choice is the cable, not the port. Each trades reach against power, cost, delay and the space it takes at the back of the rack.
Swipe sideways to see the whole diagram.
Passive DAC (direct attach copper). Twinax copper with fixed plugs and no electronics in them. It is the cheapest option, uses almost no power and adds almost no delay. The limits are reach and bulk: at 400G a passive cable stops at about 3 m and is thick and stiff.
AEC (active electrical cable). Copper again, but with a retimer chip in each end that cleans up the signal. That buys reach, about 1 m to 7 m for 800G cables, and a thinner cable, for a little power. Useful between neighbouring racks at 400G and 800G. An older ACC (active copper cable) uses a simpler amplifier, a redriver, and reaches less.
AOC (active optical cable). Fibre with the optics built into both ends. It is light, thin and reaches tens of metres. If one end fails, the whole cable is replaced, and it cannot pass through a patch panel.
Optics and structured fibre. A pluggable transceiver at each end, with patch cords, panels and a trunk between. It costs the most in money and power, but reaches from 70 m to 10 km and any link can be repatched. It is the only choice once a link leaves the row.
| Medium | 25G | 100G | 400G | Notes |
|---|---|---|---|---|
| Passive DAC | 1 m to 5 m | 1 m to 5 m | 0.5 m to 3 m | Cheapest and coolest. Bulk and bend radius grow with speed. |
| AEC | Rarely used | Rarely used | About 1 m to 7 m at 800G | A middle path between DAC and AOC for the fastest ports. |
| AOC | 1 m to 10 m | 1 m to 30 m | 1 m to 30 m | Light and thin. Replace the whole cable when an end fails. |
| Optics and fibre | 70 m to 10 km | 70 m to 10 km | 70 m to 10 km | Most flexible. The only option through patch panels. |
Lengths for DAC and AOC are the Cisco orderable ranges in its 25G, 100G and 400G cable and transceiver data sheets. The AEC range is from Molex for 800G cables. Indicative: I check the switch and the network card compatibility lists for every part.
Try it: choose the cable for a link
Pick the speed, the length and the path. The helper suggests a medium, says why, and flags the FEC (forward error correction) setting a 25G copper cable needs.
Passive DAC. At 2 m a 25G passive copper cable is the cheapest and coolest choice.
- FEC
- None needed up to 2 m on Cisco 25G copper cables. From 2.5 m to 3 m set FC-FEC, and at 4 m and 5 m set RS-FEC, on both ends.
- Example
SFP-H25G-CU2M, the next orderable length at or above the run- Why
- Short, fixed links inside a rack rarely need patching, and 25G twinax is thin enough to dress neatly.
Accelerator racks multiply the count
A GPU (graphics processing unit) server can need a dozen or more cables: one per back end network card, plus front end, storage and out of band links. Which of those can be DAC, and which must be optics, is decided by the rail design on the AI infrastructure page.
Heat and weight
Bundles of thick 400G copper block the rear of the rack and pull on the ports. Past a few cables per server I move to AEC or AOC for airflow and strain.
Compatibility first
A cable that works in one switch can fail in another, or in the server's network card. I order only parts on both vendors' lists.
Breakouts
Breakouts: one fast port, several slower links
A 40G, 100G or 400G port is built from lanes. A breakout cable or optic splits those lanes, so one switch port serves several servers or switches. It saves ports and money, but the port, the cable and the configuration all have to agree.
Swipe sideways to see the whole diagram.
The 100G QSFP28 port is still configured as one 100G interface, Ethernet1/49. Plugged into a breakout cable, none of the four 25G legs comes up.
The breakout command splits the port into four interfaces, Ethernet1/49/1 to Ethernet1/49/4. Each leg now links to its own server at 25G.
Every leg is its own interface: its own speed, FEC, VLANs (virtual local area networks) and description. The leg names go into the patching schedule like any other port.
One leg failing affects one server. But the four legs share one cable and one switch port, so replacing that cable or optic takes all four down. I spread a server pair across different breakouts.
| Parent port | Children | Cable or optic | Where it fits |
|---|---|---|---|
| 40G QSFP+ | 4 × 10G | Copper breakout to four SFP+ plugs, or 40GBASE-SR4 optics with an MPO to LC breakout cord | Older pods moving 10G servers onto 40G leaves |
| 100G QSFP28 | 4 × 25G | Copper breakout to four SFP28 plugs, 1 m to 5 m: QSFP-4SFP25G-CU | Servers at 25G on 100G leaf ports |
| 400G QSFP-DD | 4 × 100G | Copper to four QSFP56 (50G per lane) plugs running 100GBASE-CR2, 1 m to 3 m: QDD-4ZQ100-CU. Or QDD-400G-DR4-S to four 100G DR optics over MPO to LC | Spine ports feeding 100G leaves |
| 400G QSFP-DD | 2 × 200G | Copper to two QSFP56 plugs running 200GBASE-CR4, 1 m to 3 m: QDD-2Q200-CU | Fast storage or GPU servers |
| 400G QSFP-DD | 8 × 50G | QDD-400G-SR8-S broken out over MPO-16 | Rare; needs MPO-16 cabling |
Parts from Cisco's 100G and 400G data sheets. Which ports can break out, and into what, depends on the switch model and its port groups.
The NX-OS commands for a 40G port split into four 10G interfaces. Other maps follow the same shape, with the keyword the model's guide lists.
! NX-OS: split a 40G port into four 10G interfaces
interface breakout module 1 port 49 map 10g-4x
! the port now appears as Ethernet1/49/1 to Ethernet1/49/4
show interface brief | include Eth1/49
! to rejoin the lanes into one port
no interface breakout module 1 port 49 map 10g-4xTry it: plan a breakout
Choose the parent port and how to split it. The planner lists the child interfaces, the cables that fit and the NX-OS command shape.
- Child interfaces
Ethernet1/49/1toEthernet1/49/4- Cables
- Copper breakout to four SFP28 plugs, 1 m to 5 m (
QSFP-4SFP25G-CU), or aQSFP-100G-SR4-Swith an MPO to LC breakout cord to four 25G SR optics. - FEC
- Set FEC on each leg to match the server, as for any 25G link.
- Command shape
interface breakout module 1 port 49 map 25g-4x- Check
- The keyword follows the speed and count pattern of
10g-4x. I confirm the exact keyword, and which ports can break out, in the guide for the switch model and release. Replacing the cable takes all 4 legs down.
FEC
FEC: the setting that keeps a link dark
At 25G and above, links depend on FEC (forward error correction): extra check bits that let the receiver repair errors without a resend. Both ends must use the same kind, or the link never comes up, however clean the light.
Swipe sideways to see the whole diagram.
Switch A uses RS-FEC and switch B uses FC-FEC. The signal arrives cleanly, but neither end can decode the other, so the link never comes up. Nothing on the optics points at the cause.
With the same FEC set on both ends, the link comes up. Between different vendors I set it by hand on both sides rather than trust each end to choose the same.
FEC fixes small errors on every link at these speeds, so corrected codewords count up all the time. That is normal. What matters is the next counter.
Uncorrected codewords mean whole frames are being lost. That points to a dirty connector, a damaged cable or a failing optic, not to the FEC setting.
| Type | Standard | Used for | Trade off |
|---|---|---|---|
| FC-FEC, also called Firecode | IEEE 802.3 clause 74 | Short 25G copper and some 25G optics | Light correction, very little added delay |
| RS-FEC (Reed Solomon) | Clause 108 at 25G, clause 91 at 100G | 25G SR optics, longer 25G copper, 100G over four 25G lanes | Stronger correction, a little more delay |
| Built into the standard | Part of 50G per lane and faster links | 100G DR and FR, and 400G optics | Usually nothing to set on the port |
What each Cisco 25G cable needs
| Cisco 25G part | FEC the host must run |
|---|---|
| 1 m, 1.5 m and 2 m copper | None |
| 2.5 m and 3 m copper | FC-FEC |
| 4 m and 5 m copper | RS-FEC |
25G SR optic, SFP-25G-SR-S | RS-FEC |
| 25G AOC, 1 m to 10 m | FC-FEC or RS-FEC |
From Cisco's 25G SFP28 data sheet. The 25G copper part numbers run from SFP-H25G-CU1M to SFP-H25G-CU5M.
Setting FEC on each platform.
! IOS XE on a Catalyst 9000: set 25G FEC the same on both ends
interface TwentyFiveGigE1/0/1
fec cl108
! cl108 is RS-FEC at 25G, cl74 is FC-FEC, auto lets the switch choose
show interfaces TwentyFiveGigE1/0/1 | include Fec
! NX-OS on a Nexus 9000: hard set IEEE RS-FEC, for example towards another vendor
interface Ethernet1/1
fec rs-ieeeAuto is not always enough
A Nexus 9000 picks its FEC from the transceiver it detects. Another vendor's switch may pick differently, so on links between vendors I set the same FEC by hand on both ends.
The optic decides too
Some optics need a particular FEC from the host to reach their full distance. The data sheet says which, and I copy that into the port template.
Watch the right counter
Corrected codewords are normal. Uncorrected codewords, rising CRC (cyclic redundancy check) errors or a link that flaps mean a physical fault.
Stacking
Stack cables, StackPower and StackWise Virtual links
Catalyst stacks join switches through their own stack cables, not through Ethernet ports. How a stack behaves when a cable or a member fails is on the Cisco IOS and Catalyst page. Here I cover how I cable stacks, StackPower and StackWise Virtual links (SVL).
Swipe sideways to see the whole diagram.
Stack port 1 of member 1 joins stack port 2 of member 2 with a short cable. With only this cable, the two switches already form a stack, but over a single path.
The same pattern continues: port 1 of each member to port 2 of the next. Short cables keep the back of the rack tidy.
A longer cable joins the last member back to the first. Traffic now has two ways round the ring, so one failed cable halves the bandwidth but splits nothing.
StackPower cables join the power supplies on the other side of the switches, so the members share their power budget and a failed supply does not take its switch down.
| Family | Stack technology | Cables | Notes |
|---|---|---|---|
| Catalyst 9300 | StackWise-480; StackWise-1T on the 9300X | STACK-T1-50CM, STACK-T1-1M, STACK-T1-3M | Up to 8 members |
| Catalyst 9300 power | StackPower | CAB-SPWR-30CM, CAB-SPWR-150CM | Shares power supplies across members |
| Catalyst 9200 | StackWise-160; StackWise-80 on the 9200L | STACK-T4-50CM, STACK-T4-1M, STACK-T4-3M | Needs the C9200-STACK-KIT stack module |
| Catalyst 3750 and 3750 X | StackWise and StackWise Plus | Older StackWise cables | End of life; covered on the switching page |
Each family uses its own cable type, and the types do not mix. Part numbers from Cisco ordering information.
StackWise Virtual: two switches over Ethernet links
Swipe sideways to see the whole diagram.
Cabled well. The StackWise Virtual links (SVL) run between different modules on each switch, and a separate dual active detection (DAD) link runs between them on another path. One switch is active, the other is a hot standby, and the access switch sees one partner.
Every SVL lost. Each switch now thinks the other has gone. The DAD link shows both are alive, so the standby enters recovery mode and shuts its ports, rather than both acting as the active switch with the same addresses.
Cabled on one module. Both SVL links leave module A. When that module fails, every SVL fails together. I put the links on different modules, or different port groups on fixed switches, and check the model's supported SVL ports.
The configuration, from Cisco's StackWise Virtual troubleshooting guide. Port names depend on the model.
! IOS XE: StackWise Virtual on a Catalyst 9500 pair, on both switches
stackwise-virtual
domain 1
! two or more SVL ports, spread across modules or port groups
interface HundredGigE1/0/1
stackwise-virtual link 1
interface HundredGigE1/0/2
stackwise-virtual link 1
! a separate port for dual active detection
interface HundredGigE1/0/3
stackwise-virtual dual-active-detection
! save and reload both switches, then check
show stackwise-virtual link
show stackwise-virtual dual-active-detectionSupported ports only
Not every port can carry SVL. On the Catalyst 9500-32C, for example, SVL and DAD run only on ports 1 to 16. I check the model before I plan the cabling.
Reload to apply
SVL and DAD configuration takes effect only after both switches reload, so I plan the first build and any SVL change as a full outage window.
Separate paths
SVL links on different modules, the DAD link on another, and none of them in the same cable tray if the two switches sit in different rows.
Nexus optics
Nexus optics, cables and the compatibility matrix
Nexus switches take the same industry form factors as everything else: SFP28 for 25G, QSFP28 for 100G and QSFP-DD (double density) for 400G. The figure follows a leaf from its servers, through BiDi (bidirectional) and SR4 uplinks, to a FEX (fabric extender). Cisco supports only the optics and cables it lists for each model and release, so every bill of materials starts from that list. NX-OS itself is on the Nexus and NX-OS page.
Swipe sideways to see the whole diagram.
Servers. Servers in the same rack use 25G passive copper, 1 m to 5 m. From 2.5 m the switch and the network card must run the FEC the cable needs, as the FEC section shows.
BiDi uplinks. A 100G BiDi (bidirectional) optic, QSFP-100G-SR1.2, sends and receives on two wavelengths over one duplex LC pair: 70 m on OM3, 100 m on OM4 and 150 m on OM5. It lets a pod move from 10G or 40G to 100G on the fibre already in the trays.
Parallel uplinks. QSFP-100G-SR4-S uses eight fibres in an MPO-12: 70 m on OM3, 100 m on OM4. It suits new builds with MPO trunks, and its four lanes can also break out to four 25G optics.
Fabric extenders. A Nexus 2000 FEX (fabric extender) reaches its parent switch with FET-10G optics, a FET (fabric extender transceiver): 100 m on OM3 or OM4, 25 m on OM2. They work only on that link. FEX is a legacy design I plan out, covered on the Nexus page.
| Part | Speed | Connector and fibre | Reach | Notes |
|---|---|---|---|---|
SFP-25G-SR-S | 25G | Duplex LC, multimode | 70 m OM3, 100 m OM4 | Needs RS-FEC on the host |
SFP-10/25G-CSR-S | 10G or 25G | Duplex LC, multimode | 300 m OM3, 400 m OM4 at 25G | Full reach needs RS-FEC |
QSFP-100G-SR4-S | 100G | MPO-12, multimode | 70 m OM3, 100 m OM4 | Parallel, breaks out to 4 × 25G |
QSFP-100G-SR1.2 | 100G BiDi | Duplex LC, multimode | 70 m OM3, 100 m OM4, 150 m OM5 | Reuses duplex fibre |
QSFP-40/100-SRBD | 40G or 100G BiDi | Duplex LC, multimode | 100G: 70 m OM3, 100 m OM4. 40G: 100 m OM3, 150 m OM4 | For a staged move from 40G |
QSFP-100G-DR-S, QSFP-100G-FR-S | 100G | Duplex LC, single mode | 500 m and 2 km | PAM4 (four level signalling) and FEC in the module |
QDD-400G-SR4.2-BD | 400G | MPO-12 UPC, multimode | 70 m OM3, 100 m OM4, 150 m OM5 | Breaks out to 100G SR1.2 |
QDD-400G-DR4-S | 400G | MPO-12 APC, single mode | 500 m | Breaks out to 4 × 100G DR |
FET-10G | 10G | Duplex LC, multimode | 100 m OM3 or OM4, 25 m OM2 | FEX to parent switch only |
From Cisco's 25G, 100G, 400G and 10G transceiver data sheets. OM3, OM4 and OM5 are multimode fibre grades; the fibre optics page explains them.
The compatibility matrix first
Cisco publishes which optics and cables each Nexus model supports, and from which NX-OS release, in its Transceiver Module Group (TMG) compatibility matrix. I check the model and the release before anything is ordered.
Third party optics
Third party optics can work, but they are a support decision as much as a technical one. I agree the policy with the operations team and Cisco TAC (Technical Assistance Center) before a single one goes in.
Read the optic
show interface ethernet1/1 transceiver details shows DOM (digital optical monitoring): temperature, voltage, laser bias and the transmit and receive power for each lane. I record a baseline for every link at handover.
Cable management
Cable management that keeps air and hands moving
A rack that is hard to work in gets worked on badly. Cables also sit in the air path of every switch and server, so tidy cabling is part of cooling, not decoration.
Swipe sideways to see the whole diagram.
Managed. Cold air enters at the front, crosses the switches and servers, and leaves at the rear. Cords run sideways into the vertical manager, one colour per network, so the exhaust is clear and any port can be reached without moving its neighbours.
Unmanaged. Cords hang straight down across the rear of the rack. Hot air is trapped behind them and the devices run hotter. Every change means lifting a curtain of cables, which is when the wrong one gets pulled.
- Respect the bend radius. No tighter than the data sheet allows. For fibre cords that is typically about ten times the cable diameter, and thick 400G copper is stiffer than most people expect.
- Cut to length. Each cord the right length, with a little slack in the manager, never a coil stuffed behind the rack.
- Colour by job. One colour for each network, written into the standard: for example production, storage, out of band and console each in their own colour.
- Power apart from data. Power on its own side of the rack and in its own tray, with the separation BS EN 50174-2 sets out.
- Keep the air path clear. Nothing across fan outlets, and blanking panels in every empty unit so hot air cannot loop back to the front.
- Support the weight. Hook and loop ties, never tight plastic ties on fibre, and heavy copper bundles supported so they do not hang off the ports.
- Overhead for data. I prefer overhead trays for data cabling: easy to see, easy to add to, and out of the cold air path under a raised floor.
Labels and schedules
Labels and schedules: the records the floor must match
A cable with no record is a cable nobody dares remove. I label both ends of every cable with the same name the records use, and treat the patching schedule as the source of truth: it changes first, and the switches prove it afterwards.
Swipe sideways to see the whole diagram.
A ticket asks for a new link, naming both ends and the speed. I check the ports, the panel capacity and the power and cooling before it is approved.
The patching schedule is updated before anything is touched. It allocates the cable ID and the panel ports, in NetBox or the DCIM (data centre infrastructure management) tool.
Labels are printed from the schedule, so the same ID appears at both ends of every cord, and nobody types a label by hand on the floor.
The installer patches from the work order in the change window, and photographs the panel before and after.
Both switches must see the neighbour the schedule expects, with
show lldp neighborsorshow cdp neighbors, and the light levels are recorded as a baseline.The change closes only when the floor, the schedule and the switches agree. Anything that does not match is fixed in the same window.
| Cable ID | From | To | Medium | Length | Status |
|---|---|---|---|---|---|
| C0412 | R12 · U42 · port 1 | R20 · panel 1 · port 1 | OM4 LC duplex | 3 m | Live |
| C0413 | R12 · U42 · port 2 | R20 · panel 1 · port 2 | OM4 LC duplex | 3 m | Live |
| T0021 | R12 · panel 1 | R20 · panel 1 | OS2 MPO-12 trunk, 24 fibres | 38 m | Tested |
| C0440 | R12 · U01 · port 48 | R12 · U42 · mgmt | Cat 6A | 2 m | Planned |
An example patching schedule. The format follows the TIA-606 idea of a unique identifier at both ends that the records also use.
Try it: make labels and schedule rows
Enter the two ends of a run of links. The tool builds the cable IDs, the label for each end and the schedule rows, ready to paste into a spreadsheet.
4 cables, LON1.C0412 to LON1.C0415.
| Cable ID | Label, near end | Label, far end |
|---|---|---|
| LON1.C0412 | C0412 · R12.U42.1 to R20.P1.1 | C0412 · R20.P1.1 to R12.U42.1 |
| LON1.C0413 | C0413 · R12.U42.2 to R20.P1.2 | C0413 · R20.P1.2 to R12.U42.2 |
| LON1.C0414 | C0414 · R12.U42.3 to R20.P1.3 | C0414 · R20.P1.3 to R12.U42.3 |
| LON1.C0415 | C0415 · R12.U42.4 to R20.P1.4 | C0415 · R20.P1.4 to R12.U42.4 |
Rows as comma separated values:
cable,from,to
LON1.C0412,R12.U42.1,R20.P1.1
LON1.C0413,R12.U42.2,R20.P1.2
LON1.C0414,R12.U42.3,R20.P1.3
LON1.C0415,R12.U42.4,R20.P1.4Same name everywhere
The ID on the cable is the ID in the schedule, the DCIM tool and the ticket. If the label and the records disagree, the records are fixed first.
NetBox as the model
NetBox models cables between interfaces and the front and rear ports of patch panels, so it can trace a link end to end. I keep it in step with the switches through automation, as on the network automation page.
Prove it from the switch
LLDP (Link Layer Discovery Protocol) or CDP (Cisco Discovery Protocol) neighbours show what is really plugged in. A nightly comparison against the schedule finds the cord moved without a ticket.
Install and test
Install and test: proving every link before handover
I prefer factory terminated trunks and cords: tested before they arrive, quick to install and clean. Whatever the medium, nothing is handed over until it has been tested against the design and recorded. Fibre testing in depth is on the fibre optics page, and copper certification on the copper cabling page.
| What | Acceptance test | Evidence |
|---|---|---|
| Fibre trunks and cords | Every connector inspected and cleaned to IEC 61300-3-35. Tier 1 loss test with an OLTS (optical loss test set) against the budget, and Tier 2 with an OTDR (optical time domain reflectometer) on long or critical links. | A test report for every link, in the handover pack |
| Copper, such as Cat 6A for out of band | Certified to the permanent link limits with a cable certifier. | Certifier reports, every link a pass |
| DAC, AEC and AOC | Link up at the designed speed and FEC, then a soak with no CRC errors and no uncorrected FEC codewords. | Interface counters, and a light level baseline for AOC |
| Stack cables and SVL | Every stack port up and the ring complete. SVL and DAD up, then a planned failover test. | Show outputs before and after each test |
| Labels and records | A sample of labels checked against the schedule, and every LLDP neighbour compared with it. | An audit sheet signed by both parties |
- Pre terminated by default. Factory terminated and tested trunks and cords are faster to install, cleaner and more consistent. Field termination is for odd lengths and repairs.
- Clean before every connection. Dirt on an end face is the most common optical fault. Inspect, clean and inspect again, even on new cords out of the bag.
- Test to the design, not just for light. A link that lights up can still be outside its loss budget. I test against the budget for the fastest optic the path is meant to carry.
- A handover pack, not a promise. As built rack elevations, port maps, the schedule export, every test report, the spares list and the cabling system warranty.
- Warranty paperwork. Many cabling vendors offer a long system warranty when a certified installer does the work. I make sure the certificate is in the pack before sign off.
Troubleshooting
When a link will not come up: a cabling fault finder
Most link faults in a data centre are physical, or a mismatch between the two ends. I work from the port outwards: what the switch says, what the optic or cable reports, then the path. My general method is on the troubleshooting page.
Swipe sideways to see the whole diagram.
No link and no light. Check the receive power on both ends. Nothing at all usually means swapped polarity on a duplex cord or an MPO path, a cord in the wrong panel port, or single mode and multimode mixed in one path. Trace it against the schedule.
Light is good, link stays down. Compare speed and FEC on both ends with show running-config interface ethernet1/1 on NX-OS or show interfaces on IOS XE. Between vendors, set the same FEC by hand on both.
The optic is refused. The port logs the transceiver as not supported, or goes err-disabled. Check the part against the TMG compatibility matrix for this model and release, and check the release notes before any upgrade.
Breakout legs dark. A breakout cable is plugged into a port still configured as one interface. Configure the breakout map for that port, then check the child interfaces with show interface brief.
Errors on a live link. CRC errors, or uncorrected FEC codewords that keep rising, point to the physical path. Clean and inspect both ends, check the bend radius and the cord, then retest the loss against the budget.
Low receive power. Read DOM with show interface ethernet1/1 transceiver details and compare with the handover baseline. A drop on one link means dirt or a tight bend. A slow fall on one optic means a failing laser.
Stack at half bandwidth. show switch stack-ports summary shows which stack port is down. Reseat the cable at both ends, then replace it. Until it is fixed, one more failure splits the stack.
Virtual links down. show stackwise-virtual link shows each SVL port. If all failed and DAD is cabled, one switch is in recovery mode with its ports shut. Restore an SVL link, and that switch reloads and rejoins.
The commands I start with
| Task | NX-OS | IOS XE |
|---|---|---|
| Port state and reason | show interface status | show interfaces status |
| Optic and light levels | show interface ethernet1/1 transceiver details | show interfaces transceiver detail |
| Error counters | show interface ethernet1/1 counters errors | show interfaces counters errors |
| FEC and speed | show running-config interface ethernet1/1 | show interfaces TwentyFiveGigE1/0/1 | include Fec |
| Breakouts | show running-config | include breakout | Not used on Catalyst access ports |
| What is really connected | show lldp neighbors, show cdp neighbors | show lldp neighbors, show cdp neighbors |
| Disabled ports | show interface status err-disabled | show interfaces status err-disabled |
| Stack and SVL | Not applicable | show switch stack-ports summary, show stackwise-virtual link |
Engagement
How I run a cabling project, from survey to a records audit
Cabling work usually arrives with a new hall, a pod refresh or a move to faster optics. I follow the same path each time: agree what the cabling must carry, survey what is really there, write the standard down, install and test every link, then keep the records and the floor in step.
Swipe sideways to see the whole diagram.
I start by agreeing which rooms, rows and racks are in scope, which speeds the cabling must carry now and in two refreshes, and who approves work on the floor. A RACI (responsible, accountable, consulted, informed) chart records who decides, including the colocation provider where there is one.
I turn the scope into requirements: ports and speeds per rack, the reach and loss budget of every link type, fibre counts with spares, and the rules for labels and records. Facilities sign off the tray capacity, power separation and airflow, because cabling touches all three.
I walk every room, row and tray with the installer, photograph and measure, and audit labels against the schedule rack by rack. Neighbour exports from every switch show what is really connected. Old test reports and fibre types decide which existing cabling can carry the new optics.
The high level design sets the layout, the media for each link type, the connector and polarity standard and the label scheme. The low level design draws every rack elevation and port map, schedules every trunk with its fibre count and length, and is signed off before anything is ordered.
Factory terminated trunks go in first, then every link is tested against its loss budget and recorded. Patching follows work orders printed from the schedule, in change windows. The build is accepted only when the switches' neighbours, the labels and the schedule agree.
Operations receive the handover pack: as built elevations, port maps, test reports and the warranty certificate. Patch requests follow one process, spares of every cord, cable and optic sit on site, and a nightly comparison of neighbours against the schedule catches undocumented moves.
Cabling outlives several generations of switches. Each year I review tray fill, panel ports and spare fibres, check which new optics the existing fibre can carry, retire old media as rows are rebuilt, and audit the records against the floor before any major migration.
Functional requirements
What it must do.
- Connect every server to its leaf switches inside the rack, and every leaf to the spines through structured fibre.
- Carry today's speeds and the next two generations without pulling new trunks.
- Let any link between racks be changed with a patch cord, not a new cable.
- Label both ends of every cable with an ID that the records also use.
- Keep power, data and the airflow path separate in every rack and tray.
- Report what is connected, from the switches, against the schedule.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Loss budget
- Every path within the budget of the fastest optic it must carry, with margin to spare (typical 0.5 dB or more).
- Spare capacity
- At least a quarter of the fibres in every trunk spare, and trays with room to add (typical).
- Installation quality
- Every link tested against the design, with a report for each, before handover.
- Records accuracy
- Schedule and floor agree for every link, proved by a nightly neighbour comparison.
- Change
- A standard patch request completed within an agreed window (typical: the next working day).
- Airflow
- No cables across fan outlets, and blanking panels in every empty unit.
- Standards
- Spaces and pathways to TIA-942 or EN 50600-2-4, with separation to BS EN 50174-2.
- Lifetime
- Structured cabling planned to outlast several switch generations (typical: ten years or more).
Numbers are typical starting points. Each organisation agrees its own.
Who I talk to
- Facilities and data centre managers, who own the rooms, trays and power
- Colocation providers, for cross connects and their site rules
- Network, server and storage teams, who own what plugs in
- Network operations, who know which records can be trusted
- Cabling installers, for surveys, test data and the warranty
How I find out
- A walk of every room, row and tray, with photos and measurements
- An audit of labels against the schedule, rack by rack
- Neighbour exports from every switch, compared with the schedule
- Existing test reports and fibre types checked against the new optics
- Port use and growth per rack from monitoring
Discovery tools
| Tool or source | What it tells me |
|---|---|
show lldp neighbors, show cdp neighbors | What is really connected to what |
show interface transceiver details | Optic types, light levels and DOM on every port |
| NetBox or a DCIM tool | The schedule, rack elevations and cable traces |
| Inspection scope and OLTS | End face condition and loss on existing links |
| OTDR | Where the loss sits along a long or suspect link |
| Cable certifier | Copper performance against its category |
| Photos and rack elevations | The physical truth the records must match |
Cabling faults show up as a dark link, a flapping link or errors that will not stop. I start at the port and work outwards, and I trust what the switch reports over what the label says.
My approach
- Scope. Which links, which racks, since when? What changed: a patch, a new optic, a software release or work in the tray?
- Port. Speed, FEC, breakout and error counters on both ends, before anything is unplugged.
- Optic. DOM on both ends against the handover baseline: transmit and receive power, laser bias and temperature.
- Path. Trace the link in the schedule, then on the floor. Inspect and clean every end face on the path.
- Test. Loss test the path against its budget, and use an OTDR where the loss is high or unexplained.
- Fix and record. One change at a time, counters cleared and watched, and the schedule updated with what was found.
Faults to check first
- Polarity swapped after a repatch
- A dirty end face on a new cord
- FEC set differently by two vendors
- A breakout cable on an unbroken port
- An optic missing from the compatibility list
- A cord moved without a ticket
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Light levels, end faces, polarity, bend radius, fibre type and the loss budget along the path. |
| 2 · Data link | Speed, FEC, LACP (Link Aggregation Control Protocol) bundles, CRC errors, err-disabled ports and neighbours against the schedule. |
| 3 · Network | Routing adjacencies and BFD (Bidirectional Forwarding Detection) sessions over the link, which flap first when a marginal link takes errors. |
| 4 · Transport | Retransmissions on servers behind a link with a small error rate. |
| 5 to 7 · Application | Which services use the link, and whether they have another path while it is fixed. |
My general method, with worked faults, is on the troubleshooting page.
Cabling stays right when every change follows one process and the spares are already on site. I agree both before handover.
Service model
The service desk takes patch requests, data centre operations patch and verify, and the installer handles warranty repairs, each with agreed response times.
Patch requests
One standard request with both ends, the speed and the cable ID, approved against capacity and closed only when verified.
Spares
Cords, copper cables, optics and a spare trunk of each common length kept on site and recorded like any other asset.
Monitoring
Alerts on light levels, CRC and FEC counters and link flaps, plus a nightly comparison of neighbours against the schedule.
Audits
A sample of racks audited each quarter, labels and records together.
Warranty
The installer's system warranty kept valid: approved parts and certified installers only on the structured cabling.
Cabling outlives several generations of switches, so I plan it for speeds that are not in the estate yet.
- Next speeds. Single mode and base 8 multimode trunks chosen so 400G and 800G optics fit the same cabling.
- Reuse. BiDi optics keep older duplex multimode in service at 100G and 400G while new trunks arrive.
- Retire in stages. OM1 and OM2, and OM3 where reach is short, retired as rows are rebuilt, not all at once.
- Capacity. Tray fill, panel ports and spare fibres reviewed each year, with growth turned into orders early.
- Records. A full records audit each year, and before any major migration.
- Standards. The cabling standard updated as connectors change, for example very small form factor fibre connectors.
Projects
Where I have used it
Cabling has been part of every data centre I have built or refreshed, from an empty shell for Network Rail to AI ready rack layouts today.
A data centre from an empty shell
Delivered the £25M Network Rail data centre build from an empty shell, covering structured cabling, building management systems and mechanical and electrical integration.
AI ready rack layouts
Designed AI ready data centre infrastructure for high density compute clusters, covering network, power, cooling and structured cabling, with the rack layouts and deployment documentation the build teams worked from.
Cabling pathways for high performance workloads
Prepared data centre infrastructure for high performance workloads: rack allocation, structured cabling pathways and deployment coordination.
A Tier 3 data centre build
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI (Application Centric Infrastructure) and Nexus.
Replacing end of life equipment
Audit, high and low level design and implementation of LAN (local area network), WAN (wide area network), SD-WAN (software defined WAN) and data centre infrastructure, replacing end of life equipment with minimal impact on global operations.
A 42U rack I cable myself
My lab is a 42U rack of Cisco routers and switches and HP servers, with labelled patch cables and a terminal server for console access. Kit is built and configured on the bench before it goes in the rack. See the rack.
Backed by training in fibre and copper structured cabling and in power and electrical systems, and by CCDP (Cisco Certified Design Professional) and CCNP (Cisco Certified Network Professional) Enterprise. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.