Wire to wire
Time from a frame arriving at one port to a frame leaving another, measured on the wire with taps. It removes guesswork about where a host stamps its own time.
Low latency trading networks
Low latency trading networks, measured in microseconds
Trading firms win or lose on how quickly they see a price and act on it. This page follows a market data message and an order through a trading network: the distance, the switches, the servers, the clocks and the rules. It covers the hardware the industry uses, MiFID II (Markets in Financial Instruments Directive II) clock accuracy, and how I find faults that last only microseconds.
Basics
What latency means when prices move
A trading firm makes money when it sees a price change and acts on it before others do, or when it avoids being caught by a price that has already moved. Both depend on latency: the time a message takes to get somewhere. Step through a tick to trade path to see where the time goes.
Swipe sideways to see the whole diagram.
The venue's matching engine fills an order and publishes the change as a market data message. The clock starts when that message leaves the venue's port. Everything inside the venue is the venue's latency, not mine.
The message crosses a short cross connect and a Layer 1 switch to my server. About 50 metres of fibre takes about a quarter of a microsecond (µs), and a Layer 1 switch adds only a few nanoseconds.
The network interface card (NIC) takes the frame, and the feed handler decodes it and updates its copy of the order book. With kernel bypass this takes a microsecond or two in software (indicative).
The strategy reads the new book and decides whether to trade. This is usually the largest and least predictable part, so firms keep it small, or move it into hardware.
Every order passes pre trade checks before it leaves: price collars, maximum order value and size, and message limits. MiFID II requires them, so they must be fast rather than skipped.
The order leaves through the NIC and the order switch to the venue's gateway. The time from the tick arriving to the order leaving is the tick to trade time.
Added up, a software system in colocation spends a few microseconds from wire to wire. Systems built on an FPGA (field programmable gate array) are quoted in hundreds of nanoseconds. These ranges are indicative, not measurements from any firm.
The words people use
Tick to trade
Time from a market data update arriving to the order it triggers leaving. This is the figure trading teams track most closely.
Jitter and tails
How much latency varies. A fast average with a slow 99.9th percentile loses trades at the busiest moments, so I always look at the tail, not the mean.
Determinism
Doing the same work in the same time every time. A predictable 5 µs is often worth more than an average of 3 µs with spikes of 50.
Units and what fits in them
| Unit | In seconds | What fits in it (indicative) |
|---|---|---|
| Millisecond (ms) | One thousandth | A fibre round trip of about 100 km. A typical internet round trip inside the UK is a few of these. |
| Microsecond (µs) | One millionth | Light in fibre travels about 200 metres. A software trading system decides in a few of these. |
| Nanosecond (ns) | One billionth | Light in fibre travels about 20 centimetres. A Layer 1 switch adds 3 to 5 of these. |
Most enterprise networks are judged in milliseconds. A trading network is judged in microseconds, and its fastest parts in nanoseconds. The rest of this page follows a market data message and an order through each part of that path.
Latency budget
Where the time goes, part by part
Every microsecond in a trading path belongs to one of five parts: the distance the light travels, the time to put a frame on the wire, the switches, any queuing, and the servers at each end. I write a budget for each path before choosing equipment, because it shows which part is worth money.
| Part | How I work it out | Typical size |
|---|---|---|
| Propagation | Distance times about 5 µs per km in standard fibre, or about 3.34 µs per km through air | 0.25 µs for 50 m of cross connect; about 4.2 ms one way on a published London to Frankfurt fibre route |
| Serialisation | Frame size plus 20 bytes of preamble and gap, in bits, divided by the link speed | 67 ns for a 64 byte frame at 10 Gb/s; 1.2 µs for 1,500 bytes |
| Switching | The time a frame spends inside each switch, from the first bit in to the first bit out | A few ns at Layer 1; hundreds of ns cut through (indicative) |
| Queuing | Time spent waiting behind other frames. Zero on an idle port, large during a burst | Microseconds when feeds arrive together; unbounded when a port is oversubscribed |
| The servers | NIC, driver, operating system and application on both ends | Several µs through a normal kernel; around 1 µs with kernel bypass (indicative) |
Try it: build a latency budget
Choose the distance and medium, the switches in the path, the frame and link speed, and how each server handles packets. Switch and server figures are indicative planning values, not vendor quotes.
- Distance
- Serialisation
- Switches
- Servers
- Distance
- 250 ns
- Serialisation
- 221 ns
- Switches
- 5.0 ns
- Servers, both ends
- 2.00 µs
- One way
- 2.48 µs
- Round trip
- 4.95 µs
The servers take 81% of the time. In the same hall, the wire and a Layer 1 switch cost almost nothing. Faster servers help more than a faster switch here.
Worked example: two servers in the same data hall, 50 m of fibre, one Layer 1 switch, 256 byte frames at 10 Gb/s and kernel bypass on both servers. One way takes about 2.5 µs, and the servers account for 2 µs of it.
Two lessons come out of every budget I build. Across a city, distance dominates, so the route and the medium matter more than any box. Inside a data hall, the servers dominate, so the work is in the host and the application. Serialisation only matters on slower links and larger frames, which is one reason trading networks moved quickly to 25 Gb/s and beyond.
Switching
Store and forward, cut through and Layer 1
Inside one data hall, distance is small and the switches matter. A normal switch receives a whole frame before sending it on. A cut through switch starts sending once it has read the header. A Layer 1 switch does not read the frame at all. The basics of switching are on my switching page; this section is about the time each approach adds.
Swipe sideways to see the whole diagram.
The orange bars are the same frame in each lane. A store and forward switch holds every frame until the last bit and its FCS (frame check sequence) arrive, which for a 1,500 byte frame at 10 Gb/s is at least 1.2 µs. A cut through switch starts sending once it has read enough of the header to choose a port. A Layer 1 switch never reads the frame at all: it copies the electrical signal from one port to another.
| Kind | How it works | Latency (vendor or indicative) | Where I see it |
|---|---|---|---|
| Layer 1 switch | A crosspoint copies the signal from port to port, or to many ports at once. No MAC (media access control) table, no frame reading | 3 to 5 ns (Cisco Nexus 3550-F data sheet); about 5 ns (Arista 7130L) | Market data fan out, taps, patching that can be changed by software |
| FPGA mux | Logic in an FPGA merges several inputs onto one output, queuing only when frames collide | 39 to 48 ns (Cisco FastMux); 39 ns average (Arista MetaMux) | Orders from many servers to one venue session port |
| Cut through switch | Reads the header, picks a port and starts sending before the frame has fully arrived | Hundreds of ns on low latency switches (indicative) | Server access and aggregation where routing or filtering is needed |
| Store and forward switch | Receives the whole frame, checks the FCS, then forwards it | The frame time plus the pipeline, about 1 µs or more for large frames at 10 Gb/s (indicative) | General purpose and deep buffer switches, and any port with a speed change |
Swipe sideways to see the whole diagram.
A Layer 1 switch copies one market data cross connect to many servers, so every strategy sees the tick at the same moment. Going the other way, an FPGA multiplexer (mux) merges orders from many servers onto one venue port, adding tens of nanoseconds, and only queues when two orders collide. Cisco quotes 3 to 5 ns port to port and 39 to 48 ns for its fastest mux on the Nexus 3550-F; Arista quotes an average of 39 ns for MetaMux on the 7130.
Cut through is not always cut through
A cut through switch falls back to store and forward when a frame goes from a slower port to a faster one, or when the egress port is busy. I keep speeds matched along the fast path and check the switch counters that show which mode was used.
Bad frames travel further
A cut through switch cannot check the FCS before it starts sending, so a corrupt frame is forwarded and marked rather than dropped. CRC (cyclic redundancy check) errors then appear on several ports, and I trace them back to the first port that saw them.
Layer 1 has no brains
No VLANs (virtual local area networks), no routing, no access lists and no counters by flow. That is why it is fast. Filtering and routing live one hop away, on a switch that can do them in hardware.
Fan out copies faults too
A noisy feed port or a bad optic is copied to every server on that fan out. I keep a capture port on every fan out group so I can see exactly what each strategy received.
The Arista 7130 and the deep buffer 7280R families are compared on the Arista and Juniper page.
Servers
Inside the server, where most of the time goes
In colocation, the wire and the switches cost well under a microsecond. The server usually costs more: the network card, the operating system and the application. Trading firms remove as many of those steps as they can. Choose a path to compare them.
Swipe sideways to see the whole diagram.
The normal path. The NIC raises an interrupt, the driver hands the frame to the kernel's IP (Internet Protocol) and UDP (User Datagram Protocol) code, the data waits in a socket buffer, and a system call copies it into the application. Each step is reasonable on its own, but together they add several microseconds and plenty of variation.
Kernel bypass. A user space library maps the NIC's queues straight into the application, so packets arrive with no interrupts, system calls or copies on the fast path. AMD Solarflare adapters with Onload or TCPDirect are a common example. The application keeps the same sockets code with Onload.
FPGA. Logic on the network card decodes market data and can send an order without involving the server at all. The software sets prices, limits and strategy parameters and watches the result. It is the fastest option, and the hardest to change and test.
How I tune a trading host
| What I set | Why it matters |
|---|---|
| Pin the hot threads to their own CPU (central processing unit) cores, and keep the operating system off them | A thread that is never moved or interrupted keeps its cache warm and its timing steady |
| Poll the NIC rather than wait for interrupts | Busy polling burns a core, but removes the wake up delay from every packet |
| Keep the NIC, its memory and the hot cores on the same CPU socket | Crossing between sockets adds time and variation on every access (NUMA, non uniform memory access) |
| Turn off deep power saving states and frequency scaling on trading hosts | A core waking from sleep can add microseconds at exactly the wrong moment |
| Use hardware timestamps from the NIC | They show when a packet really arrived, not when software got round to reading it |
| Measure every change with the same replayed feed | Tuning that helps the average can hurt the tail, so I compare percentiles before and after |
The server side is mostly the trading firm's own engineering. My part as an infrastructure architect is to give it the right network cards, timing and switch ports, and the measurements that show whether a change helped. The physical servers and hypervisors behind this are on my servers and virtualisation page.
Market data
Market data, twice over
Venues publish prices and order book changes as streams of small UDP messages, sent by multicast so every subscriber gets them at once. Because UDP never resends a lost packet, venues send everything twice, on an A feed and a B feed, and the receiver keeps whichever copy arrives first.
Swipe sideways to see the whole diagram.
Every message carries a sequence number. The feed handler listens to both feeds and keeps whichever copy of each number arrives first, so a loss on one side costs nothing. Only when both copies are missing does it fall back to the venue's recovery service, which is slow by comparison, so I keep the two feeds on separate switches, optics, power and paths.
| Feature | What it is | Why it matters to the network |
|---|---|---|
| UDP multicast | The venue sends each message once, and the network copies it to every subscriber | Every subscriber sees the update at the same moment, without the venue sending a copy per firm |
| Channels or units | The instrument list is split across many multicast groups | A server joins only what it needs; I size ports and buffers for the busiest groups |
| Sequence numbers | Each channel numbers its messages | The feed handler can see a gap at once, and pick the first copy from A or B |
| A and B feeds | Two identical copies from separate infrastructure | One side can fail or drop a packet with no gap, if the two paths share nothing |
| Recovery | A TCP (Transmission Control Protocol) service replays missed messages or sends a snapshot of the book | A safety net only; LSEG (London Stock Exchange Group) offers replay and recovery services, Cboe a gap request proxy and a spin server |
Two separate halves
Feed A and feed B arrive on different cross connects, into different switches, on different power, and leave to different NIC ports. A shared optic panel or one switch in the middle quietly turns two feeds into one.
Multicast routing
Inside a firm, feeds are routed with PIM (Protocol Independent Multicast), sparse mode or source specific, and every hop checks RPF (reverse path forwarding). The basics are in the multicast section of my routing page.
Snooping and queriers
On Layer 2 segments, IGMP (Internet Group Management Protocol) snooping sends each group only to ports that asked for it. Snooping needs a querier on the VLAN, or joins slowly time out and a feed stops for no clear reason.
Burst headroom
Market data arrives in bursts at the open, the close and on news. A link that averages 20% can drop packets in a burst, so I size for the peak microsecond, not the average minute.
The building blocks on one side of a feed, with example addresses:
! NX-OS example, feed A side only. The B side mirrors it on separate switches.
feature pim
ip pim rp-address 10.10.0.1 group-list 239.10.0.0/16
interface Ethernet1/1
description Venue feed A cross connect
ip pim sparse-mode
vlan configuration 20
ip igmp snooping querier 10.20.0.1
Orders and risk
The order path, with its brakes on
Market data is only half the path. The other half carries orders to the venue and fills back, through risk checks that the regulations require and that no firm wants to live without. Step through an order, including one that is blocked and an emergency stop.
Swipe sideways to see the whole diagram.
The strategy builds an order: instrument, side, price and size. Nothing leaves the firm until the risk layer has seen it.
MiFID II RTS 6 (regulatory technical standard 6) lists the checks: price collars, maximum order value and volume, and maximum message rates, plus credit and position limits. They run on every order, so they are built for microseconds.
The order goes out on a session to the venue gateway. Sessions number every message in both directions, so a lost or repeated message is spotted at once. Venues offer FIX (Financial Information eXchange) and a faster native binary protocol.
The matching engine fills the order and sends an execution report back along the session to the strategy, which updates its position.
A drop copy session sends every fill again to the firm's risk and compliance systems, so positions are known even if a trading server fails.
If an order breaks a limit, the risk layer blocks it and tells the strategy. The block must be immediate; a check that is skipped when the market is busy is no check at all.
In an emergency the firm must be able to cancel every unexecuted order on every venue at once, and know which algorithm, trader or client sent each one. RTS 6 calls this kill functionality.
The algorithmic trading rules, in infrastructure terms
| RTS 6 article | What it asks for | What it means for the infrastructure |
|---|---|---|
| Articles 5 to 7: testing | Tested algorithms, conformance testing with each venue, and test environments kept apart from production | A separate test network and venue test connections, with feeds that can be replayed |
| Article 8: controlled deployment | Limits set before an algorithm goes live | Limits held as configuration, under change control |
| Article 9: annual self assessment | A yearly self assessment and validation report, approved by senior management | Evidence: diagrams, test results and incident records kept current |
| Article 12: kill functionality | Cancel any or all unexecuted orders at once, on all venues, and know who sent each one | A kill path that does not depend on the trading servers being healthy |
| Article 15: pre trade controls | Price collars, maximum order value and volume, maximum messages, and credit and market risk limits | Checks in the order path, fast enough that nobody wants to bypass them |
| Article 16: real time monitoring | Alerts within five seconds of the event, and staff who can act on them | Monitoring and alerting that work during the trading day, with named people on call |
The table summarises Commission Delegated Regulation (EU) 2017/589, which the UK keeps as part of its MiFID rules under the FCA (Financial Conduct Authority). The legal text is the authority; this is how it lands on the network and the servers.
Colocation
Colocation, and the map of London trading
The shortest wire is the one inside the venue's own building, so trading firms rent racks next to the matching engines. Each venue sits in a particular data centre, and a firm that trades on several needs fast links between them. Choose a view to move from one data hall to the whole of Europe.
Swipe sideways to see the whole diagram.
Colocation. A trading firm rents racks in the venue's own data hall, and the venue runs cross connects from its feed publishers and order gateways to the firm's switches. Inside one hall the cable is a few tens of metres, the shortest path anyone can buy, so the competition moves to the switches and servers.
Around London. LSEG runs its London venues from a data centre near the Docklands. ICE (Intercontinental Exchange) runs ICE Futures Europe from its European Liquidity Center in Basildon, Essex. Many FX (foreign exchange) venues and trading firms sit on Equinix's Slough campus, where LD4 is the best known building. A firm trading across all three needs fast links between them.
Into Europe. Deutsche Börse's colocation is with Equinix in Frankfurt. Euronext moved its core data centre and colocation from Basildon to Bergamo in Italy in 2022, so London firms now reach it over long haul links. Distance sets a floor no equipment can beat, which is why microwave links exist.
| Where | What runs there | What it means for my design |
|---|---|---|
| Near the Docklands, London | LSEG's London venues and its colocation service | Cross connects inside LSEG's hall for strategies on its venues |
| Basildon, Essex | ICE's European Liquidity Center, home of ICE Futures Europe | A separate site; firms trading futures against London equities link the two |
| Slough, west of London | Equinix's campus (LD4 and neighbours), with many FX venues and trading firms | The usual hub for foreign exchange and for firms spread across several venues |
| Frankfurt | Deutsche Börse colocation with Equinix | Long haul: about 640 km in a straight line from London |
| Bergamo, Italy | Euronext's core data centre since 2022, in Aruba's IT3 site | Further still; Euronext moved there from Basildon |
Venue locations change, as Euronext's move shows, so I confirm the current site, the colocation rules and the cross connect options with each venue before any design is fixed. Facility design itself, from power to cabling, is on my data centre facilities page.
Long haul
Between cities, the route is everything
Over hundreds of kilometres, no switch or server choice comes close to the effect of the path itself. Light is slower in glass than in air, and fibre rarely runs in a straight line. That is why trading firms built chains of microwave towers between financial centres, and why hollow core fibre is attracting interest.
Swipe sideways to see the whole diagram.
One way, London to Frankfurt. Light in standard fibre travels at about two thirds of its speed in air, and fibre follows roads and railways rather than a straight line. A published fibre round trip is 8.364 ms, about 4.2 ms each way. Microwave follows a near straight line through air, and hollow core fibre carries light through air inside the fibre, about 30% faster than standard fibre on the same route.
| Medium | Speed of light | Strengths | Limits |
|---|---|---|---|
| Standard single mode fibre | About 5 µs per km | Huge capacity, unaffected by weather, available everywhere | Slowest per km, and routes follow ducts, roads and railways |
| Hollow core fibre | About 3.3 to 3.4 µs per km | About 30% lower latency than standard fibre on the same route, and the same immunity to weather | New, with few routes; needs its own splicing and testing skills |
| Microwave | About 3.34 µs per km through air | Near straight paths between towers; the fastest long haul option | Low capacity, rain fade, licences and towers, and every hop is a site to maintain |
| Millimetre wave | About 3.34 µs per km through air | More capacity than microwave | Shorter hops and more sensitive to rain |
Send only what must be fast
Microwave capacity is small and expensive, so only the most time critical data rides on it, often a filtered or compressed feed. Everything else stays on fibre.
Plan for bad weather
Heavy rain can drop a microwave link. I design the failover to fibre as a normal event, tested often, with the strategy told which path it is on.
Measure the real route
A carrier's figure is a promise; I measure the one way latency myself with synchronised clocks at both ends, and again after every carrier change.
Keep it simple
Every box on a long haul path adds time and a place to fail. Amplifiers and regenerators are part of the budget, so I ask the carrier for the full list.
Enterprise WAN (wide area network) design, with MPLS (Multiprotocol Label Switching) and carrier services, is on my WAN, MPLS and VPN page. Trading links are different mainly in what they optimise: one way latency first, then everything else.
Precise time
Precise time, to the microsecond and below
A trading network needs one clock that every server agrees with, closely enough to order events microseconds apart and to prove it to a regulator. PTP (Precision Time Protocol), defined in IEEE (Institute of Electrical and Electronics Engineers) 1588, does this by measuring the path delay and correcting each clock against a grandmaster. The grandmaster takes its time from GNSS (Global Navigation Satellite System) or from a fibre feed traceable to UTC (Coordinated Universal Time).
Swipe sideways to see the whole diagram.
The grandmaster sends a Sync message and notes when it left: t1. The client notes when it arrived on its own clock: t2. Times here are in nanoseconds.
In two step mode, the exact departure time t1 follows in a Follow_Up message, read from the hardware as the packet left. One step clocks write t1 into the Sync message itself.
The client sends a Delay_Req and notes when it left: t3. The grandmaster notes when it arrived: t4.
The grandmaster returns t4 in a Delay_Resp. The client now holds all four times.
Mean path delay is ((t2 − t1) + (t4 − t3)) ÷ 2 = 1,100 ns. Offset is ((t2 − t1) − (t4 − t3)) ÷ 2 = 250 ns, so the client is 250 ns ahead and slews its clock back.
The maths assumes the path takes the same time each way. If one direction is 200 ns slower, the offset is wrong by 100 ns, and nothing in the protocol can see it. Different fibre lengths and switches that do not support PTP cause it.
Try it: work out delay and offset
Enter the four timestamps in nanoseconds, each on its own clock, then add some asymmetry: how much slower the path from the grandmaster is than the path back.
- Mean path delay
- 1,100 ns
- Offset as calculated
- +250 ns, the client is ahead
- True offset
- +250 ns
The client is 250 ns ahead. With a symmetric path, the calculated offset is the true offset, and the client corrects its clock by 250 ns.
| Method | Accuracy (typical) | Where it fits |
|---|---|---|
| NTP (Network Time Protocol) over the internet | Milliseconds; the NPL (National Physical Laboratory) quotes about 10 ms | Office systems. Not enough for high frequency trading under MiFID II |
| PTP with hardware timestamps and PTP aware switches | Well under a microsecond on a good design (indicative) | Trading floors and colocation, from a local grandmaster |
| NPLTime over fibre | Within 1 µs of UTC, certified by NPL | A feed traceable to the UK's national time scale, with no GNSS antenna, offered at Telehouse in the Docklands |
| White Rabbit | Sub nanosecond | Now the High Accuracy profile in IEEE 1588-2019; science facilities and some timing networks |
Boundary clocks
A boundary clock ends PTP on one side and acts as a grandmaster on the other, so each switch hop corrects its own delay. It keeps large networks stable, with the switch's own accuracy added at every hop.
Transparent clocks
A transparent clock passes PTP messages through and adds the time each spent inside it to a correction field. The client can then remove switch delay, even under load.
Choosing the grandmaster
The best master clock algorithm (BMCA) picks the grandmaster from announced quality and priority. I set priorities on purpose, so a failover lands on the clock I chose, not the first one that answers.
When GNSS goes wrong
Satellite time can be jammed or spoofed. Grandmasters hold time on an oscillator during an outage, called holdover, and I add a second, terrestrial source, such as a fibre time feed, with alarms when the two disagree.
The commands I use to set up and check PTP, with example interface names:
# Linux server with linuxptp: discipline the NIC clock, then the system clock from it
ptp4l -i ens1f0 -m
phc2sys -s ens1f0 -w -m
# check the offset from the grandmaster, in nanoseconds
pmc -u -b 0 'GET CURRENT_DATA_SET'
! Arista EOS: run the switch as a boundary clock
ptp mode boundary
interface Ethernet1
ptp enable
! Cisco NX-OS: the same idea
feature ptp
ptp source 10.0.0.1
interface Ethernet1/1
ptp
show ptp parent
MiFID II
MiFID II clocks, and how to prove them
Since January 2018, MiFID II has set how close business clocks must stay to UTC, and how finely trading events must be timestamped. The rule is RTS 25. It applies to trading venues and to the firms that trade on them, and in the UK it carries on as part of UK law, supervised by the FCA.
Swipe sideways to see the whole diagram.
RTS 25 asks firms to show an unbroken chain from UTC to every timestamp, to document how it works, to identify exactly where each timestamp is applied, and to review the whole system at least once a year. The pulses show time flowing down the chain; the tags show what I keep as evidence for each link.
RTS 25 accuracy tables
| Operators of trading venues: gateway to gateway latency | Maximum divergence from UTC | Timestamp granularity |
|---|---|---|
| More than 1 millisecond | 1 millisecond | 1 millisecond or better |
| 1 millisecond or less | 100 microseconds | 1 microsecond or better |
| Members or participants: type of activity | Maximum divergence from UTC | Timestamp granularity |
|---|---|---|
| High frequency algorithmic trading | 100 microseconds | 1 microsecond or better |
| Voice trading systems | 1 second | 1 second or better |
| Request for quote, where the response needs a human or the system does not allow algorithmic trading | 1 second | 1 second or better |
| Concluding negotiated transactions | 1 second | 1 second or better |
| Any other trading activity | 1 millisecond | 1 millisecond or better |
From the annex to Commission Delegated Regulation (EU) 2017/574, RTS 25. In the UK the same tables apply through the UK version, which the FCA Handbook calls MiFID RTS 25.
Which UTC
Clocks must follow UTC from a timing centre listed by the BIPM (International Bureau of Weights and Measures), or from a satellite system with any offset from UTC removed. In the UK, UTC(NPL) is the national version.
Prove it
Firms must document the design, functioning and specifications of the system that keeps them traceable to UTC, and be able to show it.
Know the stamp point
Firms must identify exactly where in the system each timestamp is applied, and show that the point stays the same.
Review it yearly
The traceability system is reviewed for compliance at least once a year. I keep the evidence from monitoring, so the review is a report rather than a project.
Try it: check a clock against RTS 25
Choose who you are and what you do, then enter the worst divergence from UTC you have measured and the granularity of your timestamps.
- Maximum divergence
- 100 µs
- Granularity needed
- 1 µs or better
Within the rule. High frequency algorithmic trading allows 100 µs of divergence from UTC and needs timestamps of 1 µs or finer. 40 µs and 1 ns timestamps meet both, as long as the traceability is documented and reviewed every year.
This is my reading of the published text for design purposes, not legal advice. Each firm's compliance team decides how the rules apply to its own activities.
Measuring
Measuring latency on the wire
Latency is only real once it is measured, and averages hide the problems that matter. I copy traffic with passive taps, timestamp every packet in hardware against PTP, and keep the captures, so any order can be traced microsecond by microsecond.
Swipe sideways to see the whole diagram.
A passive optical tap copies the light on a fibre without touching the traffic. An aggregation switch stamps each copy with a precise time, so the same packet seen at tap A and tap B gives the true latency between them. Arista quotes sub nanosecond timestamp accuracy for MetaWatch on its 7130 platform. Capture and analytics systems, such as CorvilNet, turn the stamps into latency per hop, feed gaps and burst reports.
| What I use | What it shows | What to watch for |
|---|---|---|
| Passive optical taps | An exact copy of every frame, with no effect on the traffic | Each tap takes some light, so I check the optical budget on long runs |
| SPAN (Switched Port Analyzer) mirror ports | Copies from inside a switch, with no extra hardware | Copies can be delayed or dropped under load, so they are fine for content and poor for timing |
| Aggregation with hardware timestamps | When each packet passed each point, to the nanosecond | The timestamping device needs PTP itself, or the stamps drift |
| Capture and analytics appliances | Latency per hop and per order, feed gaps, and market data bursts | Storage fills fast at the open, so I size it for the busiest day, not the average |
| Switch queue monitoring | Queue depth over time, which reveals microbursts | On Arista, LANZ (Latency Analyzer) with show queue-monitor length |
| Host counters | Drops in the NIC and the operating system | ethtool -S ens1f0 and netstat -su for receive buffer errors |
I measure wire to wire, with taps, rather than trusting timestamps written by the application. The application's own times are still useful, but only once I know how far they sit from the wire. The same measurements feed the RTS 25 evidence above.
Resilience
Resilient and secure, without adding latency
A trading network has to be fast, but an outage at the open costs more than any microsecond saved. The challenge is to add resilience and security in ways that do not sit in the fast path. These are the rules I apply.
- Two halves that share nothing. Feed A and feed B, and the primary and secondary order paths, use separate switches, optics, power feeds, NIC ports and fibre routes. I walk the racks to check, because diagrams hide shared patch panels.
- Fail over by design, not by accident. Spanning tree has no place in the fast path. I use routed links with simple, predictable failover, and I know how long each failure takes to recover.
- Filter in hardware at line rate. Access lists on the switches enforce segmentation without adding measurable delay. Firewalls guard the edges: management, the corporate network, the internet and remote access, never the market data path itself.
- Manage out of band. Device management, monitoring and logging travel on a separate network, so a busy feed never slows a fix and a fault never cuts off access. The pattern is on my data centre page.
- Change outside trading hours. Every change is tested on a replica with replayed market data first, scheduled outside the trading day, and has a rollback that has been tried. RTS 6 already expects separate test environments for algorithms.
- Size for the busiest microsecond. Capacity plans use the burst at the open, the close and on big news, measured with timestamps, not five minute averages.
- Know the venue's plan too. Venues run secondary data centres, and their feeds from a secondary site may arrive with extra latency. I agree with the trading team what we do when a venue fails over, and test it in the venue's own exercises.
Access control
TACACS+ (Terminal Access Controller Access Control System Plus) for device logins, named accounts and multi factor access to jump hosts. The general model is on my security page.
Segmentation
Separate VRFs (virtual routing and forwarding instances) or physical networks for market data, orders, management and corporate traffic, with no path between them that bypasses a control.
Records
Clock evidence, change records and captures are kept together, because an incident review needs all three to rebuild what happened.
Troubleshooting
Finding faults measured in microseconds
Faults in a trading network are often invisible to normal monitoring: a burst of a few microseconds, one feed of two, or a clock that agrees with itself but not with the world. Choose a symptom to see where it fails and what I run.
Swipe sideways to see the whole diagram.
Gaps on both feeds at once. When A and B lose the same messages, the cause is usually upstream of both: at the venue, or in something the two paths share. I check the venue's status notices, confirm what arrived with a capture at my edge, recover the gap from the venue's replay or snapshot service, and look for the hidden shared part.
Gaps on feed A only. Arbitration hides it, so nobody notices until B has a problem too. The cause is usually physical on the A side: a dirty or failing optic, a bad patch or a port taking errors. I check error counters and optical levels on every hop of the A path, clean or replace the optic, and alert on single feed gaps from now on.
Drops at the open, with low average use. Microbursts: several feeds peak in the same microseconds into one port, too briefly to show in averages. I confirm with queue monitoring and timestamped captures, then spread the feeds over more ports, move to faster links, or give the queue more buffer.
The feed stops after a failover. The new path has no multicast state: the PIM join did not follow, an RPF check fails on the new interface, or a VLAN lost its IGMP querier and snooping timed the group out. I check the multicast routes and snooping tables hop by hop, fix the RPF path or querier, and test failover with the feed running.
Drops on one server only. The network delivered the packets, but the server could not keep up: a receive buffer too small, a busy core, interrupts on the wrong CPU, or the NIC on the far CPU socket. I read the NIC and UDP counters, pin the feed handler to its own core, and size the buffers for the burst.
Latency up after a change. Often a speed mismatch: a link changed speed, so the cut through switch now stores and forwards. An extra hop, a new optic type or a changed route does the same. I compare taps before and after the change, then restore matched speeds or the shorter path.
CRC errors on many ports. A cut through switch forwards a corrupt frame before it can check it, so one bad source shows up everywhere downstream. I find the first port with input errors, which is the real fault, and fix that optic, cable or device.
The clock offset jumps. The grandmaster changed, or lost its GNSS lock and went into holdover. I check which clock is now the parent, why the best master clock algorithm chose it, and the grandmaster's alarms, then fix the priorities or the antenna and record the event for the RTS 25 evidence.
Clocks agree with PTP, but not with the taps. Path asymmetry: one direction of a link is longer or slower, or a switch in the path does not support PTP. PTP cannot see it, so the offset looks perfect. I compare against an independent reference, find the asymmetric hop, and correct it or make every switch a boundary or transparent clock.
The tools I reach for
| Question | Command or tool | What it tells me |
|---|---|---|
| Is a port taking errors? | show interfaces counters errors, and optical levels with show interfaces transceiver | CRC and input errors, and whether an optic is weak |
| Is a queue bursting? | show queue-monitor length (Arista LANZ), timestamped captures | Queue depth over threshold, to the microsecond |
| Is multicast state right? | show ip mroute 239.10.1.1, show ip igmp snooping groups, show ip pim neighbor | Whether the tree reaches the receiver, and why not |
| Is the server dropping? | ethtool -S ens1f0, netstat -su | NIC drops and UDP receive buffer errors |
| Is the clock right? | pmc -u -b 0 'GET CURRENT_DATA_SET', show ptp parent | Offset from the grandmaster, and which clock is the parent |
| Where did the time go? | Captures from taps at each point, compared by timestamp | Latency per hop for the same packet |
I fix one thing at a time and replay the same feed after each change, because many of these faults only appear at full speed. My general method is on the troubleshooting page, and any acronym on this page is in the glossary.
Engagement
How I approach a trading network, from first brief to steady state
Trading network work starts with a question from the trading team: which venues, which strategies, and how fast must they be. I follow the same path I use on any regulated infrastructure. I agree what must work and how fast, measure what exists, design two independent halves, prove them with replayed market data, and keep the evidence a regulator expects.
Swipe sideways to see the whole diagram.
I start with the trading team: which venues and products, which strategies, and what each microsecond is worth to them. Compliance sets the regulatory scope, including RTS 6 and RTS 25. I record who decides and who is consulted in a RACI (responsible, accountable, consulted, informed) chart, so every trade off has an owner.
I turn the brief into requirements with numbers: wire to wire latency and its tail for each path, clock divergence and timestamp granularity, recovery times for each failure, and the trading hours when nothing may change. Each number has an owner who signs it off, and a way to measure it.
Before designing anything I measure: taps and timestamped captures on the existing paths, queue monitoring at the open and the close, and PTP offsets on every host. I also walk the racks to find anything A and B share. I ask each venue and carrier for their current options and timelines.
The high level design sets the latency budget, the colocation and long haul choices, and the time architecture. The low level design fixes every port, optic, cross connect, multicast group and PTP role, with a test plan whose pass marks were agreed before the build started.
I build on a replica first and replay recorded market data at full speed, including the busiest day available. I time every failover and check clock accuracy against an independent reference. Only then is the cut over scheduled outside trading hours, with a rollback that has been tried.
Before handover, latency per hop, feed gaps, queue bursts and clock offsets are all in monitoring with thresholds. Support teams get runbooks for the common faults and the venue and carrier contacts. Hypercare, the period of extra support after go live, ends only when the agreed criteria are met.
Venues change protocols, sites and colocation rules, so I track their release calendars and test each change in their test environments. The RTS 25 traceability review happens at least yearly, and the algorithm self assessment under RTS 6 needs infrastructure evidence too. Hardware and firmware follow a planned refresh.
Functional requirements
What it must do.
- Deliver market data from each venue to every subscribed server on two independent feeds.
- Carry orders to each venue with pre trade risk checks in the path, and drop copies to risk and compliance.
- Keep every business clock traceable to UTC, with timestamps applied at documented points.
- Provide kill functionality that still works when trading servers fail.
- Capture and timestamp traffic at agreed points for latency analysis and records.
- Keep management, monitoring and corporate traffic off the trading path.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Latency
- Wire to wire within the agreed budget for each path, judged on the 99th and 99.9th percentiles as well as the median (typical).
- Jitter
- Tail latency within a small multiple of the median at peak load (typical).
- Clock accuracy
- Within 100 µs of UTC for high frequency trading under RTS 25, with internal targets often 1 µs or better (typical).
- Resilience
- No single failure stops a feed or the order path; feed A or B can fail with no gap.
- Failover
- Order path failover measured and agreed, in seconds or less (typical).
- Capacity
- Sized for the busiest microsecond burst seen, plus headroom.
- Change
- No changes in trading hours except emergencies, all tested on a replica first.
- Evidence
- Clock, change and capture records kept for review and incident analysis.
Numbers are typical starting points. RTS 25 sets the legal minimum for clocks; each firm agrees its own targets with its trading and compliance teams.
Who I talk to
- Trading and quant teams, who know which microseconds are worth paying for
- Compliance and risk, who own the RTS 6 and RTS 25 obligations
- Venue onboarding and technical teams, for colocation, cross connects and test environments
- Carriers and long haul providers, for routes, latency and diversity
- Server and application engineers, who own the hosts and the feed handlers
How I find out
- Taps and timestamped captures at each point of the existing paths
- Queue monitoring through the open, the close and a busy news day
- PTP offsets and grandmaster status on every host and switch
- A physical walk of racks, panels and power to find shared risks
- A review of each venue's current sites, protocols and release calendar
Discovery tools
| Tool or source | What it tells me |
|---|---|
| Optical taps and an aggregation switch with hardware timestamps | Latency per hop, to the nanosecond |
| Capture and analytics appliances, such as CorvilNet | Latency per order, feed gaps and bursts |
show queue-monitor length (Arista LANZ) | Microbursts on each port |
linuxptp pmc and switch show ptp commands | Clock offsets, and which grandmaster each clock follows |
ethtool -S and netstat -su on hosts | Drops in the NIC and the UDP stack |
| Feed handler gap and arbitration statistics | Losses on each feed, before arbitration hides them |
| Venue test environments and replay tools | Conformance tests and full speed replays before changes |
Most trading network faults are physical, multicast or timing problems hidden behind a fast average, so I check those early, with timestamps rather than averages.
My approach
- Scope. Which venue, feed, strategy or server, which direction, and since when? What changed: a release, a carrier, a venue or a clock?
- Feeds. Compare gaps on feed A and feed B separately, before arbitration hides one side.
- Path. Check error counters, optics and speeds on every hop, and whether a cut through switch is now storing and forwarding.
- Bursts. Read queue monitoring and timestamped captures at the time of the fault, not five minute averages.
- Time. Check PTP offsets, the parent clock and GNSS alarms, and compare them against the taps.
- Fix and prove. Change one thing outside trading hours, replay the same feed, and record the cause for problem management.
Faults to check first
- A failing optic on feed A, hidden by arbitration
- Microbursts at the open on ports with low average use
- A multicast join or IGMP querier missing after failover
- UDP receive buffer drops on one busy server
- A speed change that turned cut through into store and forward
- PTP asymmetry that keeps offsets small but wrong
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Optic levels, CRC errors and speeds, and whether A and B share any panel, cable or power. |
| 2 · Data link | IGMP snooping and queriers, VLANs, and cut through or store and forward behaviour per port. |
| 3 · Network | Multicast routes, RPF and PIM neighbours with show ip mroute, and unicast routes for the order path. |
| 4 · Transport | UDP receive buffer errors with netstat -su, and order session sequence numbers. |
| 5 to 7 · Application | Feed handler gaps and arbitration, order acknowledgements, and application timestamps against the taps. |
My general method, with worked faults, is on the troubleshooting page.
A trading network needs support that works in microseconds and in trading hours. I agree who watches what, and what they do when it changes.
Service model
Who owns each part: the trading support desk, the network operations centre (NOC), server engineering, each venue and each carrier, with escalation times written down.
Monitoring
Latency per hop, feed gaps on A and B, queue bursts and clock offsets, with alerts that reach a person within seconds during trading hours.
Clock evidence
Grandmaster status, offsets and alarms stored as the RTS 25 record, not just shown on a screen.
Runbooks
Step by step guides for feed gaps, bursts, failover, clock jumps and venue outages, each starting with the counters to read.
Change
Changes outside trading hours, tested on a replica with replayed data, with before and after latency from the taps.
Reviews
A monthly look at latency tails and near misses, and a yearly traceability review with compliance.
Trading infrastructure changes whenever the venues do, so I plan its upkeep from the start.
- Venues. Protocol versions, colocation rules and even sites change. I follow each venue's release calendar and test in its environment.
- Regulation. The RTS 25 review runs at least yearly, and RTS 6 self assessments draw on infrastructure evidence.
- Hardware. NICs, switches and timing equipment follow a planned refresh, with firmware tested against replayed feeds.
- Routes. Carrier routes and microwave paths are measured again after every change, and once a year.
- Capacity. Market data rates grow, so I review burst headroom each quarter.
- Standards. I track PTP profiles, hollow core fibre and new NIC options, and adopt them when the evidence supports it.
Projects
Related work behind this page
These are the projects closest to trading networks. They carry over the skills this page draws on: change and compliance in a MiFID II regulated bank, data centre transitions that kept transactional services running, planning for low latency workloads, and provider grade data centres.
A refresh in a MiFID II regulated bank Related
Delivered the EMEA (Europe, the Middle East and Africa) network refresh: audit, high and low level design, and implementation. It covered LAN (local area network), WAN, SD-WAN (software defined WAN) and data centre infrastructure, replacing end of life equipment with minimal impact on global operations. I kept the estate compliant with MiFID II throughout, with no service outages, working with a blended team of 5 engineers and 4 vendors.
Transactional services at 99.99% uptime Related
Safeguarded over £200M of transactional services through data centre transitions at 99.99% uptime, while leading 7 engineers and architects and coordinating 5 vendors.
Planning for low latency workloads Related
Shaped SD-WAN, SD-Access and Catalyst Center infrastructure planning so that backbone, data centre and edge environments could carry high throughput, low latency workloads.
A Tier 3 data centre for 300 customer sites Related
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI (Application Centric Infrastructure) and Nexus. I managed the provider IP space and AS (autonomous system) numbers with RIPE NCC, the regional internet registry for Europe.
Nexus and multi cloud transformation Related
Led network and security transformation from discovery to provisioning across Cisco Nexus, Catalyst SD-WAN, SD-Access, AWS (Amazon Web Services), Azure and Google Cloud environments. The platform is on my Nexus and NX-OS page.
Failover I can watch Related
My lab runs an MPLS core between five sites. I fail links on purpose to watch how routing recovers, the same habit I bring to the A and B halves of any network that must not stop. See the topology.
Backed by CCNP (Cisco Certified Network Professional) Enterprise, CCNP Routing and Switching and the CCIE (Cisco Certified Internetwork Expert) Routing and Switching written exam, which covers the IP multicast this page relies on. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.