Load balancing

Getting every request to a server that can answer

I design application delivery on F5 BIG-IP and the public cloud load balancers. This page follows a request from the DNS (Domain Name System) answer to the server and back. On the way it covers methods, persistence, address translation, TLS (Transport Layer Security), security and failover, then shows how I find the fault when it breaks.

  • F5 BIG-IP LTM
  • BIG-IP DNS
  • Layer 4 and 7
  • SNAT and Automap
  • TLS offload
  • iRules
  • Advanced WAF
  • Cloud load balancers
A load balancer sharing requests Users send HTTPS requests to one virtual address on a load balancer. It ends TLS and shares the requests between two healthy web servers. A health monitor has marked the third server down, so it gets no traffic. POOL UsersHTTPS BIG-IPVIP :443 web 1up web 2up web 3monitor down TLS ends here then the method picks
Live view · requests shared across healthy servers
Overview

One address in front, many servers behind

A load balancer gives an application one stable address and spreads users across a pool of servers, removing any that fail their health checks. The examples use F5 BIG-IP, and the same ideas apply to cloud load balancers.

How a load balancer spreads requests and removes a failed server Users connect to a virtual server on a public address. The load balancer sends each new connection to a member of the web pool in turn. A health monitor requests a check page from every member every 5 seconds. When web2 stops answering correctly it is marked down and new connections go only to web1 and web3, until it passes its monitor again. POOL · WEB Usersthe internet Virtual server203.0.113.10:443 web110.1.1.11:80 web210.1.1.12:80 web2marked down web310.1.1.13:80 DNS points app.example.com at 203.0.113.10 Round robin across the healthy members GET /health every 5 seconds, expect 200 OK No good reply for 16 seconds: web2 marked down Passes its monitor again: back in the pool

Swipe sideways to see the whole diagram.

Step 1
  1. Users connect to a virtual server, often called the VIP (virtual IP address): an address and port, here 203.0.113.10 on 443, that DNS points the application name at. Nobody needs to know how many servers sit behind it.

  2. The virtual server hands each new connection to its pool. The pool picks a member, an IP address and port, using its load balancing method. Round robin is the default: one each in turn.

  3. A health monitor tests every member on a schedule. I use an HTTP (Hypertext Transfer Protocol) monitor that requests a real health page and expects a specific answer, not just an open port, so a broken application is caught too.

  4. When web2 stops giving the right answer for the timeout, 16 seconds with F5's HTTP defaults, it is marked down. New connections go only to web1 and web3. Existing connections to web2 stay until they close or time out, unless I set the pool's action on service down to reject, so clients reconnect to a healthy member at once.

  5. When web2 passes its monitor again it rejoins the pool. For a slow starting application I add a slow ramp time, so it is not flooded the moment it returns.

Why I put one in front

More capacity by adding servers, no outage when one fails, and zero downtime patching: drain a member, patch it, return it. The same box can also end TLS and apply a web application firewall.

The F5 building blocks

A virtual server (the address users hit) uses profiles (TCP, HTTP, TLS) and sends to a pool. The pool holds members, each a node address and a port, and monitors decide which members are up.

Health checks that mean something

A monitor should prove the application works, for example by requesting a health page that checks the database, and the receive string should match something only a healthy page returns.

Health monitors I use

MonitorWhat it provesNote
Gateway ICMP (Internet Control Message Protocol), or a node monitorThe host answersSays nothing about the application
TCP (Transmission Control Protocol)The port accepts a connectionA hung application still passes
HTTP or HTTPS (HTTP over TLS) with send and receive stringsThe page and what it depends on workThe timeout is three times the interval plus one: 5 and 16 seconds by default
Receive disable stringThe server asks to be drainedThe member stops taking new users but keeps persisted ones
External or inbandCustom checks, or failures in real trafficInband reacts to live errors with no extra probes

A pool can also require at least two of several monitors to pass before a member counts as up.

Layer 4 and 7

Reading packets, or reading requests

The first design choice is how deep the load balancer looks. At Layer 4 it only sees addresses and ports. At Layer 7 it understands the application request and can make far smarter decisions.

Layer 4 compared with Layer 7 load balancing A client connects through a load balancer to three pools. At Layer 4, one connection passes straight through and the load balancer only reads the IP address and port. At Layer 7, the load balancer ends the client's connection, reads the request, and opens a second connection to the right pool. API requests go to the API pool, images to the images pool, and everything else to the web pool. Clientbrowser Load balancer reads IP and portone connection reads host, URLand cookies Web pooleverything else API pool/api Images pool/images connection 1connection 2 Fast, simple, blind to content Two connections: it reads the request in the middle GET /api /api /images / Load balancer reads IP and portone connection reads host, URLand cookies

Swipe sideways to see the whole diagram.

Layer 4. The load balancer decides on the IP address and port alone, then forwards packets, rewriting addresses on the way. On F5 this is a performance virtual server with a FastL4 profile. It is fast and works for any TCP or UDP (User Datagram Protocol) application, but it cannot see URLs (web addresses), cookies or anything inside TLS.

Layer 7. The load balancer is a full proxy: it ends the client's connection, reads the HTTP request, and opens its own connection to a member. That lets it route by host name or URL, insert cookies, rewrite headers, offload TLS and run a web application firewall. It costs more processing per connection.

Animation
Methods

Choosing who gets the next request

The method decides which member gets each new connection. The right one depends on how similar the servers are, how long requests take, and whether the application keeps state.

Four load balancing methods Requests arrive at a pool of three servers. With round robin, each server gets one in turn. With ratio, server 1 has weight 3 and gets three times as many. With least connections, the slow server 2 gets fewer new requests because it is still busy. With a source address hash, each client always reaches the same server, which can leave one server quiet. Poolmethod Requestsarriving Server 1fast Server 1weight 3 Server 2fast Server 2weight 1 Server 2slow, long requests Server 3fast Server 3weight 1 One each, in turn Bigger servers get a bigger share New work goes where the fewest connections are Same source address, same server

Swipe sideways to see the whole diagram.

Round robin. The default: each new connection goes to the next member. Simple and even when members are the same size and requests take about the same time.

Ratio, or weighted. Each member has a weight, so a bigger server gets a bigger share. I use it during migrations to send a small slice of traffic to new servers first.

Least connections. New work goes to the member with the fewest open connections. It adapts when one server is slow or requests vary in length, which is why it is my usual choice for applications.

Source address hash. The client's address picks the member, so a client keeps reaching the same server without any state on the load balancer. Behind a large NAT (network address translation), a whole office looks like one client and lands on one server.

Animation

Try the methods with 30 requests

Server 2 can be made slow, so its connections last three times longer. Requests come from five offices, and the first office sits behind one NAT address and sends half of all traffic.

Server 1
10 requests · peak 2 open
Server 2
10 requests · peak 4 open
Server 3
10 requests · peak 2 open

Round robin shares requests evenly, but the slow server ends up holding the most open connections.

MethodHow it choosesWhere I use it
Round robinNext member in turnIdentical servers with short, similar requests
RatioIn proportion to a weight per memberMixed server sizes, or sending a slice of traffic to new servers
Least connectionsFewest open connectionsMost applications, especially with long or uneven requests
Fastest or observedQuickest response, or connections weighted over timeMembers whose performance varies
Source address hashA hash of the client addressSimple stickiness without state. On BIG-IP I get it from source address affinity or hash persistence, not a pool method
Global availabilityFirst healthy site in a listActive and standby data centres, in global DNS balancing
Persistence

Keeping a user on the same server

Some applications keep a user's session on the server that started it. Persistence, sometimes called stickiness, sends that user back to the same member. The best fix is a stateless application with shared session storage, but persistence is how most estates run today.

Keeping a user on the same server Two users reach three servers through a load balancer. Without persistence, user A's second request lands on a different server and the session is lost. With source address persistence, every request from user A's address goes to server 1. With cookie persistence, the load balancer adds a cookie to the first reply naming server 1, and the browser sends it back with every request. User Ashopping basket User Bsecond user Load balancerpersistence Server 1 Server 2 Server 3 Request two lands on another server: the basket is lost Same source address, same server, for a timeout A cookie names the server, so the browser carries it 1 2 1 2 cookie Load balancerpersistence

Swipe sideways to see the whole diagram.

No persistence. Each request is balanced on its own. That is fine for stateless applications, but if the server keeps the session in memory, such as a shopping basket, the second request lands elsewhere and the user is logged out.

Source address affinity. Requests from the same client address go to the same member until a timeout expires. It works for any protocol, but everyone behind one NAT address, such as a whole office, sticks to one server.

Cookie insert. On the first reply the load balancer adds a cookie that encodes the chosen member (on F5 it is named after the pool). The browser sends it back, so the user returns to the same server whatever their address. It needs HTTP, so with HTTPS the load balancer must decrypt.

Animation

Other methods

SSL (Secure Sockets Layer) session ID persistence for TLS 1.2 that is not decrypted, universal persistence on any value an iRule extracts, such as a header, and destination address affinity for caches.

Persistence beats the method

Once a user is persisted, the load balancing method is skipped for them. Long persistence timeouts can leave servers uneven, especially after one comes back from maintenance.

Draining for maintenance

Disabling a member stops new users but still accepts persisted ones. Forcing it offline lets existing connections finish but refuses every new one, even from persisted users. I disable, wait for sessions to end, then patch.

Return traffic

Every reply must come back the same way

A load balancer changes addresses in both directions, so the reply has to pass back through it. When it does not, the connection breaks in a way that looks like an application fault. This is the first thing I check in any new design.

Why return traffic must come back through the load balancer A client connects to a virtual address on a load balancer, which forwards to a server. In a two arm design the server's gateway is the load balancer, so replies come back through it. In a one arm design without source translation, the server replies straight to the client through its own router, and the client drops the reply because it came from the wrong address. With source translation, the load balancer replaces the client address with its own, so replies always return to it. Client10.20.1.7 Load balancerVIP 10.1.0.10 Server10.1.1.11 Routerserver's gateway C to VIP C to S S to C VIP to C C to VIP C to S S to C C to VIP LB to S S to LB VIP to C The server's gateway is the load balancer, so replies return Reply from 10.1.1.11 when the client expects 10.1.0.10: dropped Source becomes the load balancer: replies must come back to it Load balancerVIP 10.1.0.10

Swipe sideways to see the whole diagram.

Two arm, routed. The load balancer sits inline and its floating self IP is the servers' default gateway. It only changes the destination address, so servers see the real client address and replies naturally return through it.

One arm without SNAT. The load balancer hangs off one network and only changes the destination. The server sees the client's real address and replies through its own gateway, straight to the client. The client gets a reply from 10.1.1.11 for a connection it opened to 10.1.0.10, and drops it. This is the classic "works direct, fails through the VIP" fault.

One arm with SNAT. Source address translation replaces the client address with one of the load balancer's, so the server always replies to the load balancer. On F5, SNAT (source network address translation) Automap uses the self IP, the floating one in a pair. The server loses the client's address, so I pass it in the X-Forwarded-For header and log it there.

Animation

Watch the SNAT pool size

One source address gives about 64,000 ports to each server. Very busy virtual servers need a SNAT pool with several addresses, or port exhaustion appears as random connection failures.

Keep the client address

With SNAT, I insert X-Forwarded-For on HTTP and teach the application and its logs to read it. For non HTTP traffic, the routed two arm design keeps the real address.

Firewalls in the path

If a stateful firewall sits between the load balancer and the servers, asymmetric return paths are dropped there too. Return symmetry is a design rule for both.

Direct server return

The load balancer forwards only the requests. Servers hold the VIP on a loopback and reply straight to clients. It suits very heavy outbound traffic such as video, but the load balancer sees half the flow, so there is no Layer 7 routing, TLS offload or SNAT.

TLS

Where the encryption starts and stops

Most applications are HTTPS, so the load balancer either ends TLS, re encrypts it, or passes it through. The choice decides what the load balancer can see, and therefore what it can do.

Three ways to handle TLS at a load balancer A browser connects over TLS to a load balancer in front of a server. With TLS offload, the load balancer decrypts and sends plain HTTP to the server. With re encryption, it decrypts, inspects and opens a new TLS session to the server. With pass through, one TLS session runs from browser to server and the load balancer cannot see inside it. BrowserTLS 1.3 Load balancer decrypts, inspectsWAF and iRules cannot see insideLayer 4 only Serverweb pool client TLS plain HTTPnew TLS sessionone TLS session end to end TLS HTTP TLS TLS Certificate on the load balancer, servers do no crypto Inspected in the middle, encrypted on every hop Certificate on the servers, no inspection possible Load balancer decrypts, inspectsWAF and iRules cannot see insideLayer 4 only

Swipe sideways to see the whole diagram.

TLS offload. The certificate and key live on the load balancer, which decrypts and sends plain HTTP to the servers. It saves server CPU (central processing unit) and lets the load balancer route by URL, insert cookies and run a web application firewall. Only acceptable where the server network is trusted.

Re encryption, also called TLS bridging. The load balancer decrypts with a client side profile, inspects, then encrypts again with a server side profile. Every hop is encrypted, which most security policies now ask for. This is my default.

Pass through. The load balancer forwards TLS untouched at Layer 4. The servers hold the certificates and nothing in the middle can see the traffic, so no URL routing, cookie persistence or WAF (web application firewall). Persistence falls back to source address. SSL session ID persistence only works with TLS 1.2 and earlier, because TLS 1.3 does not reuse session IDs.

Animation

Certificates and ciphers

TLS 1.2 and 1.3 only, strong cipher suites, the full certificate chain on the profile, and renewal dates in monitoring. An expired certificate on a busy virtual server is an outage with a very public symptom.

SNI for many names

One virtual server can hold several certificates and pick one by the server name the browser asks for, called SNI (Server Name Indication). I keep a default certificate for clients that do not send it.

iRules, used sparingly

An iRule is a small script, in the Tcl language, that runs on events such as each HTTP request. It can route, rewrite or block. I prefer built in policies where they can do the job, because every iRule is code to maintain.

A small iRule that routes API (application programming interface) calls to their own pool

when HTTP_REQUEST {
    # send API calls to their own pool, everything else to the web pool
    if { [HTTP::uri] starts_with "/api/" } {
        pool pool_api
    } else {
        pool pool_web
    }
}
Global balancing

Choosing the site before the server

Local load balancing spreads users across servers in one site. Global server load balancing (GSLB) chooses the site, using DNS. LTM (Local Traffic Manager) balances servers, BIG-IP DNS balances sites.

Global load balancing with DNS A user asks a DNS resolver for an application name. The resolver asks BIG-IP DNS, which is authoritative for that name. BIG-IP DNS knows the health of the virtual servers in London and Manchester from the local load balancers and answers with London. The user connects to London. When London fails, the next answer is Manchester. Usernear London DNS resolverISP or company BIG-IP DNSauthoritative for app LONDON MANCHESTER LTM virtual server203.0.113.10 LTM virtual server198.51.100.10 Where is app.example.com? The resolver asks the authoritative server Both sites healthy: answer London The user connects to London London down: the next answer is Manchester London Manchester

Swipe sideways to see the whole diagram.

Step 1
  1. The user's device asks its DNS resolver, at the internet provider or the company, for app.example.com.

  2. The resolver asks BIG-IP DNS, formerly GTM (Global Traffic Manager), which is authoritative for that name, called a wide IP. It answers with an address, not a fixed record.

  3. BIG-IP DNS learns the health of each site's virtual servers from the local load balancers over iQuery. With both healthy, the method, here topology, sends a user near London to London.

  4. The user connects to the London virtual server, and the local load balancer spreads the work across its servers. The global layer only chooses the site.

  5. When London is down, the next DNS answer is Manchester. Users whose resolver still holds the old answer wait for the time to live (TTL) to expire, so I keep it short, around 30 seconds, and test that clients honour it.

MethodWhat it doesWhen I use it
Global availabilityAlways the first healthy site in an ordered listActive and standby data centres
Round robin or ratioShares answers across sitesActive and active sites of similar size
TopologyAnswers by where the user or resolver isSending users to their nearest region
Round trip timeAnswers with the site closest in network termsGlobal applications sensitive to latency
Web application firewall

Stopping attacks before the servers see them

Because the load balancer already decrypts and reads every request, it is a natural place for a web application firewall. On F5 this is BIG-IP Advanced WAF, which grew out of ASM (Application Security Manager).

A web application firewall on the load balancer A user, an attacker and a bot send requests to a WAF policy on the virtual server. The user's clean request passes to the server pool. The attacker's request carries SQL injection, matches a signature and is blocked. The bot sends requests too fast and cannot run JavaScript, so it is challenged and then blocked. Userbrowser Attackercrafted request Botscript WAF policyon the virtual server Server poolweb servers GET /basket HTTP/1.1Clean, and inside the policy: allowed GET /item?id=1' OR '1'='1SQL injection signature: blocked, support ID logged 40 requests a second, no JavaScript runFailed the browser challenge: blocked Clean request: inspected, then balanced to the pool Attack signature matched: blocked before the servers Bot defence: challenged, then blocked or rate limited challenge

Swipe sideways to see the whole diagram.

Clean traffic. The web application firewall (WAF) policy sits on the virtual server. It checks each request against attack signatures and against what the application should accept: its URLs, parameters, file types and lengths. A clean request passes to the pool as normal.

An attack. A parameter carries SQL (Structured Query Language) injection, which matches a signature, so the WAF blocks it and returns a page with a support ID. I start a new policy in transparent mode, which logs without blocking, then tune out false positives before I switch it to blocking.

A bot. A script sends requests far faster than a person and cannot run JavaScript. Bot defence challenges it, then blocks or rate limits it. Layer 7 denial of service protection works the same way, watching transactions per second and server latency.

Animation

The common attacks

The policy covers the web attacks in the OWASP (Open Worldwide Application Security Project) Top 10: injection (which now includes cross site scripting), broken access control and the rest, with signatures kept up to date.

Learn, then block

The policy builder learns the application from real traffic and suggests what to allow. I review each suggestion, stage new signatures before enforcing them, and log every exception with a reason.

APIs too

An API gets its own policy built from its OpenAPI specification, so only the documented paths, methods and fields are accepted. Anything else is blocked and logged.

Cloud WAF services

The same protection comes as a service: AWS (Amazon Web Services) WAF, Azure Web Application Firewall on Application Gateway or Front Door, Google Cloud Armor and F5 Distributed Cloud. I choose by where the application lives and who runs it.

High availability

Two units, one set of addresses

A load balancer in front of every application must not become the single point of failure. BIG-IP runs as a pair, or a larger cluster, and moves the addresses between units when one fails.

Failover between two load balancers Two BIG-IP units sit between an upstream router and a server pool. Unit A is active for traffic group 1, which holds the virtual addresses and the floating self IP, and carries the traffic. The units exchange heartbeats and synchronise their configuration. When A fails, B stops hearing heartbeats, makes the traffic group active, sends gratuitous ARP to the router and the servers, and traffic flows through B. Upstream routerARP cache heartbeat, UDP 1026 config sync and mirroring BIG-IP Aactive · traffic group 1failed BIG-IP Bstandbyactive · traffic group 1 Server poolgateway: floating IP VIP · floating self IP VIP · floating self IP One unit carries the traffic group Heartbeats and sync keep the pair aligned A fails: heartbeats stop, B waits out the timeout B goes active and sends gratuitous ARP Traffic flows through B GARP GARP

Swipe sideways to see the whole diagram.

Step 1
  1. Two units trust each other and form a sync failover device group. Traffic group 1 holds the virtual addresses and the floating self IP. It is active on unit A, so A carries the traffic.

  2. The units exchange network failover heartbeats on UDP port 1026, over more than one path. Config sync keeps their configuration identical, and connection mirroring can copy live connections to the standby.

  3. Unit A loses power. Unit B stops hearing heartbeats and, after a short timeout, decides its peer has gone. A failed VLAN (virtual local area network) check, a lost gateway or an HA (high availability) group score can trigger the same move.

  4. Unit B makes traffic group 1 active and sends gratuitous ARP (Address Resolution Protocol) for each address, so the router and the servers update their ARP caches. MAC (media access control) masquerade keeps the MAC address the same and removes the wait.

  5. Traffic now flows through unit B. Mirrored connections carry on, and the rest reconnect. When A returns it stays standby unless I fail back on purpose, which avoids a second interruption.

Active and active

Each traffic group can be active on a different unit, so both units carry live traffic. I size each unit to carry everything alone, otherwise a failover becomes an outage.

More than one heartbeat path

I send heartbeats over the HA VLAN and the management network. One failed link then cannot make both units go active, the split brain that takes everything down.

Failsafes and HA groups

VLAN and gateway failsafes catch failures that heartbeats cannot see. An HA group scores each unit on its trunks and pools, so the healthier unit takes the traffic.

Mirroring, with care

Connection mirroring keeps long sessions alive through a failover, at a cost in CPU and HA link traffic. I mirror only the virtual servers that need it, such as long running database connections.

Cloud

The same ideas in the public cloud

Every public cloud splits its load balancers the same way: Layer 7 for HTTP, Layer 4 for everything else, and DNS or anycast for global entry. This is how they line up when I move an application. More on each platform is on the cloud platforms page.

NeedAWSMicrosoft AzureGoogle Cloud
HTTP and HTTPS at Layer 7Application Load Balancer (ALB)Application GatewayApplication Load Balancer
TCP and UDP at Layer 4Network Load Balancer (NLB)Load BalancerNetwork Load Balancer, proxy or passthrough
Global HTTP entry with edge securityCloudFront with AWS WAFFront Doorglobal external Application Load Balancer with Cloud Armor
Global choice by DNSRoute 53 routing policiesTraffic ManagerCloud DNS routing policies
Global anycast entry for TCP and UDPGlobal AcceleratorLoad Balancer, global tierglobal external proxy Network Load Balancer
Firewalls inserted inlineGateway Load Balancer (GWLB)Gateway Load Balancerinternal passthrough Network Load Balancer as the next hop
Private, inside the networkinternal ALB or NLBinternal Load Balancer or Application Gatewayinternal Application or Network Load Balancer

Native first

For a new cloud application I start with the native load balancer. It scales on its own, is billed by use, and is built from the same code as the rest of the platform. I add BIG-IP where a requirement needs it.

BIG-IP in the cloud

BIG-IP Virtual Edition runs in all three clouds when an application needs iRules or the same WAF policy as on premises. Cloud networks do not honour gratuitous ARP, so failover moves addresses and routes through the cloud API instead.

Health checks need a way in

Cloud health checks come from known sources, such as Azure's 168.63.129.16 or Google Cloud's published ranges. If a security group or firewall rule blocks them, every member shows as down.

Containers

In Kubernetes, an ingress controller or the Gateway API does the Layer 7 work inside the cluster, with a cloud or BIG-IP load balancer in front of it.

Troubleshooting

Finding the fault one stage at a time

When an application behind a load balancer fails, the load balancer is blamed first. Often it is right, sometimes it is the application. I follow the connection through each stage in order and let the evidence decide. Choose a symptom to see where it fails and what I run. My general method is on the troubleshooting page.

A stage by stage load balancer fault finder Six checks in order: DNS and routing, the virtual server, the TLS handshake, the pool, the return path and persistence. For each symptom, test traffic flows as far as the stage that fails and stops there. A panel shows what I see and the command I run for that symptom. DNSname, route VIPlistening TLShandshake Poolmembers up Returnpath back Persistencesame member What I seeWhat I run Symptom: Some users reach the wrong site dig +short app.example.com returns 198.51.100.10, the site that is downdig @203.0.113.53 app.example.com Symptom: Nothing answers on the VIP Clients time out, and the virtual server counters stay at zerotmsh show ltm virtual vs_app Symptom: The TLS handshake fails Browser warning, or a handshake failure alert in the captureopenssl s_client -connect app.example.com:443 -servername app.example.com Symptom: Every connection is reset Availability : offline · Reason : The children pool member(s) are downtmsh show ltm pool pool_web members Symptom: One server does most of the work Server 1 holds 412 connections, servers 2 and 3 about 40 eachtmsh list ltm pool pool_web Symptom: Connections hang after the SYN Capture on the server VLAN: SYN to 10.1.1.11, no SYN ACK backtcpdump -ni 0.0:nnn host 10.1.1.11 Symptom: Users are logged out at random Requests from one user land on different memberstmsh show ltm persistence persist-records

Swipe sideways to see the whole diagram.

Some users reach the wrong site. The name resolves to a site that is down or far away. I ask BIG-IP DNS directly and compare with what the user's resolver returns. A resolver holding an old answer past its TTL, a topology record that places users in the wrong region, or a pool member marked up when it is not are the usual causes.

Nothing answers on the VIP. If the counters stay at zero, traffic is not reaching the box: a missing route upstream, a firewall in front, ARP turned off on the virtual address, or the virtual server enabled on the wrong VLAN. If the counters rise but clients get a reset, the virtual server is answering and the fault is further along.

The handshake fails. I check the certificate chain and dates the client sees, whether the client sent SNI and got the right certificate, and whether the client SSL profile allows its TLS version and ciphers. An expired certificate or a missing intermediate is the usual cause. With re encryption, I run the same test against a member, because the server side can fail too.

Every connection is reset. All pool members are marked down, so the virtual server has nowhere to send traffic. I run the monitor's request by hand from the BIG-IP to a member, for example with curl, and compare the reply with the receive string. A changed health page, a server firewall blocking the self IP, or no route back to the self IP are the usual causes.

One server does most of the work. Usually persistence: a long timeout, or source address affinity with a whole office behind one NAT address. Otherwise a ratio left over from a migration, or a member back from maintenance with no slow ramp time. I check the method, ratios and persistence records before blaming the servers.

Connections hang after the SYN. The member gets the SYN (synchronise, the first packet of a TCP connection), but its reply goes to its default gateway and around the load balancer, so the client drops it. In a one arm design this is the classic fault. I turn on SNAT with Automap, so replies come back to the self IP, or make the floating self IP the servers' gateway.

Users are logged out at random. Requests from one user land on different members. I check that a persistence profile is attached, that its timeout outlasts the application session, and for cookies, that the load balancer can see HTTP, which means TLS must end there.

Animation

My order of checks

  1. Scope it. One user, one site or everyone? One application or every application on the pair? The answer points at DNS, a virtual server, or the platform.
  2. Take the load balancer out. I test a member directly from the server VLAN, then through the VIP. If the direct test fails, the fault is in the application, not the load balancer.
  3. Capture both sides. A capture on the client side and the server side shows exactly where a connection stops, and which side reset it.
  4. Read the logs. The LTM log shows monitors marking members up and down, failovers and iRule errors, each with a time to match against the report.
  5. Change one thing. Then test again, and record the cause for problem management.

Commands by task

What I want to knowCommand
Is the virtual server up, and are its counters moving?tmsh show ltm virtual vs_app
Which members are up, and why not?tmsh show ltm pool pool_web members
What is the traffic doing on the wire?tcpdump -ni 0.0:nnn host 10.1.1.11, where :nnn adds the flow detail F5 support asks for
Who is persisted where?tmsh show ltm persistence persist-records
Which connections does one client have?tmsh show sys connection cs-client-addr 198.51.100.7
Which unit is active, and are they in sync?tmsh show sys failover, tmsh show cm sync-status
What happened, and when?tail -f /var/log/ltm
Is the certificate right?openssl s_client -connect app.example.com:443 -servername app.example.com
Engagement

How I run an application delivery change, from discovery to steady running

A load balancer sits in front of the applications the business notices first. I treat each change as a small programme with the application owners, because they know how their service really behaves.

How an engagement runs, from first conversation to lifecycle Seven stages from left to right: engage (stakeholders), requirements (FR and NFR), discovery (facts first), design (HLD and LLD), build and test (lab and pilot), support (handover), lifecycle (keep current). For each stage the panel shows who I work with and what comes out of it. After the lifecycle stage the loop returns to the start. Engagestakeholders RequirementsFR and NFR Discoveryfacts first DesignHLD and LLD Buildlab and pilot Supporthandover Lifecyclekeep current Step 1 · EngageAgree what each application needsWho I work withWhat comes outApplication ownersSecurity and riskService managementApplications in scope, by tierGoals: availability or securityConstraints and change windowsDecision log and RACI Step 2 · RequirementsHow each service must behaveWho I work withWhat comes outApplication ownersDevelopers and testersSecurity architectsVirtual servers and poolsPersistence and TLS rulesAvailability and failover targetsWAF policy scope Step 3 · DiscoveryEvery virtual server and what it doesWho I work withWhat comes outPlatform engineersApplication supportCertificate ownersVirtual server inventoryiRules and profiles in useCertificates and expiry datesTraffic and connection profile Step 4 · DesignStandard patterns, few exceptionsWho I work withWhat comes outDesign authoritySecurity architectsVendor or partner specialistsPlatform and HA designStandard virtual server patternsTLS and WAF standardsMigration and rollback plan Step 5 · Build and testTest failover before real usersWho I work withWhat comes outEngineers and partnerApplication testersChange advisory boardLab tested configurationFailover and monitor testsPilot sign off per applicationCutover plan by DNS change Step 6 · SupportMonitoring, runbooks and an ownerWho I work withWhat comes outOperations and service deskApplication supportSecurity operationsDashboards and alertsRunbooks per symptomCertificate renewal processKnown errors Step 7 · LifecycleSoftware, certificates and capacityWho I work withWhat comes outPlatform ownerSecurity and riskFinance and procurementSoftware release planCertificate calendarCapacity and licence reviewEnd of life register

Swipe sideways to see the whole diagram.

Step 1
  1. I start with why the change is happening: a platform at end of life, a move to cloud, a new security requirement or an outage. I agree who owns each application and who decides on risk (a RACI: responsible, accountable, consulted, informed).

  2. Functional requirements become virtual servers, pools, monitors, persistence and TLS settings. Non functional ones set availability, failover time, connections per second, certificate standards and how quickly a member must be removed when it fails.

  3. I export the configuration and build an inventory of every virtual server, pool, monitor, iRule and certificate, then match each one to an owner. Unowned virtual servers with no traffic are retired, not migrated. Statistics show the real peaks.

  4. The high level design (HLD) covers the platform, placement, HA and global balancing. The low level design (LLD) sets standard patterns per application type, so most services fit a template and only a few need an iRule.

  5. I build the patterns in a lab, then fail each unit, member and link on purpose. Applications move one at a time, usually by changing DNS to the new VIP with a short TTL, so rollback is a DNS change too.

  6. Operations get dashboards for pool health, connections and certificates, and runbooks for the faults on this page. Each application has a named owner who approves changes to its virtual server, policy and certificates.

  7. Load balancer software, signatures, certificates and licences all have their own cycles. I plan them together, upgrade the standby unit first, and review capacity each quarter against connections, throughput and TLS transactions.

Functional requirements

What it must do.

  • Publish each application on a virtual address and port, with its own pool and health monitor.
  • Keep a user on the same server when the application holds session state.
  • End TLS at the load balancer with approved certificates, ciphers and protocol versions.
  • Apply a WAF policy to every internet facing application.
  • Route users to the nearest healthy site, and away from a failed one.
  • Give each application owner the logs and statistics for their service.

Non functional requirements

How well it must do it, with a number wherever one can be agreed.

Availability
No single unit, link or member failure stops a service: a pair per site, members across hosts.
Recovery
Traffic moves to the standby unit within seconds, and a failed member leaves the pool within one monitor timeout.
Performance
Connections per second, throughput and TLS transactions sized with 30% headroom over the measured peak.
Security
TLS 1.2 or later, certificates renewed before 30 days remain, and WAF in blocking mode after tuning.
Global recovery
A failed site leaves the DNS answers within one monitor timeout, and clients follow within one TTL of about 30 seconds.
Operability
Health, connections and certificate expiry monitored, and every change logged.

These are typical starting targets for a load balanced service. Each application owner agrees the numbers for their own service, and each one is traced to a failover or load test.

Projects

Where I have designed and run it

Load balancing has come with the data centre, security and disaster recovery work in my career. F5 is named in my iSystems Integration architecture work. The other programmes below are data centre moves and refreshes where the applications had to keep running throughout.

iSystems Integration · three engagements, 2015 to 2019

Disaster recovery and security architecture

Cloud, software defined networking (SDN), disaster recovery and security architecture on Nexus, Viptela, Palo Alto, F5 and Aruba for the firm's enterprise clients. I returned for cloud and security architecture, then network transformation.

AVEVA · 2020 to 2021

99.99% through data centre moves

Safeguarded over £200M of transactional services through data centre transitions, holding 99.99% uptime throughout, while leading network and security transformation across every site.

BNY Mellon · 2018 to 2019

A refresh without disruption

Audit, design and implementation across LAN (local area network), WAN (wide area network), SD-WAN (software defined wide area network) and data centre infrastructure, replacing end of life equipment with minimal impact on global operations.

XIDA · 2021 to 2024

Cloud and data centre modernisation

Lead architect on network, cloud and data centre modernisation programmes, from discovery to operational handover, across several client environments.

F5 sits in my core security skills alongside Palo Alto, Check Point and Cisco Firepower. Backed by AWS Certified Solutions Architect Professional, CCNP (Cisco Certified Network Professional) Enterprise and the CCIE (Cisco Certified Internetwork Expert) Security written exam. See all credentials