Servers and virtualisation, from the motherboard up

The board. Each processor (CPU) has its own memory channels and its own PCIe (Peripheral Component Interconnect Express) lanes for drives and network interface cards (NICs). The two processors talk over a fast interconnect: UPI (Ultra Path Interconnect) on Intel, Infinity Fabric on AMD.

NUMA (non uniform memory access). A core can read the other socket's memory, but the request crosses the interconnect, so it is slower. I keep each virtual machine inside one NUMA node where I can, and I state the largest VM size per node in the server standard.

PCIe placement. A network card or GPU (graphics processing unit) on CPU 1 serving a workload on CPU 0 sends every packet across the interconnect. I check the slot to socket map and put fast cards on the socket that runs the workload.

Out of band management. The BMC (baseboard management controller), called iLO (Integrated Lights-Out) on HPE servers and iDRAC (integrated Dell Remote Access Controller) on Dell, gives remote power, console, sensors and firmware updates even when the operating system is down. It sits only on the out of band network, never on a user network, with central accounts, current firmware and IPMI (Intelligent Platform Management Interface) over the LAN (local area network) turned off.

DAS (direct attached storage). Disks inside the server or in a shelf cabled to it. It is fast and cheap, but only that server can use it, so a host failure takes the data offline with it. vSAN and other hyperconverged systems pool the local disks of many hosts to get round this.

NAS (network attached storage). A filer shares files over NFS (Network File System) or SMB (Server Message Block). Many clients can use the same share, and the network carries it like any other traffic, so I give it its own VLAN (virtual LAN) and quality of service.

Fibre Channel SAN (storage area network). Block storage over two separate fabrics, A and B, which never connect. Zoning on the switches controls which server sees which array port, and LUN (logical unit number) masking on the array decides which volumes it sees. Multipathing on the host uses both paths.

iSCSI (Internet Small Computer Systems Interface). The same block storage, carried over IP (Internet Protocol) on Ethernet. I use two separate VLANs or switches, jumbo frames end to end, port binding on the hosts and no routing in the storage path. Every hop must carry the same MTU (maximum transmission unit).

1. Steady state

   Four hosts form one cluster. The HA agent on each host exchanges heartbeats over the management network, and admission control keeps a quarter of the cluster free as a failover reserve. With four hosts, that is the same as N+1: room to lose one host.

2. Admission control

   Someone tries to power on another VM. Admission control refuses, because it would eat into the capacity needed to restart VMs after a failure. I never turn admission control off to get round this. The answer is more capacity.

3. A host fails

   Host 3 stops. The others miss its network heartbeats, then check its datastore heartbeat to tell a dead host from one that is only cut off from the network. An isolated host follows its isolation response instead.

4. HA restarts the VMs

   HA restarts host 3's VMs on the other hosts, in restart priority order, from the shared datastores. This is a restart, not a live move: each VM boots again, so services see a short outage.

5. DRS rebalances

   When host 3 returns, DRS uses vMotion to move VMs back and even out the load, and the reserve is free again. Anti affinity rules keep pairs such as two domain controllers on different hosts.

1. Compatibility checks

   vCenter checks that host B can run the VM: compatible processors or a shared EVC (Enhanced vMotion Compatibility) baseline, the same port group and VLAN, access to the VM's storage, and enough free resources.

2. Copy the memory

   The VM keeps running on host A while its memory pages copy to host B over the vMotion VMkernel network. This is the pre copy phase, and it is why vMotion needs a fast, dedicated link.

3. Send the changes

   Pages the VM changed during the copy are sent again, in smaller and smaller rounds, until what is left can move almost instantly.

4. Switch over

   The VM pauses for a moment, usually well under a second, while the last pages and device state move. It then resumes on host B, and TCP (Transmission Control Protocol) sessions carry on.

5. Tell the network

   Host B sends a RARP (Reverse Address Resolution Protocol) frame for the VM's MAC address, so the switches learn its new port at once. An occasional MAC move in the switch log after a vMotion is normal.

6. Storage vMotion

   Storage vMotion copies the disk files to another datastore while the VM runs, and a mirror driver writes every change to both copies until the switch. Combined with vMotion, a VM can move between hosts that share no storage.

Logical structure. A forest is the security boundary and holds one schema. Domains inside it trust each other both ways. Organisational units (OUs) group users, computers and servers so that Group Policy and admin rights can be delegated. Five FSMO (flexible single master operation) roles exist: two per forest, schema master and domain naming master, and three per domain, RID (relative identifier) master, PDC (primary domain controller) emulator and infrastructure master. Most organisations need one domain, not several.

Sites and replication. Sites map to IP subnets, so clients find a nearby domain controller. Inside a site, changes replicate within seconds. Between sites they travel over site links, chosen by cost and sent on a schedule, every 180 minutes by default, which I usually shorten. A read only domain controller (RODC) suits sites with weak physical security: it receives changes but never sends them.

1. Find a domain controller

   The laptop asks DNS for the SRV (service locator) record _ldap._tcp.dc._msdcs.example.com, which lists domain controllers, preferring ones in its own site. If DNS points somewhere else, nothing that follows works.

2. Get a ticket granting ticket

   The laptop sends an authentication request with a timestamp encrypted by a key made from the user's password. The KDC (key distribution centre) on the domain controller checks it and returns a TGT (ticket granting ticket), valid for 10 hours by default.

3. Ask for a service ticket

   To open a share, the laptop presents its TGT and asks for a ticket for the file server's service principal name, cifs/fs01.example.com. The KDC returns a service ticket encrypted with the file server's own key.

4. Present it to the server

   The laptop hands the service ticket to the file server, which decrypts it with its own key. The domain controller is not asked again, which is why Kerberos scales so well.

5. Time keeps it safe

   Tickets carry timestamps, and a clock more than 5 minutes out makes logons fail. Members take time from domain controllers, and the PDC emulator in the forest root domain takes it from a reliable external source over NTP (Network Time Protocol).

A VM has no network. Inside the guest, ipconfig /all or ip -br addr shows whether it has an address; 169.254 means DHCP never answered. I compare the port group VLAN with the switch, using esxcli network vswitch standard portgroup list or the distributed switch in vCenter, then show interfaces trunk on the switch to confirm the VLAN is allowed on both host ports.

vMotion fails or times out. I test the vMotion network with vmkping -I vmk1 -d -s 8972 to the other host, adding ++netstack=vmotion if vMotion has its own TCP/IP stack. If a 1,472 byte test passes and 8,972 fails, one hop is not set for jumbo frames. Then I check the VLAN on both hosts and processor compatibility or EVC.

A host shows not responding. Its VMs keep running; only management is lost. I check the management address answers, then restart the agents with /etc/init.d/vpxa restart and /etc/init.d/hostd restart, and read /var/log/vpxa.log and /var/log/hostd.log. A full services.sh restart is a last resort.

A VM is slow but the host looks fine. In esxtop, CPU ready (%RDY) above about 5 percent per vCPU or a high co stop (%CSTP) means contention, a balloon (MCTLSZ) or swap (SWCUR) means memory pressure, and DAVG or KAVG show disk latency. The fix is often fewer vCPUs, not more.

Users cannot log on. I check time first with w32tm /query /status, then that the client's DNS points only at domain controllers, then nltest /dsgetdc:example.com to see which one it finds. dcdiag /v and repadmin /replsummary show whether the domain controllers and their replication are healthy.

A datastore is degraded or a VM hangs on disk. esxcli storage core path list shows the state of every path, and esxcli storage nmp device list the multipathing policy. One dead path should cause no outage. If it did, the multipathing or the zoning was wrong. On iSCSI I also check port binding and the MTU on both storage networks.