Speed
A new segment, gateway or rule is a software change through the NSX Manager or its API (application programming interface). The physical switches do not change.
VMware NSX
Networking and security inside the hypervisor
NSX moves switching, routing and firewalling into the ESXi kernel, so the physical network only carries IP between hosts. This page explains how it is built, how traffic really flows through it, and how I prove which layer holds a fault.
Why NSX
A firewall on every virtual machine, not just at the edge
In a classic data centre the firewall sits at the perimeter, and servers in the same VLAN can reach each other freely. NSX puts switching, routing and a stateful firewall into the hypervisor kernel, so every virtual NIC (network interface card) has its own checkpoint and the physical network only carries IP between hosts.
Swipe sideways to see the whole diagram.
Perimeter only. The firewall checks traffic coming in from outside, but the three tiers share a segment. Once an attacker controls the web server, nothing stops it reaching the database directly.
Distributed firewall. NSX enforces a stateful firewall at every virtual NIC, inside the same subnet. Web reaches App on port 8443, and the attempt to reach the database is dropped before the packet leaves the web server's host.
Policy that follows the workload
Rules are written against groups and tags, and the firewall state moves with a virtual machine during vMotion.
The trade off
There is a new layer to troubleshoot, and changes depend on healthy managers. Forwarding carries on with the last known state if the managers fail, so I plan for both.
Architecture
Four planes, one desired state
NSX separates deciding from forwarding. The management plane holds what I asked for. The central control plane works out what each host needs to know. A local agent on every host programs the kernel, and the data plane forwards packets.
Swipe sideways to see the whole diagram.
I make a change in the UI (user interface), the REST (representational state transfer) API or Terraform. The management plane checks it and stores it as desired state in a database replicated across three Manager appliances.
The central control plane, inside the same appliances since NSX-T 2.4, works out MAC, ARP (Address Resolution Protocol) and tunnel endpoint tables, routes and firewall rules. It pushes them only to the hosts and Edges that need each segment or rule.
An agent on each host and Edge receives its share and programs the kernel or the Edge datapath. It also reports what it learns, such as a new virtual machine's MAC and IP address.
Packets are switched, routed and firewalled in the ESXi kernel, and in the Edge's DPDK (Data Plane Development Kit) datapath for traffic leaving the overlay. No packet passes through the Managers.
With one Manager down the cluster still has quorum and works. With two down there is no quorum: no changes, and new virtual machines or host reboots suffer. The data plane keeps forwarding with the last state.
Manager cluster
Three appliances, never one in production, spread across hosts and datastores with anti affinity rules. A cluster virtual IP in one subnet, or an external load balancer, gives a single address. Storage latency is a common cause of trouble, so their datastores must be fast and monitored.
Compute managers
NSX is independent of vCenter but registers it as a compute manager for inventory, Edge deployment and the distributed switch. If that link breaks, host preparation and dynamic groups go stale, but forwarding continues.
Policy and manager modes
The policy API is declarative and is the main UI and API today. The older manager API is imperative and hidden by default. The CLI (command line interface) still shows older names, such as logical switch for segment and logical router for gateway.
| Name | What it was | What changed |
|---|---|---|
| NSX-V | VXLAN (Virtual Extensible LAN) overlay tied to one vCenter, with Edge services gateways and separate controller VMs | End of general support in January 2022. Migration Coordinator moves estates to the newer product. |
| NSX-T 2.x and 3.x | Geneve overlay independent of vCenter, Tier-0 and Tier-1 gateways, control plane inside the Managers from 2.4 | Version 3 added VDS (vSphere Distributed Switch) 7 as a host switch, virtual routing and forwarding (VRF) Lite and EVPN (Ethernet virtual private network) on the Tier-0, and OSPF (Open Shortest Path First) from 3.1.1. |
| VMware NSX 4 | The same product renamed from version 4.0 in 2022 | N-VDS removed on ESXi hosts, which must use VDS 7 or later. Edges keep their own switch. |
| NSX 9 | Released with VMware Cloud Foundation (VCF) 9 in 2025 | Bundled with VCF. Security features such as the distributed firewall are sold separately as vDefend. |
Transport
Transport zones decide which hosts see which networks
A transport node is a prepared ESXi host or an Edge. Each one has tunnel endpoints (TEPs) on a dedicated VLAN. A transport zone is the set of transport nodes that can carry a group of segments: overlay zones for Geneve segments, and VLAN zones for segments tagged on the physical network.
Swipe sideways to see the whole diagram.
Overlay zone. Every member has TEPs, and NSX builds Geneve tunnels between them with BFD (bidirectional forwarding detection) to track their state. Overlay segments exist on every member, without any change to the physical switches.
VLAN zone. Segments here are tagged straight onto the physical network. They carry Edge uplinks to the top of rack (ToR) switches, one VLAN per switch, and for workloads that must stay on a VLAN but still get the distributed firewall.
Host left out. A host missing from the transport zone, usually configured by hand rather than from the cluster's profile, never receives the overlay segments. A virtual machine placed there has no network, which is why I prepare hosts only from transport node profiles.
Host preparation, in order
- VDS 7 or later in vCenter, with its MTU (maximum transmission unit) at 1,600 or more.
- An uplink profile: teaming policy, the TEP transport VLAN and named teaming for Edge uplinks.
- A transport node profile for the cluster: host switch, transport zones, uplink profile, TEP addresses from a pool or DHCP (Dynamic Host Configuration Protocol), and the vmnic mapping.
- Apply it to the cluster, then check every host shows Success and every tunnel is up.
TEP addressing
Hosts get TEPs as vmk10 and vmk11 on their own TCP/IP (Transmission Control Protocol and Internet Protocol) stack, called vxlan, with its own gateway. A common design gives each rack its own TEP subnet with a gateway in the IP pool, so TEP traffic is routed between racks. Duplicate TEP addresses, an exhausted pool and the wrong transport VLAN are the most common faults.
| What NSX needs from the physical network | How it is checked |
|---|---|
| TEP VLAN trunked to every host and Edge uplink | show interface trunk on the switches, and the uplink profile VLAN |
| MTU of at least 1,600 on every hop. VMware recommends 9,000 on the physical network, with the tunnel endpoint MTU at its 1,700 default | vmkping ++netstack=vxlan -I vmk10 -d -s 1572 to every remote TEP (1,672 for 1,700) |
| Routing between per rack TEP subnets | show ip route for each TEP subnet, and the gateway in the IP pool |
| Edge TEPs placed correctly when Edge VMs run on prepared hosts. They can share the host TEP VLAN only if the Edge TEP port group is NSX backed. Otherwise they need their own VLAN, routed to the host TEPs. | Tunnel status between Edges and hosts, not only between hosts. A wrong placement leaves host to host tunnels up and raises the Edge tunnels down alarm. |
| Edge uplink VLANs to the switches, with BGP (Border Gateway Protocol) or OSPF and BFD | get bgp neighbor summary on the Edge and show bgp ipv4 unicast summary on the switches |
| Geneve on UDP (User Datagram Protocol) 6081 and BFD allowed between TEP subnets | Ideally no firewall in that path at all |
| NTP (Network Time Protocol) and DNS (Domain Name System), forward and reverse, for Managers, Edges, hosts and vCenter | Name lookups and time checks before installing |
Geneve
The tunnel between hosts, and the MTU it needs
NSX carries overlay segments between TEPs in Geneve (Generic Network Virtualization Encapsulation) over UDP port 6081. Geneve is defined in Request for Comments (RFC) 8926. It looks like VXLAN, but its header can carry variable length options, which is why NSX asks for more MTU headroom.
Swipe sideways to see the whole diagram.
VM A sends a frame to VM B on another host. The host finds VM B behind host 2's TEP in the table the control plane gave it.
8 bytes with the segment's 24 bit VNI (virtual network identifier) and the protocol type for Ethernet, followed by optional metadata of variable length.
8 bytes of UDP to port 6081. The hashed source port spreads flows across the physical links.
20 bytes from TEP to TEP with the do not fragment bit set. The physical network only ever routes TEP addresses.
50 bytes plus options. NSX needs at least 1,600 bytes on every hop, VMware recommends 9,000 on the physical network, and the test is
vmkping ++netstack=vxlan -I vmk10 -d -s 1572. If the underlay cannot carry more than 1,500, the NSX overlay is not supported. Fix the underlay, or use VLAN backed segments until it is fixed.
| VXLAN | Geneve | |
|---|---|---|
| Standard | RFC 7348 | RFC 8926 |
| UDP port | 4789 | 6081 |
| Header | Fixed 8 bytes | 8 bytes plus options up to 252 bytes |
| Segment ID | 24 bit VNI | 24 bit VNI |
| Control plane | Flood and learn, or BGP EVPN | The NSX central control plane |
| Underlay MTU | 1,550 minimum for 1,500 byte hosts | 1,600 minimum, 1,700 recommended |
| Where I meet it | Switch fabrics and Cisco ACI (Application Centric Infrastructure) | NSX and some other hypervisor overlays |
Copying broadcast traffic between hosts
Swipe sideways to see the whole diagram.
Hierarchical two tier. The sending host copies the frame to hosts in its own TEP subnet, and to one proxy TEP in each other subnet. Each proxy copies it to the hosts in its rack. It suits a design with a TEP subnet per rack and needs no multicast in the physical network.
Head end. The sending host makes one copy for every other host on the segment. It is simple but puts all the work on the sender's uplinks. With a single TEP subnet, both modes behave the same.
Segments
A Layer 2 network that lives in software
A segment, once called a logical switch, is a Layer 2 network. An overlay segment gets a VNI and travels in Geneve. A VLAN segment is tagged on the physical network. Either way it appears in vCenter as a port group, and every port gets the distributed firewall.
Swipe sideways to see the whole diagram.
VM A powers on in host 1. The host learns its MAC address and, from ARP, DHCP or VMware Tools, its IP address. The local agent reports both, with host 1's TEP, to the central control plane.
The control plane updates three tables for the segment: MAC to TEP, IP to MAC, and which TEPs carry the segment. It pushes them only to hosts that have the segment, so host 3 receives nothing.
VM B on host 2 sends an ARP request for VM A. Host 2 answers from its ARP table, so the broadcast never crosses the network. Unknown entries fall back to replication.
IP and MAC discovery
The IP discovery profile learns addresses by ARP and neighbour discovery snooping, DHCP snooping or VMware Tools, and feeds both the ARP table and the firewall's address sets. MAC discovery allows MAC learning for nested hosts and appliances.
SpoofGuard
SpoofGuard binds a virtual NIC to its IP and MAC. A virtual machine given a new IP address cannot talk until the binding is approved, which is the first thing I check when a readdressed server goes silent.
Segment security
The segment security profile filters BPDUs, blocks rogue DHCP servers, guards router advertisements and rate limits broadcast traffic. The QoS profile marks or limits traffic per segment.
Gateways
Tier-0 for the provider, Tier-1 for each tenant
NSX routes in two tiers. A Tier-0 gateway connects to the physical network through Edges. Tier-1 gateways connect segments for each tenant or application. Each gateway has a distributed router (DR) in every host's kernel, and may have a service router (SR) on an Edge for anything stateful.
Swipe sideways to see the whole diagram.
Distributed only. The Tier-1 has no Edge cluster, so it is a DR in every host. Web to App is routed on the source host. Traffic to the outside goes Tier-1 DR, Tier-0 DR, then a Geneve tunnel to an Edge where the Tier-0 SR talks BGP to the switches.
With a service router. Adding NAT (network address translation), a gateway firewall, a VPN (virtual private network) or a load balancer gives the Tier-1 an active and standby SR on an Edge. Web to App still routes on the host, but everything leaving the Tier-1 now passes through that SR. Slowness that starts when NAT is enabled is often this extra hop.
Distributed router
The DR has an interface in every connected segment, with the same gateway IP and the same virtual MAC on every host, so a virtual machine keeps its gateway after vMotion. Routing between segments never leaves the source host.
Service router
The SR lives only on Edges and holds everything that needs a central point: uplinks, BGP and OSPF, NAT, the gateway firewall, VPN and load balancing. The DR and SR are joined by an internal transit link.
| Internal address range | Default | Where it appears |
|---|---|---|
| Tier-1 to Tier-0 router links | 100.64.0.0/16, a /31 per link | Edge get route output, host DR tables and traceroutes. It overlaps the RFC 6598 range used for carrier NAT and some cloud links, so I check before build. |
| Transit between SR and DR | 169.254.0.0/24 | Inside each gateway, between its SRs and its DR |
| Single tier: segments on the Tier-0 | Two tier: segments on Tier-1s | |
|---|---|---|
| Simplicity | Fewer objects | One more logical hop, still in the kernel |
| Blast radius | Every change touches the Tier-0 and its BGP | Application changes stay on Tier-1s |
| Stateful services | Force the Tier-0 into active and standby | Run on Tier-1 SRs while the Tier-0 stays active and active |
| Tenancy | Weak | A natural boundary per tenant or application |
Projects and virtual private clouds
From NSX 4.1, Projects give each tenant its own slice of NSX: its own administrators, Tier-1 gateways, segments, groups and distributed firewall rules, on a Tier-0 and Edge cluster the enterprise administrator may share between Projects. From NSX 4.1.1, a Project can hold virtual private clouds (VPCs), where an application team creates its own subnets, NAT and rules within quotas. NSX 9 in VCF 9 builds on the same model.
Edges and BGP
Where the overlay meets the physical network
An Edge node is a transport node that runs service routers. Tier-0 SRs peer with the top of rack switches, normally with eBGP and BFD. The choice between active and active or active and standby decides throughput, failover and which stateful services the gateway can run.
Swipe sideways to see the whole diagram.
Active and active. Every Edge forwards, and hosts and switches load share across up to eight paths. Losing an Edge removes one path. Classic releases allow only stateless reflexive NAT in this mode, and newer releases add a stateful option for some services, so I check the version.
Active and standby. One Edge forwards and the other waits, with its routes made less attractive. NAT, the gateway firewall and VPN need this mode in classic releases. Failover is brief, but half the capacity sits idle.
| Edge size | vCPU and memory | Typical use |
|---|---|---|
| Small | 2 vCPU, 4 GB | Proof of concept only |
| Medium | 4 vCPU, 8 GB | Light production and labs |
| Large | 8 vCPU, 32 GB | Most production Tier-0 Edges |
| Extra Large | 16 vCPU, 64 GB | Heavy services such as load balancing or VPN |
| Bare metal | Physical server | The highest throughput and fastest failover |
VM form factors as documented for NSX 4. The Edge datapath polls its cores with DPDK, so vCenter shows high CPU even when the Edge is quiet. get dataplane cpu stats shows the real load.
Swipe sideways to see the whole diagram.
The segment must be connected to a Tier-1 gateway, and that Tier-1 linked to the Tier-0. A segment with no gateway has nowhere to route.
The Tier-1 advertises only what is ticked: all connected segments and service ports, NAT IPs, load balancer addresses, static routes and others, each one separately.
The Tier-0 redistributes Tier-1 connected routes into BGP, optionally through a route map. I check with
get routeon the Tier-0 SR, where the subnet shows as t1c.The Tier-0 sends only NSX prefixes, and the switches accept only NSX ranges.
get bgp neighbor 192.0.2.1 advertised-routeson the Edge andshow bgp ipv4 unicast 10.30.5.0/24on the switch prove each side.The switches send a default route, or the routes NSX needs, back to the Tier-0. Without it, traffic reaches the virtual machine and the reply has nowhere to go.
BGP to the switches
eBGP from each Edge to both switches: two Edges and two switches make four sessions. A private AS (autonomous system) for the Tier-0, such as 65100, different from the switches. BFD on every session, because NSX's default BGP timers are 60 seconds keepalive and 180 seconds hold. Multipath on both sides. Prefix lists both ways. If two sites reuse one AS, the Tier-0 neighbour has an allowas-in option so it accepts routes that already carry its own AS.
Edge clusters and failure
Edges sit in an Edge cluster, grouped into failure domains, typically one per rack, so the active and standby Tier-1 SRs never share a rack. An Edge is declared down when BFD to its peers is lost over both the management and TEP networks. A Tier-0 SR also fails over when all of its BGP sessions are down.
OSPF or static
The Tier-0 also supports OSPF from NSX-T 3.1.1, and static routes with BFD tracking the next hops. I prefer BGP for its filtering and load sharing, but I pick what the operations team can support at 3am.
VRF Lite, EVPN and Federation
A parent Tier-0 can hold VRF gateways, each with its own routing table and BGP sessions over tagged interfaces. The Tier-0 can also speak EVPN to a data centre gateway. Federation stretches policy and segments across sites through a Global Manager, with remote TEPs on the Edges.
| NAT type on a gateway SR | What it does | Watch for |
|---|---|---|
| SNAT (source NAT) | Changes the source address of outbound traffic | Stateful, so active and standby in classic releases |
| DNAT (destination NAT) | Publishes a service on a public or shared address | The NAT address must be advertised and redistributed |
| Reflexive | Stateless one to one translation | Suits an active and active Tier-0 |
| No SNAT and No DNAT | Exemptions for traffic that must keep its address | Rules apply in priority order, lowest number first |
The distributed firewall sees a virtual machine's real address. On a gateway, each NAT rule's Firewall setting decides whether the gateway firewall matches the internal address, the external address, or is bypassed.
Distributed firewall
First match wins, category by category
The distributed firewall is a stateful firewall in the ESXi kernel, one instance per virtual NIC. Rules are written once in the NSX Manager and pushed to the hosts that need them. They are read in fixed categories, top to bottom, and the first match decides. This is micro segmentation: a policy boundary around every workload, not only around every subnet.
Swipe sideways to see the whole diagram.
Layer 2 rules by MAC address and EtherType are read first and are always stateless.
Rules that must win over everything else, such as quarantining a compromised host or a break glass allow during an incident.
Shared services every workload needs: DNS, NTP, Active Directory, patching, backup and monitoring. An allow here is final, so a DNS query never reaches later categories.
Broad separation between environments and zones, such as test from production or PCI (Payment Card Industry) from everything else.
Per application rules between tiers, written against groups and tags rather than IP addresses, each policy ending in a logged deny.
Out of the box it allows everything. I change it to drop with logging only after flows are learned and every application has its rules, under a planned change with a quick rollback.
Groups and tags
Rules use groups. Members can be static, such as virtual machines, segments, IP ranges or Active Directory groups, or dynamic, such as every virtual machine tagged app|payroll and tier|web. Renaming a virtual machine or removing a tag silently changes the firewall, so I control who can tag, automate it, and review effective members regularly.
Applied To
Applied To decides which virtual NICs receive a rule. Narrowing it saves memory and avoids accidental matches. Too narrow, such as only the source group, and the destination never sees the allow, so its own default deny drops the traffic. I apply rules to both ends, or to the application's group.
Context profiles
App ID matches the real Layer 7 application whatever the port. FQDN (fully qualified domain name) rules need a DNS snooping rule with the DNS context profile placed above them. Identity firewall rules match Active Directory groups, which suits virtual desktops. All of these cost host CPU and need the right licence.
Exclusion list and TCP strict
The exclusion list removes the firewall from a virtual machine entirely. Adding one "to test the firewall" must be a recorded emergency change, reversed afterwards. TCP strict drops TCP that did not start with a handshake, so I test it against long lived sessions first.
| Distributed firewall | Gateway firewall | |
|---|---|---|
| Where it runs | ESXi kernel, at every virtual NIC | On a Tier-0 or Tier-1 SR on the Edges |
| Traffic | East to west, even inside one subnet | North to south through that gateway |
| Scale | Grows with every host added | Limited by Edge capacity |
| Proof of a drop | Traceflow shows the rule ID at a virtual NIC | Traceflow shows the drop at the SR |
Moving to default deny
I learn flows with logging, add infrastructure rules, then per application rules ending in a logged deny. I watch the logs, and only then change the default rule under a planned change. Inside the data centre I prefer reject for fast failure. At security boundaries, policy decides.
Distributed intrusion detection and prevention
An intrusion detection system (IDS) and intrusion prevention system (IPS) can run at each virtual NIC. I start in detect only, tune false positives, then move chosen signatures to prevent under change. Isolated networks need an offline path for signature updates.
Neither replaces the perimeter
The distributed firewall is a lock on every office. The gateway firewall is the door to the building. An accredited perimeter firewall still guards the boundary where policy requires one.
Packet walks
Follow the packet through the hosts
Most NSX faults become obvious once I know which path a packet should take. These walks cover the four cases that matter most: same host, across hosts, routed between segments, and out through the Edges.
Swipe sideways to see the whole diagram.
The ARP request is answered from the host's table. The firewall checks the packet leaving VM A and entering VM C, and the host switches it between them. No physical NIC, no Geneve. If this works but the next case fails, the fault is in the tunnels or the underlay.
Host 1 finds VM B behind host 2's TEP. It wraps the frame in Geneve with the Web segment's VNI and sends it TEP to TEP over the physical network. Host 2 removes the outer headers and delivers to VM B. A capture on the uplink shows UDP 6081 between the TEPs.
VM A's gateway is the Tier-1 DR on its own host. The DR routes the packet into the App segment, lowers the TTL (time to live) and tunnels it with the App segment's VNI. Host 2 only switches it to VM D.
VM D's reply is routed by the DR on host 2 and comes back with the Web segment's VNI. A capture filtered on one VNI shows only half of the conversation, which catches people out.
Swipe sideways to see the whole diagram.
The firewall checks the packet at the virtual NIC. The Tier-1 DR and the Tier-0 DR, both in the host's kernel, route it across the 100.64 router link without leaving the host.
The Tier-0 DR has equal cost routes to the SRs on every Edge. It picks one and tunnels the packet in Geneve from the host's TEP to that Edge's TEP.
The Tier-0 SR on Edge 1 uses its BGP learned route and sends the packet out of its uplink VLAN to the top of rack switches.
The switches load share too, so the reply may arrive at Edge 2. Its Tier-0 SR, Tier-0 DR and Tier-1 DR route it on the Edge, then tunnel it to the host, where the firewall checks it on the way in. Different Edges each way is normal, unless a stateful firewall sits above the switches.
Troubleshooting
Traceflow first, then prove the layer
Traceflow injects a test packet at a source virtual NIC and reports every point it was seen, forwarded, delivered or dropped, including the firewall rule ID. It tells me which layer to look at. Pick a symptom to see where the cause usually sits.
Swipe sideways to see the whole diagram.
Prove the tunnels. Tunnel status and BFD, then a full size TEP ping, then the switch trunks and VLANs.
Small works, big fails. A do not fragment TEP ping at 1,572 bytes of data fails while a small one works. Check every hop.
Follow the gates. Attached, advertised, redistributed, allowed, and a route back.
Both ends, then the path. Neighbour state on the Edge and the switch, then the uplink VLAN, ARP and filters.
Back out first. Roll back under emergency change, then use Traceflow and rule hits to find the cause.
Read the failing step. The node status names it, then check software, connectivity and the profile.
Same segment works on one host, not across hosts
- Tunnel status in the UI, then on the host
nsxdp-cli bfd sessions list. - TEP addresses and MTU:
esxcfg-vmknic -l. - Reach the remote TEP:
vmkping ++netstack=vxlan -I vmk10 172.20.2.12, then add-d -s 1572. - On the switches:
show interface trunk,show vlan id 120for the TEP VLAN, andshow mac address-table vlan 120. - Control plane: the host connected to the Managers with
get managersandget controllersin nsxcli, and the segment's tables withget logical-switch 69633 mac-tableandget logical-switch 69633 vtep-table, using the segment's VNI. - If one host only: compare its switch ports, vmnic mapping and TEP addresses with a working host.
Fix: correct the VLAN, trunk, route, rule or MTU in the underlay. Prevention: hosts prepared only from profiles, and full size TEP pings in every acceptance test.
Small pings work, large transfers stall
vmkping ++netstack=vxlan -I vmk10 -d -s 1572 172.20.2.12fails while a small ping works.- VDS MTU in vCenter, then host ports, port channels, the vPC peer link, SVIs, routed links and spine links.
- Any device added or replaced recently, because new kit often arrives at 1,500.
- For north to south only: the Edge uplink MTU against the switch SVI (switched virtual interface), and any WAN (wide area network) or firewall beyond with a smaller MTU and ICMP (Internet Control Message Protocol) blocked.
Fix: a consistent MTU under change, 9,216 on the fabric, then retest at 1,572 or 8,972. MSS (maximum segment size) clamping is only a workaround. Prevention: MTU in the design standard and in every acceptance test.
New segment unreachable from the physical network
- Segment attached to the Tier-1, and the Tier-1 linked to the Tier-0.
- Tier-1 route advertisement for connected segments, and for NAT IPs if NAT is in use.
- On the Edge:
get logical-routers,vrf 1for the Tier-0 SR, thenget route 10.30.5.0/24shows t1c. get bgp neighbor 192.0.2.1 advertised-routesshows it leaving the Edge.- On the switch:
show bgp ipv4 unicast 10.30.5.0/24,show ip route 10.30.5.0/24and the inbound prefix list. - Return path: the default route learned by the Tier-0 with
get bgp neighbor 192.0.2.1 routes.
Fix: tick advertisement or redistribution and update filters through change. Prevention: a prefix list and redistribution check in the segment build runbook.
Tier-0 BGP neighbour down
- Scope: one Edge or both, one switch or both. All sessions at once points at the switch or the uplink VLANs.
- Edge:
get bgp neighbor summary,get interfaces, then a ping to the switch from the SR's VRF. - Switch:
show bgp ipv4 unicast summary,show bfd neighbors,show ip arpfor the Edge uplink, and the SVI state. - Uplink VLAN trunked to the ESXi host running the Edge, and present on the VDS port group.
- Neighbour IP, AS, password, hop limit, ACL (access control list) or control plane policing on TCP 179, and BFD timers on both sides.
Fix: correct the Layer 2 path, BGP settings or filter. With ECMP (equal cost multipath) the impact is reduced resilience, so I fix it through a normal change unless traffic is affected.
An application broke straight after a firewall change
- Fix first: back out under emergency change, using the saved configuration version.
- Traceflow from client to server on the application port. It names the rule that dropped it.
- Rule hit counts and
get firewall packetlog last 20in nsxcli on the host. - Category and order: a new deny above an allow, or in an earlier category.
- Group effective members: a tag removed, a virtual machine renamed or rebuilt with a new address.
- What reached the host:
summarize-dvfilterto find the filter, thenvsipioctl getrules -fandvsipioctl getaddrsets -f.
Prevention: peer review of firewall changes, a Traceflow test in every change plan and logging on every new deny.
Gateway, Manager, Edge and host problems
- A virtual machine cannot reach its gateway: its IP, mask and gateway, the virtual NIC on the right segment, and the segment connected to a Tier-1. Then Traceflow to the gateway IP, and the SpoofGuard or IP discovery binding after an IP change. A working virtual machine on the same segment is the best comparison.
- One Manager down:
get cluster statuson a healthy node, then datastore latency, disk space and certificates. Freeze non urgent changes and confirm the last good backup. - Two Managers down: no quorum, so a priority one incident and a vendor case. Avoid host reboots and vMotion until it is fixed.
- Edge busy, north to south slow:
get dataplane cpu stats,get physical-port fp-eth0 statsand how many SRs share the Edge. Move SRs or add Edges. - Host preparation failed: the node status names the step. Then the NSX VIBs,
esxcfg-vmknic -land the syslog, and retry with the host in maintenance mode. - Drops after an upgrade: stop the upgrade, move workloads to healthy hosts and open a vendor case.
- Intermittent drops with a stateful firewall above the switches: the two directions use different Edges. I agree a fix with the design authority rather than turning off ECMP.
| Where | Command or tool | What it tells me |
|---|---|---|
| NSX Manager | Traceflow | Every point a test packet was seen, delivered or dropped, with the rule ID |
| Edge | get logical-routers, then vrf 1 | Every Tier-0 and Tier-1 SR and DR on the Edge, then enter one |
| Edge | get route, get bgp neighbor summary | Routes with their source, and BGP peers with prefix counts |
| Edge | get tunnel-ports, get edge-cluster status | Tunnels to remote TEPs, and the health of each Edge |
| Edge | get high-availability status, get dataplane cpu stats | Which SR is active, and the real datapath load |
| ESXi host | esxcfg-vmknic -l, vmkping ++netstack=vxlan | TEP addresses and MTU, and full size tunnel tests |
| ESXi host | nsxdp-cli bfd sessions list, esxcli network ip route ipv4 list -N vxlan | BFD state to every remote TEP, and the TEP routes |
| ESXi host | pktcap-uw --dvfilter with PreDVFilter and PostDVFilter | Captures before and after the firewall, which proves a drop |
| nsxcli on a host | get firewall packetlog last 20, get logical-switch 69633 mac-table, get logical-switch 69633 arp-table and get logical-switch 69633 vtep-table | Recent firewall hits, and the MAC, ARP and TEP tables for a segment |
| nsxcli on a host | get managers, get controllers, get firewall status | The host connected to the Managers, and the firewall enabled |
Command names change between releases, and newer releases rename some Edge commands. I check the CLI reference for the version in front of me.
NSX, ACI and EVPN
The same idea, built in different places
NSX, Cisco ACI and a VXLAN EVPN fabric all separate the network people use from the network that carries it. NSX does it in the hypervisor, the other two in the switches. Knowing one makes the others easy to read.
| VMware NSX | Cisco ACI | VXLAN EVPN | |
|---|---|---|---|
| Tunnel endpoints | Hosts and Edges | Leaves | Leaves |
| Encapsulation | Geneve, UDP 6081 | VXLAN based | VXLAN, UDP 4789 |
| Control plane | NSX central control plane | APIC (Application Policy Infrastructure Controller), COOP (Council of Oracle Protocol) and MP-BGP (Multiprotocol BGP) | MP-BGP EVPN |
| Policy enforcement | Distributed firewall at the virtual NIC | Contracts on the leaves | ACLs or external firewalls |
| Gateway | Distributed router in every host | Pervasive gateway | Anycast gateway |
| Exit to the outside | Tier-0 SR on the Edges | L3Out on border leaves | Border leaves |
| Management | NSX Manager cluster | APIC cluster | CLI, Nexus Dashboard |
| Cisco ACI concept | Nearest NSX concept |
|---|---|
| APIC cluster | NSX Manager cluster |
| Endpoint group | Group |
| Contract | Firewall rule with services |
| Bridge domain and subnet | Segment with a gateway |
| VRF | Tier-1 or Tier-0, or a Tier-0 VRF |
| L3Out on border leaves | Tier-0 uplinks on Edges |
| Leaf VTEP (VXLAN tunnel endpoint) | Host TEP on a vmkernel port |
NSX on top of a fabric
NSX can run over ACI or a VXLAN EVPN fabric as its underlay. Then the fabric only sees Geneve between TEPs, so both layers need MTU headroom, and I decide in one recorded decision where virtual machine policy lives.
Licensing, with care
NSX networking comes with VMware Cloud Foundation. Security features such as the distributed firewall and IDS are sold as the vDefend add on, and the Avi load balancer is separate. Older estates may hold different entitlements, so I check before proposing a feature.
Controller based networks
NSX, SD-WAN (software defined WAN) and SD-Access share the same split between management, control and data planes. When the controller is unhealthy, changes stop but forwarding carries on. I troubleshoot them plane by plane.
The switch fabric side of this comparison is on the VXLAN EVPN and ACI page, and vSphere itself is on the servers and virtualisation page.
Engagement
How I would run an NSX programme, from flow capture to refresh
NSX is not in my production history. This is the method I would bring from Cisco ACI overlay work and VMware data centre estates. NSX touches three teams at once: virtualisation, network and security, so I would bring them together early and agree what each one owns before design starts.
Swipe sideways to see the whole diagram.
I would first agree why NSX is wanted: micro segmentation, faster network changes, tenant isolation or a private cloud platform. Then I would settle who owns the gateways, the firewall rules and the hosts, because that decides the support model.
Functional requirements would cover segments, gateways, NAT, load balancing and firewall policy. Non functional ones would cover north to south throughput, Edge failover time, firewall rule scale and how fast a new application network is delivered. Each one is traced to a test.
I would read the vSphere estate with RVTools, check the physical network for MTU, VLANs and routing, and collect real flows before any firewall rule is written. Application owners confirm which flows are expected and which are noise.
The design would cover transport zones, TEP addressing, Edge clusters and failure domains, Tier-0 peering with the fabric, a Tier-1 per tenant pattern, and the firewall categories and tags. It is the same discipline as an ACI tenant, VRF and contract design, reviewed by the design authority.
Hosts are prepared from transport node profiles, then tested: TEP pings with full size packets, Edge failover, BGP and BFD to the switches, and Traceflow for every pilot flow. Firewall rules start with logging before any default deny, and a partner with NSX delivery depth joins the build.
Each team would get runbooks for its part, a shared triage flow that starts with Traceflow, alarms routed to the right queue, and templates for common rule and segment changes. Hypercare ends when the agreed exit criteria are met.
NSX 4.x, vSphere 8 and VCF 5.2 reach end of service on 11 October 2027, so the move to VCF 9 needs a plan now. Upgrades follow the interoperability matrix and the upgrade order: Edges, then hosts, then the Managers. Licensing is reviewed at every renewal.
Functional requirements
What it must do.
- Give each application its own segments and gateway, without changes to the physical switches.
- Control traffic between virtual machines, including inside the same subnet.
- Route between tenants and to the outside through Tier-0 gateways peered with the fabric.
- Provide NAT, VPN and load balancing where an application needs them.
- Keep policy attached to a virtual machine when it moves between hosts.
- Deliver a new application network from a template or the API.
Non functional requirements
How well it must do it, with a number wherever one can be agreed.
- Availability
- No single Edge, host or Manager node failure stops traffic. Edge failover in a few seconds or less.
- Performance
- North to south throughput per Edge cluster agreed and tested, with headroom for growth.
- MTU
- At least 1,600 bytes on every underlay hop. VMware recommends 9,000 on the physical network, with the tunnel endpoint MTU at its 1,700 default, proved with full size TEP pings.
- Security
- Default deny between applications, logged denies, and firewall logs sent to the security information and event management (SIEM) system.
- Scale
- Rules, groups, segments and gateways within the published configuration maximums, with headroom.
- Change
- A new application network in hours, with peer review for every firewall change.
- Recovery
- Manager backups every day, tested restores, and a documented rebuild for each Edge.
These are typical starting targets for an NSX platform. Each organisation sets its own, and each one is traced to an acceptance test.
Who I talk to
- vSphere and platform teams
- Network team for the switches and routing
- Security team for policy and logging
- Application owners for flows and change windows
- Operations and the service desk
How I find out
- Workshops per application group
- Flow collection before any rule is written
- Reading host, VDS and switch configuration
- Checking MTU and VLANs on every path a TEP will use
- Mapping who changes hosts, switches and rules today
Discovery tools
| Tool or source | What it tells me |
|---|---|
| RVTools or vCenter inventory | Clusters, hosts, virtual machines, port groups and VLANs. |
| VCF Operations for networks (formerly Aria Operations for Networks) | Flows per application and suggested distributed firewall rules. |
Top of rack switch configuration and show interface trunk | TEP and Edge uplink VLANs on every trunk, with MTU and routing. |
vmkping ++netstack=vxlan with the do not fragment bit | Proof that every TEP path carries full size packets. |
| The current perimeter and internal firewall policies | Which zones exist today, so the first NSX categories match them. |
| Configuration management database and IP address management | Owners, environments and addresses, which become tags and groups. |
NSX faults sit in the underlay, the tunnels, the control plane, the gateways or the firewall. The method starts with Traceflow, because it shows the exact point where a packet stopped, then proves the layer underneath with host and Edge commands.
My approach
- Scope. One virtual machine, one host, one segment, one gateway or everything? Since when, and what changed?
- Traceflow. Inject a packet from source to destination and read where it was delivered, dropped or which firewall rule hit it.
- Tunnels. TEP state and BFD between hosts, then a full size
vmkpingbetween TEPs. - Control plane. Host connected to the Managers, and the segment's MAC, ARP and TEP tables populated.
- Routing. Route advertisement on the Tier-1, redistribution on the Tier-0, then BGP to the switches.
- Firewall and record. Rule order, groups and Applied To. Fix through change, confirm with Traceflow, and record the cause.
Common NSX faults
- Underlay MTU below 1,600, so large transfers stall
- TEP VLAN missing from a trunk after a switch change
- New segment not advertised or not redistributed
- A firewall rule placed in the wrong category or order
- A group that lost members after a tag was removed
- Asymmetric return path through a stateful firewall above the Edges
Layer by layer
| Layer | What I check |
|---|---|
| 1 · Physical | Host uplinks, Edge uplinks and switch ports: errors, speed and the right vmnic mapping. |
| 2 · Data link | TEP VLAN trunked to every host, VDS MTU, and show interface trunk on the switches. |
| 3 · Network | TEP routing, vmkping ++netstack=vxlan -I vmk10 -d -s 1572, Tier-0 get route and get bgp neighbor summary. |
| 4 · Transport | Geneve on UDP 6081 and BFD between TEPs allowed, tunnel status in the UI or nsxdp-cli bfd sessions list. |
| 5 to 7 · Application | Distributed firewall rule hits, get firewall packetlog last 20, NAT on the gateways and what the user sees. |
Worked NSX faults are in the troubleshooting section above, and my general method is on the troubleshooting page.
NSX sits between three teams, so the support model has to be explicit before go live.
Service model
Who owns hosts, Edges, gateways, segments and firewall policy, with escalation to Broadcom support.
Shared triage
Every incident starts with Traceflow, so the right team gets the ticket the first time.
Monitoring
Manager cluster health, tunnel status, Edge load, BGP sessions and firewall drops, with alerts tuned in hypercare.
Change
Firewall rules peer reviewed and tested with Traceflow, with saved configuration versions for quick rollback.
Backups
Manager backups to a Secure File Transfer Protocol (SFTP) server every day, with restores tested.
Problem management
A repeat tunnel, routing or rule fault gets a root cause and a fix in the transport node profile, the segment template or the rule standard.
NSX moves with vSphere and VMware Cloud Foundation (VCF), so I would plan its life alongside theirs.
- Plan. One approved release, checked against the interoperability matrix. NSX 4.x and VCF 5.2 reach end of service on 11 October 2027, so the move to VCF 9 needs planning now.
- Upgrade. Edges first, then hosts cluster by cluster starting with a pilot, then the Managers, with Traceflow and BGP checks at each stop point.
- Patch. Broadcom VMware security advisories (VMSA) are triaged by severity and exposure. Critical fixes go through an emergency change.
- Review. Quarterly reviews of rule counts, unused rules, group membership and Edge capacity.
- Licence. Entitlements reviewed at each renewal, because features such as the distributed firewall are licensed separately.
- Refresh. Edge hardware and form factors reviewed against growth, and exit options kept documented.
Projects
Where my overlay and VMware experience comes from
NSX is not in my production history, and I say so plainly. What I bring is overlay and policy based networking on Cisco ACI and Nexus, VMware data centres, and the habit of proving each layer with a packet walk.
Policy based overlay on Cisco ACI
Tier 3 data centre and managed infrastructure for over 300 customer sites on Cisco ACI and Nexus: a software defined, policy based overlay, the same idea NSX applies in the hypervisor.
Enterprise data centre on VMware
Post merger infrastructure integration across a multi site estate and the enterprise data centre, on Cisco, Palo Alto and VMware.
Virtualisation across the group
Local area network (LAN), WAN, voice over IP (VoIP), virtualisation and security across the group, including the warehouse and depot systems linked to the group's enterprise resource planning (ERP) system.
VMware ESXi at every site
One VMware ESXi host per site, with a separate vMotion network in London and New York, running virtual Palo Alto and Check Point firewalls, Cisco ISE and a virtual wireless controller alongside the physical routers and switches.
Backed by VMware Certified Professional, Data Centre Virtualisation, alongside CCNP Enterprise and the CCIE Routing and Switching written exam. See all credentials
Explore
More areas I design and deliver
Every page follows the same idea: how it works, the choices that matter, and where I have used it. The foundations pages explain the basics the others build on.