New to EVPN? This post stands on its own, but if you want the ground underneath it first: why EVPN/VXLAN exists and how it moves MACs with BGP.
EVPN gets all the attention. It carries the MAC addresses, it has the clever route types, it’s the thing that goes on the resume.
But EVPN moves nothing on its own. It’s an address book. It tells every switch where every host lives, then hands the packet to the layer underneath and says: your problem now.
That layer is the underlay. It’s the road the overlay drives on. And the road has exactly three jobs:
Reach every VTEP. Use every link. Recover fast.
Everything in underlay design, every protocol choice and every timer, comes back to those three jobs done well or done badly. This post is about doing them well. There are two very different ways to do it, and I built both in the same lab.
Job one: reach every VTEP
The overlay sessions run loopback to loopback. The VXLAN tunnels source from those same loopbacks. So the entire overlay rests on one thing: every VTEP’s loopback has to be reachable from every other VTEP. That’s job one. Easy to say. Two ways to do it, and this is where the designs split hardest.
Road A: BGP everywhere. Give every switch its own AS number. Run eBGP everywhere. Use it for both underlay reachability and the EVPN control plane. One protocol, top to bottom. This is the design many hyperscalers use, the one NVIDIA’s Cumulus lineage is built on, the one written down in RFC 7938. If you know BGP, you know the whole fabric. Fewer moving parts.
It has one sharp edge. The spines aren’t VTEPs. They host no tenant, so when a spine receives an EVPN route it has no local reason to keep it, and by default it throws it away. The result is a fabric with every session up and nothing learned. Making the spine relay routes it can’t itself use takes explicit configuration. That edge turned into a whole evening.
Road B: an IGP underneath, BGP on top. Run OSPF (or IS-IS) for the underlay. Its only job is to flood every loopback everywhere, and that is the one thing an IGP is great at. Then run iBGP for the EVPN overlay, with the spines as route reflectors. A route reflector exists to pass routes along without using them, so the spine-relay problem never appears. Cleaner separation: the IGP handles reachability, BGP handles services, each does one job.
Both roads get every loopback reachable. Both work. I ran them one after the other on the same four switches.
Job two: use every link
In a Clos fabric, every leaf reaches every other leaf by more than one equal-cost path, one through each spine.
The topology most of us call leaf-spine is actually a Clos fabric, named after Bell Labs engineer Charles Clos. He described it for telephone switching decades before Ethernet existed.
Send everything down one path and half your bandwidth sits idle. In a training run, idle bandwidth is idle GPUs, and idle GPUs are money. So the underlay has to spread traffic across every equal path at once. That’s ECMP.
Every tunneled packet wears the same outer IP header, VTEP to VTEP. From the outside they all look identical, so how does the fabric tell one conversation from another to spread them?
The trick is a quiet one. The VTEP computes the outer UDP source port from the inner flow. Same inner conversation, same source port. Different conversation, different port. So the outer UDP source port becomes the entropy the underlay hashes on, and per-flow load spreading works even though every packet looks tunnel-to-tunnel from the outside. Job one is getting there at all. Job two is getting there on every road at once.
The routing table shows that both roads exist. Ask leaf1 how it reaches leaf2’s VTEP loopback, and it installs two next-hops, one through each spine:
A:root@leaf1# show network-instance ipv4 route 10.0.0.2/32
================================================================================
IPv4-unicast route table for default network-instance
--------------------------------------------------------------------------------
# flags legend omitted
--------------------------------------------------------------------------------
Prefix Route Type Metric Pref Flags Next-Hop(s)
--------------------------------------------------------------------------------
10.0.0.2/32 bgp 0 170 > 10.1.1.0(route:local)
10.1.3.0(route:local)Two paths in the table mean the underlay can use both. They do not prove that it did.
For that, I captured VXLAN traffic on both leaf1 uplinks while generating separate TCP flows between the same two hosts.
One flow appeared on the link toward spine1:
10.0.0.1.49701 > 10.0.0.2.4789: VXLAN, vni 10
172.16.10.1.39904 > 172.16.10.3.5201Another appeared on the link toward spine2:
10.0.0.1.56932 > 10.0.0.2.4789: VXLAN, vni 10
172.16.10.1.39900 > 172.16.10.3.5201The VTEP addresses stayed the same: 10.0.0.1 to 10.0.0.2. The VXLAN destination port stayed 4789. The outer source ports changed: 49701 and 56932.
That difference gave the ECMP hash something to work with. One inner flow took spine1. The other took spine2.
The routing table showed two roads. The packet captures showed traffic on both.
Job three: recover fast
A link dies, or a whole spine dies. How long until traffic moves to another spine?
By default, too long. BGP and OSPF lean on hold timers measured in seconds to tens of seconds. For web traffic a few seconds of reconvergence is a blip nobody notices. For a synchronized training run, where every GPU waits on the slowest link, a few seconds is a stall you pay for by the hour.
BFD fixes this. It’s a tiny protocol that does one thing: fires fast hellos between two neighbors and screams the instant they stop arriving. Sub-second detection instead of waiting on a hold timer. BFD notices, the routing protocol withdraws the path, traffic moves to the other one.
Wiring it takes two places. On the interface you set how fast the hellos fire and how many can go missing: 300 milliseconds, three misses, neighbor declared dead in under a second. Then the routing session has to be told to act on what BFD reports. That’s a separate knob, and it’s the one people forget. In the BGP-everywhere design it lives on the BGP group. In the OSPF design it lives on the OSPF interface. Same idea, different place.
On real hardware BFD runs in the forwarding ASIC, which is the entire reason it can be trusted at 300ms. In my lab it runs in software, in a container, sharing a CPU with three other switches. Whatever millisecond figure I could show you would be a fact about my laptop, not about the protocol.
Two roads, same destination
Both designs do all three jobs. Every VTEP reachable, every link used, fast recovery wired in. They get there differently.
BGP everywhere is one protocol, a unique AS per switch, and a spine that needs explicit config to relay routes it can’t use. One routing protocol simplifies operations when you have thousands of switches, which is why it shows up in so many hyperscale fabrics.
OSPF (or IS-IS) plus iBGP with route reflectors is two protocols and a clean split: the IGP does reachability, BGP does services, and the reflector handles relay for free. It’s common in existing enterprise data centers, where an IGP is already part of the network.
Which would I pick? For a greenfield AI fabric, BGP everywhere. For a brownfield data center that already runs an IGP, the reflector design drops in with far less disruption. I built both because understanding one is what makes the other legible.