Why does EVPN/VXLAN exist? The problems that broke the old way

Roadmap

Part 1: The problem that started it all

The world before was easy: VLANs. A VLAN is a broadcast domain - hosts in VLAN 10 can talk to each other directly at Layer 2. Hosts in VLAN 20 cannot. Simple, clean, works great.

The problem: how many VLANs can you have?

The VLAN ID field in the 802.1Q tag is 12 bits. So the hard limit is 2^12 = 4,096 VLANs.

For decades, this was plenty. For a big enterprise, 50 might be enough. Maybe 200 for a data center. The limit might as well not have existed.

Then came the late 2000s, AWS and “cloud computing”: thousands of different customers. Each customer wants its own isolated private network. They want it to feel like their own data center.

The isolation tool is VLANs: Customer 1 - VLAN 100. Customer 2 - VLAN 101. Customer 3 - VLAN 102. Customer 4,097 - VLAN ?

A public cloud needs hundreds of thousands of isolated networks. VLAN gives you 4,096. This is the core problem.

The VLAN ID wasn’t the only thing breaking - spanning tree and MAC table sizes were straining too - but it’s the wall that was impossible to engineer around.

So the first fix targeted the obvious constraint: the label is too small. Make a bigger label.

Part 2: The first fix - a bigger label

That’s what VXLAN is. Instead of the 12-bit VLAN ID, a new field: the VNI - VXLAN Network Identifier. 24 bits. 2^24 = 16 million networks. Every customer gets one. Problem solved.

But there’s a catch: you can’t change the Ethernet header. Billions of switches and NICs already in the world expect the frame exactly as it is.

So engineers used the oldest trick in networking: if you can’t change the envelope, put it inside another envelope.

Take the original Ethernet frame. Don’t touch it. Wrap the whole thing in a normal UDP packet (port 4789) with a small VXLAN header carrying the 24-bit VNI. Now it’s just an IP packet like any other.

Two new words come with this: VTEP and tunnel

The device that wraps and unwraps frames is a VTEP - VXLAN Tunnel Endpoint. The path between two VTEPs is a tunnel. The hosts send normal Ethernet. Their VTEP wraps it, the network routes it as IP, the VTEP on the other side unwraps it. To the hosts, it looks like one big Layer 2 network.

The envelope also solved a second problem, one that existed far below 4,096 VLANs. Data centers were becoming routed - L3 between racks, to escape spanning tree and giant failure domains. But a VM migrating to another rack needs to keep its IP and MAC. You can’t stretch a VLAN across a routed fabric. You can stretch a tunnel.

Two problems, one envelope: more networks, and networks that can go anywhere. 16 million of them, on hardware that only understands IP.

But now here is another new problem. When VTEP-1 wraps a frame addressed to MAC X - where does it send it? Which VTEP has MAC X behind it? A VLAN never had to answer this question: everything was one wire, you flooded, done. Now the “wire” is an entire routed network with hundreds of VTEPs.

Somebody has to know where every MAC lives.

Part 3: The hidden problem - how do VTEPs learn?

VXLAN solved the label. It did not solve learning.

The original standard (RFC 7348) fell back on the oldest Ethernet trick: flood and learn. Give every VNI its own multicast group. When a VTEP doesn’t know where a MAC lives, it floods the frame to the group. Every VTEP in that VNI gets a copy. The right one delivers it. Everyone learns.

This works. For ten tenants.

The cloud math: 100,000 tenants. That’s 100,000 multicast groups. Every spine switch tracks membership for all of them. The control plane drowns in state. Multicast at that scale is hard to configure and harder to troubleshoot.

VXLAN fixed the label and quietly re-imported Ethernet’s oldest weakness. The label was 16 million. The learning was still 1990.

Fixing the learning is a control plane job. And there was already a protocol whose entire purpose was distributing “where things live” at massive scale.

Part 4: EVPN - borrowing BGP’s approach

That protocol is BGP.

BGP has moved the internet’s routing table for nearly four decades. Its whole job is answering one question at planetary scale: “where does this destination live, and how do I reach it?”

It does this for millions of IP prefixes, across hundreds of thousands of routers, run by companies that don’t trust each other. Distribution of reachability is a solved problem.

Why BGP and not OSPF or IS-IS? Because link-state protocols distribute one specific thing: a map of links inside your own network. BGP distributes arbitrary “reachability facts” - and it was already built to carry multiple address families side by side on one session. Adding a new kind of fact to BGP is routine. Teaching OSPF to carry MACs would be surgery.

So engineers asked the obvious question: what if BGP carried MAC addresses the same way it carries IP prefixes?

That’s EVPN.

Not a new protocol - a new address family for a protocol that already existed. BGP, but for MACs.

Now the learning problem disappears. VTEP-1 sees a new MAC on a local port - a server booted, a VM came up. Instead of waiting for someone to flood a question, VTEP-1 immediately tells everyone: it sends a BGP update that says “MAC X lives behind me, VTEP 10.0.0.1.” Every other VTEP receives the update and installs it in its table. Done.

When a frame for MAC X shows up at VTEP-2 later, there’s no guessing and no flooding. The table already has the answer: wrap it, send it to 10.0.0.1. One tunnel, one destination, unicast.

Notice what changed. Flood-and-learn is reactive, but EVPN is proactive: announce what you know the moment you learn it.

Broadcasts still exist - a first ARP still has to reach everyone - but now BGP itself tells each VTEP exactly who to replicate to. No multicast in the underlay, no guessing.

The same trick that let the internet scale to millions of routes now lets a data center scale to millions of MACs.

Part 5: Two protocols, two jobs

Everything above is really one distinction:

They are independent.

VXLAN existed before EVPN and can run without it - that’s the multicast flood-and-learn from Part 3. To be fair: at modest scale, flood-and-learn is genuinely fine. Plenty of networks run VXLAN without EVPN and never feel the pain. The pain is a function of scale - it arrives with the tenant count. EVPN can run without VXLAN - it was born carrying MACs over MPLS. They solve different problems and just happen to be at their best together.

One moves the data. The other distributes the knowledge.

Part 6: Why this matters for AI

This series is about networks for AI, so here’s the connection. An AI cloud is really two networks wearing one badge.

The first is everything around the GPUs: tenant networks, frontend, storage access, management. Thousands of customers, each needing isolation, workloads moving between racks. That’s exactly the problem this post described - and it’s solved exactly this way. EVPN/VXLAN runs the tenant side of every major AI cloud.

The second is the GPU training fabric itself. Here it depends on one question: who owns the GPUs? When one company trains on its own hardware (Meta training Llama) the fabric is pure routed L3. No overlay. One tenant, and nothing needs to keep its address when it moves - jobs get rescheduled, but the operator owns the whole stack and just re-IPs them. The two problems this post described don’t exist, so the tool that solves them isn’t there.

But when a GPU cloud rents that same fabric to many customers, tenancy comes back - and so does EVPN/VXLAN. NVIDIA’s Spectrum-X supports exactly this for multi-tenant AI clouds.

That’s the real lesson: the technology follows the problem. Where isolation is needed, EVPN appears - even inside a training fabric. Where it isn’t, running an overlay is cost without benefit:

• encapsulation overhead
• MTU tax
• one more control plane to operate

Knowing why EVPN/VXLAN exists is what lets you see where it doesn’t belong.

Part 7: What’s next

This post explained why the technology exists. The natural next step is seeing it actually run and break in a lab. That’s where this series is headed: real configs, real debugging, and eventually the parts that make AI fabrics different from everything above: RDMA, lossless Ethernet, and why GPU clusters refuse to share links.