Two control planes for the VXLAN technology:

  • Multicast control plane (flood and learn)
  • MP-BGP EVPN control plane

Virtual Extensible Local Area Networks (VXLANs) allows to extend reachability of a VLAN within a data center over Layer 3.

Every VTEP with specific VXLAN and certain VNI will join the same multicast group. To learn remote MAC addresses, the VTEP will use conversational MAC address learning technique: learn only actively speaking MAC addresses.

note: Always Read Guidelines and Limitation for specific hardware
Cisco Nexus 5600 Series NX-OS Layer 2 Switching Configuration Guide, Release 7.x

  • Set switching mode to store-and-forwarding
  • IP unicast reachability between VTEPs
  • PIM BIDIR reachability between VTEPs
  • Enable features “vn-segment-vlan-based” and “nv overlay”


conf t
   hardware ethernet store-and-fwd-switching
copy run start

IP unicast reachability between VTEPs

EIGRP as an example. The goal is just connectivity between Loopback interfaces:

feature eigrp
router eigrp 100
  address-family ipv4 unicast
    autonomous-system 100

interface Ethernet1/1
  no switchport
  ip address
  ip router eigrp 100
  no shutdown

interface Ethernet1/2
  no switchport
  ip address
  ip router eigrp 100
  no shutdown

interface loopback0
  ip address
  ip router eigrp 100

PIM BIDIR reachability between VTEPs

Configure PIM first, check that is working and than enable BIDIR. Configure PIM with rp-address (NXOS3) on all Nexus devices:

conf t
feature pim

ip pim rp-address group-list

interface loopback0
  ip pim sparse-mode

interface Ethernet1/1
  ip pim sparse-mode

interface Ethernet1/2
  ip pim sparse-mode

How to check multicast receiving:

On one Leaf Nexus Loopback interface configure join-group:

interface loopback0
  ip igmp join-group

Check (*, ) in the RP pim routes => receiver is ok:

How to check multicast sending:

On the second Leaf Nexus run “ping multicast interface e1/1

NXOS2# ping multicast interface e1/1
PING ( 56 data bytes
64 bytes from icmp\_seq=0 ttl=254 time=6.742 ms
64 bytes from icmp\_seq=1 ttl=254 time=10.17 ms
64 bytes from icmp\_seq=2 ttl=254 time=13.882 ms
64 bytes from icmp\_seq=3 ttl=254 time=11.793 ms
64 bytes from icmp\_seq=4 ttl=254 time=8.43 ms

--- ping multicast statistics ---
5 packets transmitted,
From member 5 packets received, 0.00% packet loss
--- in total, 1 group member responded ---

Check on the RP pim routes. There should be 2 entries: (*, ) and for Leaf NXOS2 (, - (S,G) - sender is ok:

To enable bidirectional (PIM BIDIR) - configure switches with the following command (just add bidir):

ip pim rp-address group-list bidir

Do the same test with join-group and on the rp there should be just one route with (*,G) => bidirectional is ok:

Why BIDIR? It will be used for BUM traffic (Broadcast, Unknown-Unicast and Multicast traffic) - like ARP

Enable features:

conf t
   feature vn-segment-vlan-based
   feature nv overlay

Mapping VLAN to VXLAN

conf t
vlan 10
  vn-segment 101010

Create Network Virtualization Edge (VNE) interface

conf t
interface nve1
  source-interface loopback0
  member vni 101010

There is communication between SW5 and SW6 over VXLAN:

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Success rate is 100 percent (5/5), round-trip min/avg/max = 14/15/16 ms

A Classic Remote VTEP Discovery from here:

  1. End station sends Address Resolution Protocol (ARP) packet for remote end station.
  2. Packet reaches to VTEP-A and since VTEP-A does not know about the VTEP-B, it encapsulates the packet inside VXLAN header. It puts the multicast IP address as the destination IP address. Since the same multicast address is used by all VTEPs, all joins the same multicast group.
  3. This packet reaches to all VTEP and is decapsulated, in this way remote VTEP learns about the other VTEP. Since the decapsulated VTEP has the VNID, it is forwarded in the VLAN that has the same VNID configured.
  4. Now, remote end sends the ARP reply packet and it reaches to VTEP-B, since now VTEP-B knows about VTEP-A it again encapsulates the orginal frame but now the destination IP address is of VTEP-B and it is the unicast IP address.
  5. ARP reply reaches to VTEP-A and now VTEP-A gets to know about VTEP-B it forms the neighbor relationship with VTEP-B.

BUM traffic


show mac address-table
show nve peer
show nve vni