Two control planes for the VXLAN technology:

  • Multicast control plane (flood and learn)
  • MP-BGP EVPN control plane

Virtual Extensible Local Area Networks (VXLANs) allows to extend reachability of a VLAN within a data center over Layer 3.

Every VTEP with specific VXLAN and certain VNI will join the same multicast group. To learn remote MAC addresses, the VTEP will use conversational MAC address learning technique: learn only actively speaking MAC addresses.

note: Always Read Guidelines and Limitation for specific hardware
Cisco Nexus 5600 Series NX-OS Layer 2 Switching Configuration Guide, Release 7.x

  • Set switching mode to store-and-forwarding
  • IP unicast reachability between VTEPs
  • PIM BIDIR reachability between VTEPs
  • Enable features “vn-segment-vlan-based” and “nv overlay”

Store-and-forwarding

conf t
   hardware ethernet store-and-fwd-switching
copy run start
reboot

IP unicast reachability between VTEPs

EIGRP as an example. The goal is just connectivity between Loopback interfaces:

feature eigrp
!
router eigrp 100
  address-family ipv4 unicast
    autonomous-system 100
    router-id 1.1.1.1

interface Ethernet1/1
  no switchport
  ip address 172.16.12.1/30
  ip router eigrp 100
  no shutdown

interface Ethernet1/2
  no switchport
  ip address 172.16.13.1/24
  ip router eigrp 100
  no shutdown

interface loopback0
  ip address 1.1.1.1/32
  ip router eigrp 100

PIM BIDIR reachability between VTEPs

Configure PIM first, check that is working and than enable BIDIR. Configure PIM with rp-address 1.1.1.3 (NXOS3) on all Nexus devices:

conf t
feature pim

ip pim rp-address 1.1.1.3 group-list 224.0.0.0/4

interface loopback0
  ip pim sparse-mode

interface Ethernet1/1
  ip pim sparse-mode

interface Ethernet1/2
  ip pim sparse-mode

How to check multicast receiving:

On one Leaf Nexus Loopback interface configure join-group:

NXOS1:
interface loopback0
  ip igmp join-group 227.7.7.7

Check (*, 227.7.7.7 ) in the RP pim routes => receiver is ok:

How to check multicast sending:

On the second Leaf Nexus run “ping multicast 227.7.7.7 interface e1/1”

NXOS2# ping multicast 227.7.7.7 interface e1/1
PING 227.7.7.7 (227.7.7.7): 56 data bytes
64 bytes from 172.16.12.1: icmp\_seq=0 ttl=254 time=6.742 ms
64 bytes from 172.16.12.1: icmp\_seq=1 ttl=254 time=10.17 ms
64 bytes from 172.16.12.1: icmp\_seq=2 ttl=254 time=13.882 ms
64 bytes from 172.16.12.1: icmp\_seq=3 ttl=254 time=11.793 ms
64 bytes from 172.16.12.1: icmp\_seq=4 ttl=254 time=8.43 ms

--- 227.7.7.7 ping multicast statistics ---
5 packets transmitted,
From member 172.16.12.1: 5 packets received, 0.00% packet loss
--- in total, 1 group member responded ---

Check on the RP pim routes. There should be 2 entries: (*, 227.7.7.7 ) and for Leaf NXOS2 (172.16.12.2/32, 227.7.7.7/32) - (S,G) - sender is ok:

To enable bidirectional (PIM BIDIR) - configure switches with the following command (just add bidir):

ip pim rp-address 1.1.1.3 group-list 224.0.0.0/4 bidir

Do the same test with join-group 228.8.8.8 and on the rp there should be just one route with (*,G) => bidirectional is ok:

Why BIDIR? It will be used for BUM traffic (Broadcast, Unknown-Unicast and Multicast traffic) - like ARP

Enable features:

conf t
   feature vn-segment-vlan-based
   feature nv overlay

Mapping VLAN to VXLAN

conf t
vlan 10
  vn-segment 101010

Create Network Virtualization Edge (VNE) interface

conf t
interface nve1
  source-interface loopback0
  member vni 101010
    mcast-group 228.7.7.7

There is communication between SW5 and SW6 over VXLAN:

SW5#ping 10.10.10.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.10.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 14/15/16 ms

A Classic Remote VTEP Discovery from here:

  1. End station sends Address Resolution Protocol (ARP) packet for remote end station.
  2. Packet reaches to VTEP-A and since VTEP-A does not know about the VTEP-B, it encapsulates the packet inside VXLAN header. It puts the multicast IP address as the destination IP address. Since the same multicast address is used by all VTEPs, all joins the same multicast group.
  3. This packet reaches to all VTEP and is decapsulated, in this way remote VTEP learns about the other VTEP. Since the decapsulated VTEP has the VNID, it is forwarded in the VLAN that has the same VNID configured.
  4. Now, remote end sends the ARP reply packet and it reaches to VTEP-B, since now VTEP-B knows about VTEP-A it again encapsulates the orginal frame but now the destination IP address is of VTEP-B and it is the unicast IP address.
  5. ARP reply reaches to VTEP-A and now VTEP-A gets to know about VTEP-B it forms the neighbor relationship with VTEP-B.

BUM traffic

Verifications

show mac address-table
show nve peer
show nve vni