By default:
- underlay - default VRF
- overlay - “tenant” VRF, hosts in VXLAN are isolated
Border Leafs are used to connect the internal fabric to external networks. Not necessary a box, just configuration on the Leaf. It maintains the following routing control planes:
- MP-BGP L2VPN EVPN - inside VXLAN fabric
- “tenant” VRF BGP or IGP to external routes
- MP-BGP to BGP/IGP redistribution
Main consideration: Border Leaf maintains all /32 host routes for all VRFs, but we need
- aggregate (summarization during the redistribution) them
- use traffic engineering based on the longest match (in case there are multiple external routers)
VRF on External router:
- if there is overlapping addressing inside Tenant networks => VRF => VRF=Lite => some sort of nat is needed
- if all addresses are unique, it can mix them in the default routing table
Configuration
R1 (External router) Configuration
interface Loopback 0
ip address 77.77.77.77 255.255.255.255
interface Ethernet0/0
ip address 77.77.1.1 255.255.255.0
router bgp 777
bgp log-neighbor-changes
neighbor 77.77.1.8 remote-as 65001
address-family ipv4
network 77.77.77.77 mask 255.255.255.255
neighbor 77.77.1.8 activate
NXOS8 (Border Leaf) Configuration
interface Ethernet1/7
description R1 External Router
no switchport
** vrf member CUSTOMER1**
** ip address 77.77.1.8/24**
no shutdown
!
router bgp 1
log-neighbor-changes
address-family ipv4 unicast
neighbor 1.1.1.51
remote-as 1
update-source loopback0
address-family l2vpn evpn
send-community extended
neighbor 1.1.1.52
remote-as 1
update-source loopback0
address-family l2vpn evpn
send-community extended
vrf CUSTOMER1
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map DIRECT
** neighbor 77.77.1.1**
** remote-as 777**
** local-as 65001**
** address-family ipv4 unicast**
R1 receives all routes (/24 and all /32)
- redistribute direct route-map DIRECT - /24 subnets
- evpn routes - /32 routes
Note: Cisco made import of the VPNv4 prefixes from L2VPN EVPN into unicast implicit, so there is no “advertise l2vpn evpn” command under BGP vrf configuration, which was in NXOS v7 - it happens automatically.
The goal is to advertise the less specific route by using route-map, prefix-lists etc
NXOS8
conf t
ip prefix-list NO_HOST_SUBNETS seq 5 deny 0.0.0.0/0 ge 32
ip prefix-list NO_HOST_SUBNETS seq 10 permit 0.0.0.0/0 le 31
!
router bgp 1
vrf CUSTOMER1
neighbor 77.77.1.1
address-family ipv4 unicast
prefix-list NO_HOST_SUBNETS out
/32 routes have been filtered
We can use any IGP, but BGP is more typical for this design because:
- redistribution happens automatically
- easy to apply policy on BGP
- easier to filter routes and do aggregation
Multiple Exit Points
If there are multiple exit points and configure NXOS7 the same way as NXOS8:
NXOS7
interface Ethernet1/7
description R1 External Router
no switchport
vrf member CUSTOMER1**
ip address 77.77.1.7/24**
no shutdown
!
router bgp 1
log-neighbor-changes
neighbor 1.1.1.51
remote-as 1
update-source loopback0
address-family l2vpn evpn
send-community extended
neighbor 1.1.1.52
remote-as 1
update-source loopback0
address-family l2vpn evpn
send-community extended
vrf CUSTOMER1
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map DIRECT
neighbor 77.77.1.1**
remote-as 777**
local-as 65001**
address-family ipv4 unicast**
Traffic engineering: /32 more specific from NXOS7 and /24 from NXOS8, Will fail over if NXOS7 goes down
Here is a way to manage how traffic is leav****ing - using Local Preference.
LP200 on NXOS8 (more preferred) and default LP100 on NXOS7. It will be advertised to the RR, RR will prefer NXOS8 and reflects to other Leafs (they will see only one possible exit point)
NXOS8
route-map LOCAL_PREF permit 10**
set local-preference 200**
!
router bgp 1
vrf CUSTOMER1
neighbor 77.77.1.1
remote-as 777
local-as 65001
address-family ipv4 unicast
prefix-list NO_HOST_SUBNETS out
route-map LOCAL_PREF in**
Remote Leaf now using NXOS8 for 77.77.77.77
Very good guide: VXLAN Network with MP-BGP EVPN Control Plane Design Guide