nxos

By default: underlay - default VRF overlay - “tenant” VRF, hosts in VXLAN are isolated Border Leafs are used to connect the internal fabric to external networks. Not necessary a box, just configuration on the Leaf. It maintains the following routing control planes: MP-BGP L2VPN EVPN - inside VXLAN fabric “tenant” VRF BGP or IGP to external routes MP-BGP to BGP/IGP redistribution Main consideration: Border Leaf maintains all /32 host routes for all VRFs, but we need...

VxLAN and vPC Anycast VTEP Problem with VXLAN and vPC: in a vPC both vPC peers duplicate EVPN MAC/IP routes to spine RRs with other attributes equal, one vPC peer is always preferred for dual attached hosts (based on the normal BGP Best path selection) Result: egress traffic from vPC Member is load-balanced, but return ingress traffic is polarized Solution: Anycast VTEP address - Loopback 0 ip address secondary, the same on both vPC peers:...

EVPN Integrated Routing and Bridging (IRB) has two options: Asymmetric IRB (increased ARP cache and CAM table sizes and control plane scaling issue) Symmetric IRB Symmetric IRB Ingress VTEP does both L2 and L3 lookup Egress VTEp does both L3 and L2 lookup => Bridge - Route - Route - Bridge L3 VNI should be configured on all VTEPS, L2 VNIs only where local ports exist N5K1 Configuration fabric forwarding anycast-gateway-mac 1234....

EVPN Integrated Routing and Bridging (IRB) has two options: Asymmetric IRB Symmetric IRB Assymetric IRB Ingress VTEP does both L2 and L3 lookup Egress VTEP does L2 lookup only => Bridge - Route - Bridge Pros: “easy” to configure - just copy/paste. Identical config with the only difference in SVI IP addresses. Cons: on the way back, traffic will be reversed => all VXLANs need to be configured on all VTEPs => increased ARP cache and CAM table sizes and control plane scaling issue => not very efficient....

Two control planes for the VXLAN technology: Multicast control plane (flood-and learn) MP-BGP EVPN control plane MP-BGP EVPN is a standard-based VXLAN control protocol, that provides remote VTEP discovery and MAC/ARP learning. Ethernet Virtual Private Network (EVPN) reduces flooding in the network and resolves scalability concerns. MP-BGP is used to exchange information between VTEPs Devices might be MP-iBGP EVPN peers or route reflectors, or MP External BGP (MP-eBGP) EVPN peers....

Two control planes for the VXLAN technology: Multicast control plane (flood and learn) MP-BGP EVPN control plane Virtual Extensible Local Area Networks (VXLANs) allows to extend reachability of a VLAN within a data center over Layer 3. Every VTEP with specific VXLAN and certain VNI will join the same multicast group. To learn remote MAC addresses, the VTEP will use conversational MAC address learning technique: learn only actively speaking MAC addresses....

VXLAN is a tunneling protocol that encapsulates Layer 2 Ethernet frames in Layer 3 UDP packets. Why VXLAN: VLAN Scalability - expands VLAN name space VLANs use 12 bit -4096 values VXLAN uses 24 bit - 16777216 values allows layer 2 multipathing no STP uses layer 3 ECMP over CLOS fabric (like FabricPath) allows for multi-tenancy separate of customer traffic over shared underlay fabric allows for overlapping layer 2 and layer 3 addresses (VLANs and IP are locally significant - could be VLAN 10 in one DC and VLAN20 in another DC, as long as the same subnet and VXLAN) CE - only one port is active vPC - can not scale out, only 2 distribution switches FabricPath - L2 only and there is no active control plane (legacy now, because of VXLAN) VXLAN - optimize the control plane (don’t send broadcast everywhere, not learning every possible MAC addresses) VXLAN Terminology Underlay Network - provides transport for VXLAN OSPF/EIGRP/IS-IS router fabric Overlay Network - uses the service provided by VXLAN VXLAN - Virtual eXtensivle LAN VNI / VNID - VXLAN Network Identifier (replaces the VLAN ID) VTEP - VXLAN Tunnel End Point box that performs VXLAN encap/decap hardware or software (Nexus 5600, N7K-M3, Nexus 1000v) VXLAN Segment - the resulting L2 overlay network VXLAN Gateway - device that forwars traffic between VXLANs NVE - Network Virtualization Edge logical representation of the VTEP NVE is the tunnel interface VXLAN Encapsulation VXLAN over UDP over IP Basic VXLAN Workflow Receive ARP from local host Find the remote VTEP multicast flood and learn ingress replication MP-BGP L2VPN EVPN Unicast encap frame to the VTEP throw away the VLAN replace it with the VNID

FabricPath (FP) is a L2 Routing = “MAC-in-MAC” Routing. FabricPath is Cisco proprietary and works in the same way as TRILL (Transparent Interconnection of Lots of Links) that is an IETF standard. FP: to remove STP from the topology vPC: only 2 switches FP: full mesh, partial mesh, triangle, square etc Components: Classical Ethernet (CE) regular ethernet with regular flooding, regular STP Leaf Switch connects CE domain to FP domain Spine Switch FP bacbone switch with all ports in the FP domain only FP Core Ports links on Leaf up to Spine or Spine to Spine ie the switchport mode fabricpath links CE Edge Ports links on Leaf connecting to regular CE domain ie NOT the switchport mode fabricpath links FabricPath Control Plane IS-IS for L2 Routing Goal is to compute SPT (Shortest Path Tree) between all FabricPath nodes Why IS-IS?...

vPC Orphan Ports Traffic from remote Orphan is allowed over Peer Link and exit via local Member Traffic from remote Member is allowed over Peer Link and exit via local Orphan -Orphans ports should be avoided at all costs because PL is a bottleneck of the system Ideal: vPC Peers only have vPC Member Ports and all downstream devices are dual attached vPC Consistency Checks Type 1 Global and Interface Consistency Check if global mismatch - vPC failing to form if interface mismatch - VLANs being suspended Type 2 Consistency Check if mismatch - log messaged but not vPC failure, but could be data plane failures Failure: vPC peer-link failure (link loss) Secondary waits for hold-timeout and keepalive timeouts trying to reach out to the Primary over Keep-alive link After timers expire if vPC Primary is alive: disable Member port on Secondary disable SVI on Secondary => Secondary is disabled => force all traffic to go over Primary if vPC Primary is dead: promote vPC Secondary to Operational Primary traffic over new vPC Primary if vPC Primary is alive: NXOS1(config)# int po50 NXOS1(config-if)# shutdown 2019 Oct 22 05:15:26 NXOS1 %$ VDC-1 %$ %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary....

FHRP acts as active/active forwarding over vPC: traffic received in vPC Member Port of FHRP Standby to FHRP Virtual MAC is not forwarded over Peer Link to Active FHRP - essentially HSRP Standby acts as HSRP Active peer-gateway allows to proxy not only virtual active MAC address but also to proxy physical primary MAC address (in case destination MAC address is an Active device, but not a control/management plane of the box itself) the goal - avoid using Peer Link for data plane and it should forward traffic to the upstream L3 router etc in general use GLBP for this behavior, but for vPC it is a default Nexus SVI configuration:...

The vPC Peer Link should never be blocking because this link carries important traffic such as the Cisco Fabric Services over Ethernet (CFSoE) Protocol. The peer link is always forwarding. STP from SW8 and SW9: STP from NXOS1 and NXOS2: In the correct design, the vPC Peer Link should be used only in case of failure. All links are up and active: The link between SW8 and NXOS2 is down:...

vPC Order of Operations IP connectivity for Peer Keepalive Enable vPC & LACP globally Create vPC domain define Peer Keepalive address configure vPC role priority (Optional) - lower priority => vPC primary switch. (default 32667) Establish Port Channel for vPC Peer link Verify vPC Consistency Parameters Disable vPC Member Port (optional but recommended) Configure vPC Member Ports Enable vPC Member Ports Make sure keepalive links is up and check IP reachability (mgmt0 could be used)...

Three Main Types of MCEC (Multi Chassis EtherChannel) C3750 Cross Stack Port Channels (StackWise) single control plane C6500 Virtual Switching System (VSS) single control plane via Virtual Switch Link (VSL) Nexus Virtaul Port Channel (vPC) separate control planes separate control plane protocol instances (STP/IGPs/BGP/FHRP) via a Peer Link (like VSS’s VSL) Each vPC peer has Peer Link to sync control plane between vPC peers (CAM/ARP/IGMP) uses CFSoE (Cisco Fabric Service over Ethernet) used to elect a vPC Primary and vPC Secondary Role normally not used for the data plane => much lower BW Peer Keepalive Link L3 link used as heartbeat in the control plane used to prevent active/active or “Split Brain” vPC roles not used in the vPC data plane could back to back or over routed infrastructure (vrf) vPC Member ports from downstream neighbor the vPC peers is one switch Note: VLANS on vPC Member ports must be added on the Peer link too...

Basic topology: Host Port-Channel: vPC Implementation problem - configuration must be synced between different control planes: config sync command Dual vPC or EvPC - Enhanced vPC - only N5K N7K

Nexus 2000 Series Fabric Extenders acts as a remote line card of N7K or N5K chassis. N2K FEX - ToR - Top of the Rack N5K/N7K - EoR - End of the Row Why? Solve the problem of wiring cables in the data center. Keep all cables inside the rack as much as possible Simplify the management and reduce number of management devices. All management performed on Parent Switch (management/upgrade etc) Limitations and Parent Switches No local switching inside the N2K FEX 5K as FEX’s Parent static pinning (ports on N2K pinned to the uplink port) vPC topologies FEX ports are L2 switchports 7K as FEX’s Parent not all line cards support FEX static pinning only FEX link must be Port-Channel FEX ports are L2 switchports or native L3 routed interfaces L2 FEX ports are STP “edge” ports run BPDUGuard -> not switch could be connected to the FEX port FEX configuration Enable FEX feature: on N5K:...