In this post we will look at the forwarding constructs in NX-OS in the context of VXLAN and EVPN. Having knowledge of the forwarding constructs helps both with understanding of the protocols, but also to assist in troubleshooting. BRKDCN-3040 from Cisco Live has a nice overview of the components involved:
There are components that are platform independent (PI) and platform dependent (PD). Below I’ll explain what each component does:
- ARP – Information from ARP requests/responses is needed to build adjacencies. The information learned from ARP is used to populate IP address field in RT2 and hence also to populate the ARP suppression cache.
- IPv6 ND – ND fills the role of ARP, but for IPv6.
- Adjacency Manager – Resolves directly attached hosts MAC addresses.
- Host Mobility Manager – Tracks the endpoints and their movements.
- L2FM – The Layer2 Forwarding Manager. A platform dependent component that programs ASICs for L2 forwarding. Keeps track of MAC addresses, their placement and moves, and synchronizes this information across ASICS, line cards, and vPC peers when vPC is in use.
- MFDM – Multicast Forwarding Database Manager. A platform dependent component that programs ASICs with information to perform multicast forwarding.
- L2RIB – The component that handles L2 “routing”. It has information about MAC addresses, IP addresses, and what VNI they belong to.
- MRIB – The multicast routing table. It’s a platform independent component. It has the mroutes, just like the URIB has unicast routes.
- URIB – The unicast routing table. Essentially what you see with “show ip route”. It’s platform independent and does not for example program the ASICs.
- U6RIB – Same as URIB but for IPv6.
- BGP – Populates BGP RIB based on information learned and creates and processes updates.
Now let’s take a look of the process from learning a MAC address to advertising it into BGP. The first component that learns of the MAC is L2FM:
Leaf1# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 10 0050.56ad.040e dynamic NA F F nve1(203.0.113.4)
C 10 0050.56ad.7d68 dynamic NA F F nve1(203.0.113.4)
* 10 0050.56ad.8506 dynamic NA F F Eth1/3
G 10 00ad.e688.1b08 static - F F sup-eth1(R)
Leaf1# show system internal l2fm event-history debugs | i 0050.56ad.8506
2024-01-31T05:54:41.600681000+00:00 [M 27] [l2fm] E_DEBUG [l2fm_l2rib_add_delete_local_mac_routes:2475] To L2RIB: topo-id: 10, macaddr: 0050.56ad.8506, nhifindx: 0x1a000400 peer_addr 0x1a000400
This shows the MAC being learned locally by L2FM and then added to L2RIB. This “route” can then be seen in the L2RIB:
Leaf1# show l2route evpn mac evi 10 Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending (S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override (Asy):Asymmetric (Gw):Gateway (Pf):Permanently-Frozen, (Orp): Orphan (PipOrp): Directly connected Orphan to PIP based vPC BGW (PipPeerOrp): Orphan connected to peer of PIP based vPC BGW Topology Mac Address Prod Flags Seq No Next-Hops ----------- -------------- ------ ------------- ---------- --------------------------------------------------------- 10 0050.56ad.040e BGP Rcv 0 203.0.113.4 (Label: 10000) 10 0050.56ad.7d68 BGP SplRcv 0 203.0.113.4 (Label: 10000) 10 0050.56ad.8506 Local L, 2 Eth1/3 Leaf1# show system internal l2rib event-history mac | i 0050.56ad.8506 2024-01-31T05:54:41.604556000+00:00 [M 27] [l2rib] E_DEBUG [l2rib_obj_mac_route_create:3186] (10,0050.56ad.8506,3): MAC route created with seqNum: 2, flags: L, (),
The route is then added to BGP:
Leaf1# show bgp l2vpn evpn vni-id 10000
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 6409, Local Router ID is 192.0.2.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.0.2.3:32777 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[0050.56ad.040e]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
203.0.113.1 100 32768 i
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[32]:[198.51.100.44]/272
203.0.113.4 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272
203.0.113.1 100 32768 i
*>l[3]:[0]:[32]:[203.0.113.1]/88
203.0.113.1 100 32768 i
*>i[3]:[0]:[32]:[203.0.113.2]/88
203.0.113.2 100 0 i
*>i[3]:[0]:[32]:[203.0.113.3]/88
203.0.113.3 100 0 i
*>i[3]:[0]:[32]:[203.0.113.4]/88
203.0.113.4 100 0 i
Leaf1# show bgp internal event-history events | i 0050.56ad.8506
2024-01-31T05:54:41.629287000+00:00 [M 27] [bgp] E_DEBUG [bgp_l2vpn_evpn_process_one_prefix:8912] (default) RIB: [L2VPN EVPN] (EVI L2-10000) add prefix 192.0.2.3:32777:[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:
[198.51.100.11] OK, 3 local paths in BRIB
This means that the path from learning of a MAC locally to advertising it looks like this:
The process above shows how the L2 information (MAC) is populated, but what about L3 (IP)? The first step is to learn via ARP:
Leaf1# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 1
Address Age MAC Address Interface Flags
198.51.100.11 00:04:24 0050.56ad.8506 Vlan10
This will then populate Adjacency Manager:
Leaf1# show forwarding vrf Tenant1 adjacency slot 1 ======= IPv4 adjacency information next-hop rewrite info interface -------------- --------------- ------------- 198.51.100.11 0050.56ad.8506 Vlan10
AM adds it to HMM:
Leaf1# show fabric forwarding ip local-host-db vrf Tenant1
HMM host IPv4 routing table information for VRF Tenant1
Status: *-valid, x-deleted, D-Duplicate, DF-Duplicate and frozen,
c-cleaned in 00:01:55
Host MAC Address SVI Flags Physical Interface
* 198.51.100.11/32 0050.56ad.8506 Vlan10 0x420201 Ethernet1/3
This is then added to the L2RIB:
Leaf1# show l2route evpn mac-ip evi 10 Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear (Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan (Asy):Asymmetric (Gw):Gateway (Piporp): Directly connected Orphan to PIP based vPC BGW (Pipporp): Orphan connected to peer of PIP based vPC BGW Topology Mac Address Host IP Prod Flags Seq No Next-Hops ----------- -------------- --------------------------------------- ------ ---------- ---------- --------------------------------------------------------- 10 0050.56ad.8506 198.51.100.11 HMM L, 2 Local 10 0050.56ad.7d68 198.51.100.44 BGP -- 0 203.0.113.4 (Label: 10000)
From the L2RIB, it gets added to BGP:
Leaf1# show bgp l2vpn evpn 198.51.100.11
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777 (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 517
Paths: (1 available, best #1)
Flags: (0x000302) (high32 00000000) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
203.0.113.1 (metric 0) from 0.0.0.0 (192.0.2.3)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:2
Router MAC:00ad.e688.1b08
Path-id 1 advertised to peers:
192.0.2.11 192.0.2.12
The entire process shown graphically:
In this post we learned how the MAC and IP portion of RT2 is populated. You learned what forwarding constructs are used in NX-OS and a couple of handy commands to verify each step. See you in the next post!
NX-OS Forwarding Constructs For VXLAN/EVPN