In this post we will look at the forwarding constructs in NX-OS in the context of VXLAN and EVPN. Having knowledge of the forwarding constructs helps both with understanding of the protocols, but also to assist in troubleshooting. BRKDCN-3040 from Cisco Live has a nice overview of the components involved:

There are components that are platform independent (PI) and platform dependent (PD). Below I’ll explain what each component does:

  • ARP – Information from ARP requests/responses is needed to build adjacencies. The information learned from ARP is used to populate IP address field in RT2 and hence also to populate the ARP suppression cache.
  • IPv6 ND – ND fills the role of ARP, but for IPv6.
  • Adjacency Manager – Resolves directly attached hosts MAC addresses.
  • Host Mobility Manager – Tracks the endpoints and their movements.
  • L2FM – The Layer2 Forwarding Manager. A platform dependent component that programs ASICs for L2 forwarding. Keeps track of MAC addresses, their placement and moves, and synchronizes this information across ASICS, line cards, and vPC peers when vPC is in use.
  • MFDM – Multicast Forwarding Database Manager. A platform dependent component that programs ASICs with information to perform multicast forwarding.
  • L2RIB – The component that handles L2 “routing”. It has information about MAC addresses, IP addresses, and what VNI they belong to.
  • MRIB – The multicast routing table. It’s a platform independent component. It has the mroutes, just like the URIB has unicast routes.
  • URIB – The unicast routing table. Essentially what you see with “show ip route”. It’s platform independent and does not for example program the ASICs.
  • U6RIB – Same as URIB but for IPv6.
  • BGP – Populates BGP RIB based on information learned and creates and processes updates.

Now let’s take a look of the process from learning a MAC address to advertising it into BGP. The first component that learns of the MAC is L2FM:

Leaf1# show mac address-table vlan 10
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
C   10     0050.56ad.040e   dynamic  NA         F      F    nve1(
C   10     0050.56ad.7d68   dynamic  NA         F      F    nve1(
*   10     0050.56ad.8506   dynamic  NA         F      F    Eth1/3
G   10     00ad.e688.1b08   static   -         F      F    sup-eth1(R)

Leaf1# show system internal l2fm event-history debugs | i 0050.56ad.8506
2024-01-31T05:54:41.600681000+00:00 [M 27] [l2fm] E_DEBUG [l2fm_l2rib_add_delete_local_mac_routes:2475] To L2RIB: topo-id: 10, macaddr: 0050.56ad.8506, nhifindx: 0x1a000400 peer_addr 0x1a000400

This shows the MAC being learned locally by L2FM and then added to L2RIB. This “route” can then be seen in the L2RIB:

Leaf1# show l2route evpn mac evi 10

Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote 
(Dup):Duplicate (Spl):Split (Rcv):Recv (AD):Auto-Delete (D):Del Pending
(S):Stale (C):Clear, (Ps):Peer Sync (O):Re-Originated (Nho):NH-Override
(Asy):Asymmetric (Gw):Gateway
(Pf):Permanently-Frozen, (Orp): Orphan

(PipOrp): Directly connected Orphan to PIP based vPC BGW 
(PipPeerOrp): Orphan connected to peer of PIP based vPC BGW 
Topology    Mac Address    Prod   Flags         Seq No     Next-Hops                              
----------- -------------- ------ ------------- ---------- ---------------------------------------------------------
10          0050.56ad.040e BGP    Rcv           0 (Label: 10000)                               
10          0050.56ad.7d68 BGP    SplRcv        0 (Label: 10000)                               
10          0050.56ad.8506 Local  L,            2          Eth1/3

Leaf1# show system internal l2rib event-history mac | i 0050.56ad.8506
2024-01-31T05:54:41.604556000+00:00 [M 27] [l2rib] E_DEBUG [l2rib_obj_mac_route_create:3186] (10,0050.56ad.8506,3): MAC route created with seqNum: 2, flags: L, (),                    

The route is then added to BGP:

Leaf1# show bgp l2vpn evpn vni-id 10000
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 6409, Local Router ID is
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher:    (L2VNI 10000)
                                   100          0 i
                                   100          0 i
                                   100      32768 i
                                   100          0 i
                                   100      32768 i
                                   100      32768 i
                                   100          0 i
                                   100          0 i
                                   100          0 i

Leaf1# show bgp internal event-history events | i 0050.56ad.8506
2024-01-31T05:54:41.629287000+00:00 [M 27] [bgp] E_DEBUG [bgp_l2vpn_evpn_process_one_prefix:8912] (default) RIB: [L2VPN EVPN] (EVI L2-10000) add prefix[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:
[] OK, 3 local paths in BRIB

This means that the path from learning of a MAC locally to advertising it looks like this:

The process above shows how the L2 information (MAC) is populated, but what about L3 (IP)? The first step is to learn via ARP:

Leaf1# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 1
Address         Age       MAC Address     Interface       Flags   00:04:24  0050.56ad.8506  Vlan10               

This will then populate Adjacency Manager:

Leaf1# show forwarding vrf Tenant1 adjacency 

slot  1

IPv4 adjacency information

next-hop         rewrite info    interface  
-------------- --------------- -------------   0050.56ad.8506 Vlan10        

AM adds it to HMM:

Leaf1# show fabric forwarding ip local-host-db vrf Tenant1

HMM host IPv4 routing table information for VRF Tenant1
Status: *-valid, x-deleted, D-Duplicate, DF-Duplicate and frozen, 
        c-cleaned in 00:01:55

    Host                 MAC Address        SVI        Flags      Physical Interface
*     0050.56ad.8506     Vlan10     0x420201   Ethernet1/3

This is then added to the L2RIB:

Leaf1# show l2route evpn mac-ip evi 10
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote 
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan (Asy):Asymmetric (Gw):Gateway
(Piporp): Directly connected Orphan to PIP based vPC BGW 
(Pipporp): Orphan connected to peer of PIP based vPC BGW 
Topology    Mac Address    Host IP                                 Prod   Flags         Seq No     Next-Hops                              
----------- -------------- --------------------------------------- ------ ---------- ---------- ---------------------------------------------------------
10          0050.56ad.8506                           HMM    L,            2         Local                                  
10          0050.56ad.7d68                           BGP    --            0 (Label: 10000)        

From the L2RIB, it gets added to BGP:

Leaf1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher:    (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[]/272, version 517
Paths: (1 available, best #1)
Flags: (0x000302) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated (metric 0) from (
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:2
          Router MAC:00ad.e688.1b08

  Path-id 1 advertised to peers:

The entire process shown graphically:

In this post we learned how the MAC and IP portion of RT2 is populated. You learned what forwarding constructs are used in NX-OS and a couple of handy commands to verify each step. See you in the next post!

NX-OS Forwarding Constructs For VXLAN/EVPN
Tagged on:                     

Leave a Reply

Your email address will not be published. Required fields are marked *