Nexus9000v and “Missing” Routes

I have built my lab for VXLAN on the Nexus9300v platform. Since I have a leaf and spine topology, there are ECMP routes towards the spines for the other leafs’ loopbacks. When performing labs though, I noticed that I didn’t have any ECMP routes in the forwarding table (FIB). They are in the RIB, though:

Leaf1# show ip route 203.0.113.4
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

203.0.113.4/32, ubest/mbest: 2/0
    *via 192.0.2.1, Eth1/1, [110/81], 1w0d, ospf-UNDERLAY, intra
    *via 192.0.2.2, Eth1/2, [110/81], 1w0d, ospf-UNDERLAY, intra

There is only one entry in the FIB, though:

Leaf1# show forwarding route 203.0.113.4?
  A.B.C.D      Display single longest match route
  A.B.C.D/LEN  Display single exact match route

Leaf1# show forwarding route 203.0.113.4/32

slot  1
=======


IPv4 routes for table default/base

------------------+-----------------------------------------+----------------------+-----------------+-----------------
Prefix            | Next-hop                                | Interface            | Labels          | Partial Install 
------------------+-----------------------------------------+----------------------+-----------------+-----------------
203.0.113.4/32       192.0.2.1                                 Ethernet1/1         

This seemed strange to me and I was concerned that maybe something was broken in my lab. I involved my dear friend Peter Palúch whom has extensive experience of routing and Nexus platform and also a history of being in Cisco TAC. We ran some commands that you would run on a HW platform to see if there were any inconsistencies and there were. That turned out to be a red herring, though, as a virtual platform works differently than a physical one. Peter was able to confirm the expected behavior internally at Cisco.

The Nexus9000v platform has virtual line card(s) and a supervisor module as can be seen below:

Leaf1# show module
Mod Ports             Module-Type                      Model           Status
--- ----- ------------------------------------- --------------------- ---------
1    64   Nexus 9000v 64 port Ethernet Module   N9K-X9364v            ok        
27   0    Virtual Supervisor Module             N9K-vSUP              active *  

Mod  Sw                       Hw    Slot
---  ----------------------- ------ ----
1    10.2(5)                  0.0    LC1 
27   NA                       0.0    SUP1


Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-ad-e6-88-01-01 to 00-ad-e6-88-01-40  9106JDPSGTE
27   00-ad-e6-88-1b-01 to 00-ad-e6-88-1b-12  94IVPKGHMXY

Mod  Online Diag Status
---  ------------------
1    Pass
27   Pass

* this terminal session 

On a HW platform, the expectation would be that all forwarding is done on the line card unless its packets destined to the route processor (RP), such as control plane packets (OSPF, BGP, PIM, STP), or packets destined for the device itself such as ICMP towards an interface, ARP, etc. This is not however how the virtual platform works. Implementing all forwarding features on the line card would be more difficult than simply having the RP take care of the forwarding. ALL packets in the Nexus9000v will be handled by the RP. What does this mean? This means that you can’t use show forwarding route to verify how packets are being routed but rather need to use show ip route. To verify that ECMP is working, first check the route:

Leaf1# show ip route 203.0.113.4/32 detail 
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

203.0.113.4/32, ubest/mbest: 2/0
    Extended Community: 0x08 18 00 00 00 00 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 00
    *via 192.0.2.1, Eth1/1, [110/81], 1w0d, ospf-UNDERLAY, intra
    *via 192.0.2.2, Eth1/2, [110/81], 1w0d, ospf-UNDERLAY, intra

Then to see if there is ECMP, I am sending 1000 ICMP Echo packets from Leaf1, Leaf2, and Leaf3 towards Leaf4 with ping 203.0.113.4 source-interface lo1 count 1000. First, let’s verify packet counters on Leaf4 towards spine. I cleared the counters first:

Leaf4# clear counters interface eth1/1
Leaf4# clear counters interface eth1/2

Leaf4# show int eth1/1 | i "input packet"
    15 input packets  1772 bytes
Leaf4# show int eth1/2 | i "input packet"
    25 input packets  3570 bytes

After sending the ICMP echo packets, verify again:

Leaf4# show int eth1/1 | i "input packet"
    2070 input packets  205498 bytes
Leaf4# show int eth1/2 | i "input packet"
    1072 input packets  107801 bytes

Based on counters it seems that two leafs send packets towards Spine1 and one leaf towards Spine2. Remember, ECMP does not mean load balancing, only load sharing. There could even be situations where all packets are coming via one spine. This is based on the hashing algorithm for ECMP. Normally it will check 5-tuple such as source IP, destination IP, protocol, source port, and destination port. Therefore, testing with ICMP may not be enough and you may need to send traffic towards different IPs and ports.

It was confusing at first to not see routes in the forwarding table. I now know thanks to Peter that this is the expected behavior. Thanks for learning with me!