Cisco vPC in VXLAN/EVPN Network – Part 3 – Verifying Connectivity

The following topology is used:

We want to verify connectivity and traffic flow towards:

• Gateway of Server3.
• Server1.
• Server2.
• Server4.

Let’s start with the gateway. The gateway is at 10.0.0.1 and has a MAC address of 0001.0001.0001:

server3:~$ ip neighbor | grep 10.0.0.1
10.0.0.1 dev bond0 lladdr 00:01:00:01:00:01 STALE

This is an anycast gateway MAC. When initiating a ping towards 10.0.0.1, it can go to either Leaf1 or Leaf2. I will run Ethanalyzer on the switches to confirm which one is receiving the ICMP Echo Request:

Leaf1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'

Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'

Then initiate ping from Server3:

server3:~$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=1.19 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=1.29 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.192/1.242/1.292/0.050 ms

Packets were received on Leaf2:

Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'
2   285 2024-04-07 09:03:32.459007178    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006e, seq=1/256, ttl=64
  286 2024-04-07 09:03:32.459685552     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006e, seq=1/256, ttl=255 (request in 285)
4   294 2024-04-07 09:03:33.459687481    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006e, seq=2/512, ttl=64
  295 2024-04-07 09:03:33.460371598     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006e, seq=2/512, ttl=255 (request in 294)

The next round of pings were also received on Leaf2:

Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'
2   285 2024-04-07 09:03:32.459007178    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006e, seq=1/256, ttl=64
  286 2024-04-07 09:03:32.459685552     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006e, seq=1/256, ttl=255 (request in 285)
4   294 2024-04-07 09:03:33.459687481    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006e, seq=2/512, ttl=64
  295 2024-04-07 09:03:33.460371598     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006e, seq=2/512, ttl=255 (request in 294)
4   961 2024-04-07 09:04:13.222153742    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006f, seq=1/256, ttl=64
  962 2024-04-07 09:04:13.222872197     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006f, seq=1/256, ttl=255 (request in 961)
6   978 2024-04-07 09:04:14.223805947    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x006f, seq=2/512, ttl=64
  979 2024-04-07 09:04:14.224429989     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x006f, seq=2/512, ttl=255 (request in 978)

Then they were received on Leaf1:

Leaf1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'
2  1277 2024-04-07 09:03:21.894839613    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x0070, seq=1/256, ttl=64
 1278 2024-04-07 09:03:21.895436908     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x0070, seq=1/256, ttl=255 (request in 1277)
2  1282 2024-04-07 09:03:22.896323401    10.0.0.33 → 10.0.0.1     ICMP 98 Echo (ping) request  id=0x0070, seq=2/512, ttl=64
 1283 2024-04-07 09:03:22.896906682     10.0.0.1 → 10.0.0.33    ICMP 98 Echo (ping) reply    id=0x0070, seq=2/512, ttl=255 (request in 1282)

This is the effect of hashing on Server3. As anycast gateway is used, both switches can respond.

Now let’s ping Server2, which is connected to Leaf2:

server3:~$ ping 10.0.0.22
PING 10.0.0.22 (10.0.0.22) 56(84) bytes of data.
64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.702 ms
64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.847 ms
^C
--- 10.0.0.22 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 0.702/0.774/0.847/0.072 ms

There’s two paths the packet can take here:

• Server3 sends ICMP towards Leaf2 and Leaf2 switches the frame locally.
• Server3 sends ICMP towards Leaf1 and Leaf1 switches it across the vPC peer link.

This is shown in the diagram below:

Leaf2 has the MAC address of Server2 locally:

Leaf2# show mac add add 0050.56ad.f48d
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   20     0050.56ad.f48d   dynamic  NA         F      F    Eth1/3

Leaf1 has it across the vPC peer link:

Leaf1# show mac add add 0050.56ad.f48d
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+   20     0050.56ad.f48d   dynamic  NA         F      F    vPC Peer-Link

Next, let’s try connectivity towards Server1:

server3:~$ ping 198.51.100.11
PING 198.51.100.11 (198.51.100.11) 56(84) bytes of data.
64 bytes from 198.51.100.11: icmp_seq=1 ttl=63 time=2.33 ms
64 bytes from 198.51.100.11: icmp_seq=2 ttl=63 time=2.48 ms
^C
--- 198.51.100.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.334/2.407/2.481/0.073 ms

Once again, there are two paths here:

• Server3 sends ICMP towards Leaf1 and Leaf1 routes the packet and forwards it to Server1.
• Server3 sends ICMP towards Leaf2 and Leaf2 routes the packet and forwards it across the vPC peer link.

This is shown in the diagram below:

There is a potential pitfall here, which I will describe in the next section, but first let’s look at when things are working. Leaf1 has a route locally:

Leaf1# show ip route 198.51.100.11 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

198.51.100.11/32, ubest/mbest: 1/0, attached
    *via 198.51.100.11, Vlan10, [190/0], 00:58:00, hmm

It also has the MAC address for 198.51.100.11 in the ARP cache:

Leaf1# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 3
Address         Age       MAC Address     Interface       Flags
10.0.0.22       00:04:01  0050.56ad.f48d  Vlan20          +        
10.0.0.33       00:00:26  828b.b6cf.badf  Vlan20                   
198.51.100.11   00:02:05  0050.56ad.8506  Vlan10     

Leaf2 also has a local route:

Leaf2# show ip route 198.51.100.11 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

198.51.100.11/32, ubest/mbest: 1/0, attached
    *via 198.51.100.11, Vlan10, [190/0], 00:05:20, hmm

Let’s check the ARP cache:

Leaf2# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 3
Address         Age       MAC Address     Interface       Flags
10.0.0.22       00:01:09  0050.56ad.f48d  Vlan20                   
10.0.0.33       00:02:34  828b.b6cf.badf  Vlan20          +        
198.51.100.11   00:04:14  0050.56ad.8506  Vlan10          +  

Notice the plus sign in the Flags column which indicates that this ARP entry has been learned via CFS. This entry is reachable via Po12:

Leaf2# show fabric forwarding ip local-host-db vrf Tenant1

HMM host IPv4 routing table information for VRF Tenant1
Status: *-valid, x-deleted, D-Duplicate, DF-Duplicate and frozen, 
        c-cleaned in 00:08:01

    Host                 MAC Address        SVI        Flags      Physical Interface
*   10.0.0.22/32         0050.56ad.f48d     Vlan20     0x420201   Ethernet1/3
*   10.0.0.33/32         828b.b6cf.badf     Vlan20     0x420201   port-channel33
*   198.51.100.11/32     0050.56ad.8506     Vlan10     0x420201   port-channel12

Note that the entry is from HMM, while the entry for 198.51.100.44 is via BGP:

Leaf2# show l2route evpn mac-ip evi 10
Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote 
(Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear
(Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan (Asy):Asymmetric (Gw):Gateway
(Piporp): Directly connected Orphan to PIP based vPC BGW 
(Pipporp): Orphan connected to peer of PIP based vPC BGW 
Topology    Mac Address    Host IP                                 Prod   Flags         Seq No     Next-Hops                              
----------- -------------- --------------------------------------- ------ ---------- ---------- ---------------------------------------------------------
10          0050.56ad.8506 198.51.100.11                           HMM    L,            0         Local                                  
10          0050.56ad.7d68 198.51.100.44                           BGP    --            0         203.0.113.4 (Label: 10000)   

The ARP suppression cache is also populated:

Leaf2# show ip arp suppression-cache detail

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface
       PS - Added via L2RIB, Peer Sync
       RO - Dervied from L2RIB Peer Sync Entry

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags    Remote Vtep Addrs

198.51.100.11   00:01:18 0050.56ad.8506   10 port-channel12      L
198.51.100.44      1d15h 0050.56ad.7d68   10 (null)              R        203.0.113.4 
10.0.0.33       00:00:09 828b.b6cf.badf   20 port-channel33      L
10.0.0.22       00:00:09 0050.56ad.f48d   20 Ethernet1/3         L

Finally, let’s ping Server4, which is connected to Leaf4:

server3:~$ ping 198.51.100.44
PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data.
64 bytes from 198.51.100.44: icmp_seq=1 ttl=62 time=21.3 ms
64 bytes from 198.51.100.44: icmp_seq=2 ttl=62 time=5.92 ms
^C
--- 198.51.100.44 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 5.924/13.631/21.338/7.707 ms

There are several paths the packet can take:

• Server3 sends ICMP towards Leaf1 and Leaf1 routes the packet towards Leaf4.
• Server3 sends ICMP towards Leaf2 and Leaf2 routes the packet towards Leaf4.

This is shown in the diagram below:

Note that Leaf4 can send the packet both towards Leaf1 and Leaf2 based on ECMP for the Anycast VTEP.

Leaf1 and Leaf2 both have a route to 198.51.100.44 via Leaf4:

Leaf1# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

198.51.100.44/32, ubest/mbest: 1/0
    *via 203.0.113.4%default, [200/0], 5d19h, bgp-65000, internal, tag 65000, segid: 10001 tunnelid: 0xcb007104 encap: VXLAN

Leaf2# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

198.51.100.44/32, ubest/mbest: 1/0
    *via 203.0.113.4%default, [200/0], 5d20h, bgp-65000, internal, tag 65000, segid: 10001 tunnelid: 0xcb007104 encap: VXLAN

The route is from EVPN:

Leaf1# show bgp l2vpn evpn 198.51.100.44
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.6:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.7d68]:[32]:[198.51.100.44]/272, version 8
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: Tenant1 L3-10001 L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.4 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
      Originator: 192.0.2.6 Cluster list: 192.0.2.1 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.4 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
      Originator: 192.0.2.6 Cluster list: 192.0.2.2 

  Path-id 1 not advertised to any peer


Leaf2# show bgp l2vpn evpn 198.51.100.44
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.6:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.7d68]:[32]:[198.51.100.44]/272, version 8
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: Tenant1 L3-10001 L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.4 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
      Originator: 192.0.2.6 Cluster list: 192.0.2.1 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.4 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
      Originator: 192.0.2.6 Cluster list: 192.0.2.2 

  Path-id 1 not advertised to any peer

Leaf4 has a local route for 198.51.100.44:

Leaf4# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

198.51.100.44/32, ubest/mbest: 1/0, attached
    *via 198.51.100.44, Vlan10, [190/0], 9w2d, hmm

It is also known in the ARP cache:

Leaf4# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 1
Address         Age       MAC Address     Interface       Flags
198.51.100.44   00:04:01  0050.56ad.7d68  Vlan10           

With connectivity verified, in the next blog post we’ll take a brief look at fabric peering.