The following topology is used:
We want to verify connectivity and traffic flow towards:
- Gateway of Server3.
- Server1.
- Server2.
- Server4.
Let’s start with the gateway. The gateway is at 10.0.0.1 and has a MAC address of 0001.0001.0001:
server3:~$ ip neighbor | grep 10.0.0.1 10.0.0.1 dev bond0 lladdr 00:01:00:01:00:01 STALE
This is an anycast gateway MAC. When initiating a ping towards 10.0.0.1, it can go to either Leaf1 or Leaf2. I will run Ethanalyzer on the switches to confirm which one is receiving the ICMP Echo Request:
Leaf1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0 Capturing on 'ps-inb' Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0 Capturing on 'ps-inb'
Then initiate ping from Server3:
server3:~$ ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=1.19 ms 64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=1.29 ms ^C --- 10.0.0.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.192/1.242/1.292/0.050 ms
Packets were received on Leaf2:
Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0 Capturing on 'ps-inb' 2 285 2024-04-07 09:03:32.459007178 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006e, seq=1/256, ttl=64 286 2024-04-07 09:03:32.459685552 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006e, seq=1/256, ttl=255 (request in 285) 4 294 2024-04-07 09:03:33.459687481 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006e, seq=2/512, ttl=64 295 2024-04-07 09:03:33.460371598 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006e, seq=2/512, ttl=255 (request in 294)
The next round of pings were also received on Leaf2:
Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0 Capturing on 'ps-inb' 2 285 2024-04-07 09:03:32.459007178 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006e, seq=1/256, ttl=64 286 2024-04-07 09:03:32.459685552 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006e, seq=1/256, ttl=255 (request in 285) 4 294 2024-04-07 09:03:33.459687481 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006e, seq=2/512, ttl=64 295 2024-04-07 09:03:33.460371598 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006e, seq=2/512, ttl=255 (request in 294) 4 961 2024-04-07 09:04:13.222153742 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006f, seq=1/256, ttl=64 962 2024-04-07 09:04:13.222872197 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006f, seq=1/256, ttl=255 (request in 961) 6 978 2024-04-07 09:04:14.223805947 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x006f, seq=2/512, ttl=64 979 2024-04-07 09:04:14.224429989 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x006f, seq=2/512, ttl=255 (request in 978)
Then they were received on Leaf1:
Leaf1# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0 Capturing on 'ps-inb' 2 1277 2024-04-07 09:03:21.894839613 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x0070, seq=1/256, ttl=64 1278 2024-04-07 09:03:21.895436908 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x0070, seq=1/256, ttl=255 (request in 1277) 2 1282 2024-04-07 09:03:22.896323401 10.0.0.33 → 10.0.0.1 ICMP 98 Echo (ping) request id=0x0070, seq=2/512, ttl=64 1283 2024-04-07 09:03:22.896906682 10.0.0.1 → 10.0.0.33 ICMP 98 Echo (ping) reply id=0x0070, seq=2/512, ttl=255 (request in 1282)
This is the effect of hashing on Server3. As anycast gateway is used, both switches can respond.
Now let’s ping Server2, which is connected to Leaf2:
server3:~$ ping 10.0.0.22 PING 10.0.0.22 (10.0.0.22) 56(84) bytes of data. 64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.702 ms 64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.847 ms ^C --- 10.0.0.22 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1008ms rtt min/avg/max/mdev = 0.702/0.774/0.847/0.072 ms
There’s two paths the packet can take here:
- Server3 sends ICMP towards Leaf2 and Leaf2 switches the frame locally.
- Server3 sends ICMP towards Leaf1 and Leaf1 switches it across the vPC peer link.
This is shown in the diagram below:
Leaf2 has the MAC address of Server2 locally:
Leaf2# show mac add add 0050.56ad.f48d
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 20 0050.56ad.f48d dynamic NA F F Eth1/3
Leaf1 has it across the vPC peer link:
Leaf1# show mac add add 0050.56ad.f48d
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+ 20 0050.56ad.f48d dynamic NA F F vPC Peer-Link
Next, let’s try connectivity towards Server1:
server3:~$ ping 198.51.100.11 PING 198.51.100.11 (198.51.100.11) 56(84) bytes of data. 64 bytes from 198.51.100.11: icmp_seq=1 ttl=63 time=2.33 ms 64 bytes from 198.51.100.11: icmp_seq=2 ttl=63 time=2.48 ms ^C --- 198.51.100.11 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 2.334/2.407/2.481/0.073 ms
Once again, there are two paths here:
- Server3 sends ICMP towards Leaf1 and Leaf1 routes the packet and forwards it to Server1.
- Server3 sends ICMP towards Leaf2 and Leaf2 routes the packet and forwards it across the vPC peer link.
This is shown in the diagram below:
There is a potential pitfall here, which I will describe in the next section, but first let’s look at when things are working. Leaf1 has a route locally:
Leaf1# show ip route 198.51.100.11 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
198.51.100.11/32, ubest/mbest: 1/0, attached
*via 198.51.100.11, Vlan10, [190/0], 00:58:00, hmm
It also has the MAC address for 198.51.100.11 in the ARP cache:
Leaf1# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 3
Address Age MAC Address Interface Flags
10.0.0.22 00:04:01 0050.56ad.f48d Vlan20 +
10.0.0.33 00:00:26 828b.b6cf.badf Vlan20
198.51.100.11 00:02:05 0050.56ad.8506 Vlan10
Leaf2 also has a local route:
Leaf2# show ip route 198.51.100.11 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
198.51.100.11/32, ubest/mbest: 1/0, attached
*via 198.51.100.11, Vlan10, [190/0], 00:05:20, hmm
Let’s check the ARP cache:
Leaf2# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 3
Address Age MAC Address Interface Flags
10.0.0.22 00:01:09 0050.56ad.f48d Vlan20
10.0.0.33 00:02:34 828b.b6cf.badf Vlan20 +
198.51.100.11 00:04:14 0050.56ad.8506 Vlan10 +
Notice the plus sign in the Flags column which indicates that this ARP entry has been learned via CFS. This entry is reachable via Po12:
Leaf2# show fabric forwarding ip local-host-db vrf Tenant1
HMM host IPv4 routing table information for VRF Tenant1
Status: *-valid, x-deleted, D-Duplicate, DF-Duplicate and frozen,
c-cleaned in 00:08:01
Host MAC Address SVI Flags Physical Interface
* 10.0.0.22/32 0050.56ad.f48d Vlan20 0x420201 Ethernet1/3
* 10.0.0.33/32 828b.b6cf.badf Vlan20 0x420201 port-channel33
* 198.51.100.11/32 0050.56ad.8506 Vlan10 0x420201 port-channel12
Note that the entry is from HMM, while the entry for 198.51.100.44 is via BGP:
Leaf2# show l2route evpn mac-ip evi 10 Flags -(Rmac):Router MAC (Stt):Static (L):Local (R):Remote (Dup):Duplicate (Spl):Split (Rcv):Recv(D):Del Pending (S):Stale (C):Clear (Ps):Peer Sync (Ro):Re-Originated (Orp):Orphan (Asy):Asymmetric (Gw):Gateway (Piporp): Directly connected Orphan to PIP based vPC BGW (Pipporp): Orphan connected to peer of PIP based vPC BGW Topology Mac Address Host IP Prod Flags Seq No Next-Hops ----------- -------------- --------------------------------------- ------ ---------- ---------- --------------------------------------------------------- 10 0050.56ad.8506 198.51.100.11 HMM L, 0 Local 10 0050.56ad.7d68 198.51.100.44 BGP -- 0 203.0.113.4 (Label: 10000)
The ARP suppression cache is also populated:
Leaf2# show ip arp suppression-cache detail
Flags: + - Adjacencies synced via CFSoE
L - Local Adjacency
R - Remote Adjacency
L2 - Learnt over L2 interface
PS - Added via L2RIB, Peer Sync
RO - Dervied from L2RIB Peer Sync Entry
Ip Address Age Mac Address Vlan Physical-ifindex Flags Remote Vtep Addrs
198.51.100.11 00:01:18 0050.56ad.8506 10 port-channel12 L
198.51.100.44 1d15h 0050.56ad.7d68 10 (null) R 203.0.113.4
10.0.0.33 00:00:09 828b.b6cf.badf 20 port-channel33 L
10.0.0.22 00:00:09 0050.56ad.f48d 20 Ethernet1/3 L
Finally, let’s ping Server4, which is connected to Leaf4:
server3:~$ ping 198.51.100.44 PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data. 64 bytes from 198.51.100.44: icmp_seq=1 ttl=62 time=21.3 ms 64 bytes from 198.51.100.44: icmp_seq=2 ttl=62 time=5.92 ms ^C --- 198.51.100.44 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 5.924/13.631/21.338/7.707 ms
There are several paths the packet can take:
- Server3 sends ICMP towards Leaf1 and Leaf1 routes the packet towards Leaf4.
- Server3 sends ICMP towards Leaf2 and Leaf2 routes the packet towards Leaf4.
This is shown in the diagram below:
Note that Leaf4 can send the packet both towards Leaf1 and Leaf2 based on ECMP for the Anycast VTEP.
Leaf1 and Leaf2 both have a route to 198.51.100.44 via Leaf4:
Leaf1# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
198.51.100.44/32, ubest/mbest: 1/0
*via 203.0.113.4%default, [200/0], 5d19h, bgp-65000, internal, tag 65000, segid: 10001 tunnelid: 0xcb007104 encap: VXLAN
Leaf2# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
198.51.100.44/32, ubest/mbest: 1/0
*via 203.0.113.4%default, [200/0], 5d20h, bgp-65000, internal, tag 65000, segid: 10001 tunnelid: 0xcb007104 encap: VXLAN
The route is from EVPN:
Leaf1# show bgp l2vpn evpn 198.51.100.44
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.6:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.7d68]:[32]:[198.51.100.44]/272, version 8
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 3 destination(s)
Imported paths list: Tenant1 L3-10001 L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.4 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
Originator: 192.0.2.6 Cluster list: 192.0.2.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.4 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
Originator: 192.0.2.6 Cluster list: 192.0.2.2
Path-id 1 not advertised to any peer
Leaf2# show bgp l2vpn evpn 198.51.100.44
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.6:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.7d68]:[32]:[198.51.100.44]/272, version 8
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 3 destination(s)
Imported paths list: Tenant1 L3-10001 L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.4 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
Originator: 192.0.2.6 Cluster list: 192.0.2.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.4 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.7083.1b08
Originator: 192.0.2.6 Cluster list: 192.0.2.2
Path-id 1 not advertised to any peer
Leaf4 has a local route for 198.51.100.44:
Leaf4# show ip route 198.51.100.44 vrf Tenant1
IP Route Table for VRF "Tenant1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
198.51.100.44/32, ubest/mbest: 1/0, attached
*via 198.51.100.44, Vlan10, [190/0], 9w2d, hmm
It is also known in the ARP cache:
Leaf4# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 1
Address Age MAC Address Interface Flags
198.51.100.44 00:04:01 0050.56ad.7d68 Vlan10
With connectivity verified, in the next blog post we’ll take a brief look at fabric peering.
Cisco vPC in VXLAN/EVPN Network – Part 3 – Verifying Connectivity