Troubleshooting vPC in My Virtual Lab

I’m preparing a blog post on setting up vPC in a VXLAN/EVPN environment. While doing so, I ran into some issues. Rather than simply fixing them, I wanted to share the troubleshooting experience as it can be useful to see all the things I did to troubleshoot, including commands, packet captures, etc., and learn a little about virtual networking. As always, thanks to Peter Palúch for providing assistance with the process.

Topology

The following topology implemented in ESX is used:

Background

I had just configured the vPC peer link and vPC peer link keepalive. I verified that the vPC was functional with the following command:

Leaf1# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1   
Peer status                       : peer adjacency formed ok      
vPC keep-alive status             : peer is alive                 
Configuration consistency status  : success 
Per-vlan consistency status       : success                       
Type-2 consistency status         : success 
vPC role                          : primary                       
Number of vPCs configured         : 1   
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Delay-restore Orphan-port status  : Timer is off.(timeout = 0s)
Operational Layer3 Peer-router    : Disabled
Virtual-peerlink mode             : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans    
--    ----   ------ -------------------------------------------------
1     Po12   up     1,10,20,100                                                          

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
33    Po33          up     success     success               20

Everything seemed OK at this point. I then started to verify connectivity which is described in the next section.

Symptoms

When trying to ping the gateway (10.0.0.1), sometimes there would be a response, and sometimes not:

server3:~$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
^C
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2034ms

server3:~$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=1.06 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=1.06 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.056/1.059/1.062/0.003 ms

The pings would succeed when they were sent towards Leaf1, but fail when sent towards Leaf2. I used Ethanalyzer on Leaf2 to see if there were any packets received, but there was no incoming ICMP at all:

Leaf2# ethanalyzer local interface inband display-filter "icmp" limit-captured-frames 0
Capturing on 'ps-inb'

I used my sniffer in the lab, which is connected to same port group used for Server3, to verify that the ICMP packets were being sent from Server3:

Frame 10: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens257, id 4
Ethernet II, Src: 82:8b:b6:cf:ba:df, Dst: 00:01:00:01:00:01
Internet Protocol Version 4, Src: 10.0.0.33, Dst: 10.0.0.1
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xc00b [correct]
    [Checksum Status: Good]
    Identifier (BE): 11 (0x000b)
    Identifier (LE): 2816 (0x0b00)
    Sequence Number (BE): 240 (0x00f0)
    Sequence Number (LE): 61440 (0xf000)
    [No response seen]
    Timestamp from icmp data: Apr  3, 2024 07:27:21.120752000 Romance Daylight Time
    [Timestamp from icmp data (relative): 0.001211016 seconds]
    Data (40 bytes)

Notice that packet was received on interface ens257, which is interface that is in same port group as Server3 and Eth1/7 on Leaf2.

At this point we know the following:

• The vPC is operational.
• The port towards Server3 are up.
• ICMP towards Leaf1 succeeds.
• ICMP towards Leaf2 fails.
• The ICMP Echo Request is not seen on Leaf2.
• The sniffer sees the ICMP Echo Request towards Leaf2.

Troubleshooting

Why would the switch not see the packet? Is there an issue with the vNIC? Is there an issue with the port group? I tried changing interfaces and port groups, but that did not resolve the problem. I then started looking for ways to debug what is received on the vNIC in NX-OS in the l2fwder component. The process for debugging l2fwder has changed a bit in newer NX-OS versions where you first need to attach to the module:

Leaf2# attach mod 1
Attaching to module 1 ...
To exit type 'exit', to abort type '$.' 

I then ran the following debug command:

module-1# debug l2fwder packet

Then I noticed something interesting in the output below:

l2fwd: l2fwder_decap_intf(189):Received packet on Ethernet1/7 
l2fwd: l2fwder_decap_intf(201):Before decap :
l2fwd: l2fwder_decap_intf(206): 00 01 00 01 00 01 82 8b 
l2fwd: l2fwder_decap_intf(206): b6 cf ba df 08 00 45 00 
l2fwd: l2fwder_decap_intf(206): 00 54 95 f0 40 00 40 01 
l2fwd: l2fwder_decap_intf(206): 90 97 0a 00 00 21 0a 00 
l2fwd: l2fwder_decap_intf(206): 00 01 08 00 3d 08 00 5f 
l2fwd: l2fwder_decap_intf(206): 00 01 ed 48 0d 66 00 00 
l2fwd: l2fwder_decap_intf(206): 00 00 fa 15 07 00 00 00 
l2fwd: l2fwder_decap_intf(206): 00 00 10 11 12 13 14 15 
l2fwd: l2fwder_decap_intf(206): 16 17 18 19 1a 1b 1c 1d 
l2fwd: l2fwder_init_pkt_context(141):l2fwder_init_pkt_context: Ethernet type is 0x800
l2fwd: l2fwder_decap_port_channel(71):l2fwder_decap_port_channel: Packet decap processing : derived gport 0xc000003 for trunk 0x3 
l2fwd: l2fwder_decap_sub_interface(28):l2fwder_decap_sub_interface : Received Packet on (null) 
l2fwd: l2fwder_decap_sub_interface(43):l2fwder_decap_sub_interface Return not subint packet 
l2fwd: l2fwder_vxlan_parse_l3l4_hdr(209):Entered l2fwder_vxlan_parse_l3l4_hdr 
l2fwd: l2fwder_decap_switchport(149):l2fwder_decap_switchport : Received Packet on (null) 
l2fwd: l2fwder_decap_vlan(41):l2fwder_decap_vlan : In gport = 0xc000003, vni 0 
l2fwd: l2fwder_decap_vlan(53):Received untagged packet on interface (null), vlan = 20 
l2fwd: l2fwder_decap_switchport(207):l2fwder_decap_switchport: Derived sw_bd 20 for vni 20
l2fwd: l2fwder_decap_switchport(239):l2fwder_decap_switchport: Decap done. vni 20, sw_bd 20, etype 0x800, intf (null) 
l2fwd: l2fwder_decap_intf(243):l2fwder_decap_intf: intf (type 2) update ctxt for dmac 00:01:00:01:00:01 smac  82:8b:b6:cf:ba:df etype 0x800 vlan 20, interface (null)
l2fwd: l2fwder_get_etype(249):Ethernet type is 0x800
l2fwd: l2fwder_vlan_check_stp_forward(294):in_intf: Ethernet1/7 in blk/listen state, drop pkt 
l2fwd: l2fwder_input(207):Drop the pkt rx on stp-blk state In intf: Ethernet1/7 with vni: 20 

The message that caught my attention was Drop the pkt rx on stp-blk state In intf: Ethernet1/7. The packet is being dropped on RX by l2fwder, which is why I never saw it when using Ethanalyzer. Furthermore, it also says that it’s due to STP blocking state.

While I did verify that the port was up, I never verified the STP state of the ports. Let’s verify the STP state for Po33, the interface towards Server3:

Leaf1# show span int po33

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020         Desg FWD 1         128.4128 (vPC) P2p 

Leaf1 is forwarding as expected. What about Leaf2?

Leaf2# show span int po33

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020         Desg BKN*1         128.4128 (vPC) P2p *vPC_PL_Inc

Aha! STP is in blocking state. There is also a message that the vPC peer link is inconsistent. Let’s check STP for the peer link as well:

Leaf1# show span int po12

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Desg FWD 3         128.4107 (vPC peer-link) Network P2p 

VLAN0010         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

VLAN0020         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

VLAN0100         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

Leaf2# show span int po12

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 3         128.4107 (vPC peer-link) Network P2p 

VLAN0010         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

VLAN0020         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

VLAN0100         Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc

We can get some additional information with the command below:

Leaf1# show spanning-tree inconsistentports 

Name                 Interface              Inconsistency
-------------------- ---------------------- ------------------
VLAN0010             Po12                   Bridge Assurance Inconsistent, VPC Peer-link Inconsistent
VLAN0020             Po12                   Bridge Assurance Inconsistent, VPC Peer-link Inconsistent
VLAN0100             Po12                   Bridge Assurance Inconsistent, VPC Peer-link Inconsistent

Number of inconsistent ports (segments) in the system : 3

Furthermore, both switches claim they are the Root!

Leaf1# show span vlan 20

VLAN0020
  Spanning tree enabled protocol rstp
  Root ID    Priority    32788
             Address     00ad.e688.1b08
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32788  (priority 32768 sys-id-ext 20)
             Address     00ad.e688.1b08
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po12             Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc
Po33             Desg FWD 1         128.4128 (vPC) P2p 

Leaf2# show span vlan 20

VLAN0020
  Spanning tree enabled protocol rstp
  Root ID    Priority    32788
             Address     00ad.f3bb.1b08
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32788  (priority 32768 sys-id-ext 20)
             Address     00ad.f3bb.1b08
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po12             Desg BKN*3         128.4107 (vPC peer-link) Network P2p *BA, vPC_PL_Inc
Po33             Desg BKN*1         128.4128 (vPC) P2p *vPC_PL_Inc
Eth1/3           Desg FWD 4         128.3    P2p 
Eth1/8           Desg FWD 4         128.8    P2p 

If we check spanning tree for VLAN 1, we notice something interesting, though:

Leaf1# show span vlan 1

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     00ad.e688.1b08
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     00ad.e688.1b08
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po12             Desg FWD 3         128.4107 (vPC peer-link) Network P2p 

Leaf2# show span vlan 1

VLAN0001
  Spanning tree enabled protocol rstp
  Root ID    Priority    32769
             Address     00ad.e688.1b08
             Cost        3
             Port        4107 (port-channel12)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     00ad.f3bb.1b08
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po12             Root FWD 3         128.4107 (vPC peer-link) Network P2p 

The switches agree about VLAN 1! Leaf1 is the Root.

Let’s enable some STP debugs to see what is going on:

Leaf1# debug spanning-tree bpdu_rx 
Leaf1# debug spanning-tree bpdu_tx
Leaf1# debug spanning-tree error
Leaf1# debug spanning-tree event

The debug shows something interesting:

stp: BPDU Rx: Received BPDU on vb 1 vlan 1 port port-channel12 pkt_len 64 bpdu_len 42 netstack flags 0x80001ed enc_type sstp 
stp:  BPDU RX: vb 1 vlan 1 port port-channel12 len 64 flags 0x80001ed: Ethernet Hdr 01000ccccccd<-00adf3bb0104 type/len 0032: SNAP aa aa 03 00000c 010b SSTP CFG P:0000 V:02 T:02 F:78 R:80:01:00:ad:e6:88:1b:08 00000003 B:80:01:00:ad:f3:bb:1b:08 900b A:0000 M:0014 H:0002 F:000f      T:
stp: BPDU Rx: Dropping redundant SSTP packet received on port port-channel12 vlan VLAN0001

Some BPDUs are being dropped. However, this is not really the issue here. Remember, Cisco devices running STP on a trunk port will send two types of BPDUs:

• IEEE STP BPDU to 0180.c200.0000.
  SSTP STP BPDU to 0100.0ccc.cccd.

IEEE BPDU is below:

Frame 2: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 0
IEEE 802.3 Ethernet 
    Destination: 01:80:c2:00:00:00
    Source: 00:ad:e6:88:01:04
    Length: 39
    Padding: 00000000000000
Logical-Link Control
Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x3c, Forwarding, Learning, Port Role: Designated
    Root Identifier: 32768 / 1 / 00:ad:e6:88:1b:08
    Root Path Cost: 0
    Bridge Identifier: 32768 / 1 / 00:ad:e6:88:1b:08
    Port identifier: 0x900b
    Message Age: 0
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0

SSTP BPDU is below:

Frame 3: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface ens257, id 0
IEEE 802.3 Ethernet 
    Destination: 01:00:0c:cc:cc:cd
    Source: 00:ad:e6:88:01:04
    Length: 50
Logical-Link Control
Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x3c, Forwarding, Learning, Port Role: Designated
    Root Identifier: 32768 / 1 / 00:ad:e6:88:1b:08
    Root Path Cost: 0
    Bridge Identifier: 32768 / 1 / 00:ad:e6:88:1b:08
    Port identifier: 0x900b
    Message Age: 0
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0
    Originating VLAN (PVID): 1

The main difference, except for the destination MAC, is that it contains the originating VLAN field.

In my capture, which was taken from a sniffer so not on the switch itself, there is no BPDU for anything other than VLAN 1. Is the switch generating those BPDUs, though? Let’s see if we can see them on Leaf1 using Ethanalyzer:

Leaf1# ethanalyzer local interface inband capture-filter "ether dst 01:80:c2:00:00:00 || 01:00:0c:cc:cc:cd" limit-captured-frames 200 write bootflash:STP.pcap
Capturing on 'ps-inb'

I then copied the file to my computer so I could analyze it with Wireshark. However, there were no frames there with STP originating in any other VLAN than VLAN 1. Why is that? This is because the peer link is broken so there are no BPDUs for VLAN 20 being sent:

Leaf1# show span vlan 20 detail

 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address 00ad.e688.1b08
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 7 last change occurred 43:40:16 ago
          from port-channel33
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0

 Port 4107 (port-channel12, vPC Peer-link) of VLAN0020 is broken  (Bridge Assurance Inconsistent, VPC Peer-link Inconsistent)
   Port path cost 3, Port priority 128, Port Identifier 128.4107
   Designated root has priority 32788, address 00ad.e688.1b08
   Designated bridge has priority 32788, address 00ad.e688.1b08
   Designated port id is 128.4107, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port type is network
   Link type is point-to-point by default
   BPDU: sent 44, received 0

There have been 44 BPDUs sent previously, but this counter is not incrementing. Flapping the peer link will cause BPDUs to be sent again:

Leaf1(config)# int po12
Leaf1(config-if)# shut
Leaf1(config-if)# no shut

We can also see STP moving through different stages:

Leaf1# show span vlan 20 det

 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address 00ad.e688.1b08
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 7 last change occurred 43:42:07 ago
          from port-channel33
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0

 Port 4107 (port-channel12, vPC Peer-link) of VLAN0020 is designated blocking 
   Port path cost 3, Port priority 128, Port Identifier 128.4107
   Designated root has priority 32788, address 00ad.e688.1b08
   Designated bridge has priority 32788, address 00ad.e688.1b08
   Designated port id is 128.4107, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 0
   The port type is network
   Link type is point-to-point by default
   BPDU: sent 7, received 0

Leaf1# show span vlan 20 det

 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address 00ad.e688.1b08
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 7 last change occurred 43:42:14 ago
          from port-channel33
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0

 Port 4107 (port-channel12, vPC Peer-link) of VLAN0020 is designated learning 
   Port path cost 3, Port priority 128, Port Identifier 128.4107
   Designated root has priority 32788, address 00ad.e688.1b08
   Designated bridge has priority 32788, address 00ad.e688.1b08
   Designated port id is 128.4107, designated path cost 0
   Timers: message age 0, forward delay 7, hold 0
   Number of transitions to forwarding state: 0
   The port type is network
   Link type is point-to-point by default
   BPDU: sent 11, received 0

Leaf1# show span vlan 20 det

 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address 00ad.e688.1b08
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag set, detected flag not set
  Number of topology changes 8 last change occurred 0:00:00 ago
          from port-channel12
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 34, notification 0

 Port 4107 (port-channel12, vPC Peer-link) of VLAN0020 is designated forwarding 
   Port path cost 3, Port priority 128, Port Identifier 128.4107
   Designated root has priority 32788, address 00ad.e688.1b08
   Designated bridge has priority 32788, address 00ad.e688.1b08
   Designated port id is 128.4107, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port type is network
   Link type is point-to-point by default
   BPDU: sent 15, received 0

Leaf1# show span vlan 20 det

 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address 00ad.e688.1b08
  Configured hello time 2, max age 20, forward delay 15
  We are the root of the spanning tree
  Topology change flag not set, detected flag not set
  Number of topology changes 8 last change occurred 0:01:22 ago
          from port-channel12
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0

 Port 4107 (port-channel12, vPC Peer-link) of VLAN0020 is broken  (Bridge Assurance Inconsistent, VPC Peer-link Inconsistent)
   Port path cost 3, Port priority 128, Port Identifier 128.4107
   Designated root has priority 32788, address 00ad.e688.1b08
   Designated bridge has priority 32788, address 00ad.e688.1b08
   Designated port id is 128.4107, designated path cost 0
   Timers: message age 0, forward delay 0, hold 0
   Number of transitions to forwarding state: 1
   The port type is network
   Link type is point-to-point by default
   BPDU: sent 42, received 0

Below is an example of the BPDU being sent:

Frame 31: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface ps-inb, id 0
IEEE 802.3 Ethernet 
    Destination: 01:00:0c:cc:cc:cd
    Source: 00:ad:e6:88:01:05
    Length: 50
Logical-Link Control
Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x0e, Port Role: Designated, Proposal
    Root Identifier: 32768 / 20 / 00:ad:e6:88:1b:08
    Root Path Cost: 0
    Bridge Identifier: 32768 / 20 / 00:ad:e6:88:1b:08
    Port identifier: 0x900b
    Message Age: 0
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0
    Originating VLAN (PVID): 20
        Type: Originating VLAN (0x0000)
        Length: 2
        Originating VLAN: 20

Unfortunately I wasn’t able to capture the 802.1Q header, but the originating VLAN is a good clue that the tag should be there.

There are BPDUs being sent by the switches but the other side does not receive it, why? This comes down to how I built my port groups in ESX:

My mistake here was using two port groups and considering them the equivalent of a point to point link in the physical world. This is only true to some degree. The issue here is that the vPC peer link is a trunk. It will send tagged frames on its vNICs and the vSwitch will discard them as it is expecting untagged frames. To support tagged frames, the VLAN ID of 4095 must be used. Now, there is another interesting aspect here. If I were to simply create a new port group with VLAN ID 4095 and use for both the links, then traffic would “leak” between the interfaces so that the switches would get frames that were not intended for that interface. To work around this, I will need to provision two new vSwitches that use VLAN 4095 to keep them separate.

After adding two new vSwitches and mapping the vNICs to them, the vPC peer link is repaired:

Leaf2# show span int po12

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 3         128.4107 (vPC peer-link) Network P2p 

VLAN0010         Root FWD 3         128.4107 (vPC peer-link) Network P2p 

VLAN0020         Root FWD 3         128.4107 (vPC peer-link) Network P2p 

VLAN0100         Root FWD 3         128.4107 (vPC peer-link) Network P2p 

The interface towards the vPC-connected host is now also forwarding:

Leaf2# show span int po33

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020         Desg FWD 1         128.4128 (vPC) P2p 

Now I can ping 10.0.0.1 without issues:

server3:~$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=255 time=16.4 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=255 time=1.17 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=255 time=1.53 ms
^C
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.168/6.360/16.381/7.087 ms

There are now also BPDUs with VLAN tags as well:

Frame 1: 68 bytes on wire (544 bits), 68 bytes captured (544 bits) on interface ens257, id 0
Ethernet II, Src: 00:ad:f3:bb:01:04, Dst: 01:00:0c:cc:cc:cd
802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 20
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = DEI: Ineligible
    .... 0000 0001 0100 = ID: 20
    Length: 50
Logical-Link Control
    DSAP: SNAP (0xaa)
    SSAP: SNAP (0xaa)
    Control field: U, func=UI (0x03)
    Organization Code: 00:00:0c (Cisco Systems, Inc)
    PID: PVSTP+ (0x010b)
Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x78, Agreement, Forwarding, Learning, Port Role: Root
    Root Identifier: 32768 / 20 / 00:ad:e6:88:1b:08
    Root Path Cost: 3
    Bridge Identifier: 32768 / 20 / 00:ad:f3:bb:1b:08
    Port identifier: 0x900b
    Message Age: 0
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0
    Originating VLAN (PVID): 20
        Type: Originating VLAN (0x0000)
        Length: 2
        Originating VLAN: 20

So what did we learn from this post?

• Always check what the STP state is of a port when having forwarding issues.
• A debug of l2fwder component informs us what is happening to the packet.
• Packets are dropped before they can be seen by packet capturing tools like Ethanalyzer.
• How to use tools like Ethanalyzer to capture packets.
• The vPC peer link needs to get BPDUs for the VLANs it carries or the link will be inconsistent.
• The vSwitch in ESX will drop tagged frames when using VLAN ID other than 4095.
• To tag frames in ESX, use a VLAN ID of 4095 for the port group.
• To simulate a point to point link in ESX when using VLAN ID of 4095, multiple vSwitches are needed.

I hope this was a good learning experience for you as well! I could have fixed this error much faster than I did, but I used it as a learning experience to dive deeper into why it was broken and to learn more about the forwarding and protocols. See you in the next one!