In this post I walk you through all the steps and packets involved in two hosts communicating over a L2 VNI in a VXLAN/EVPN network. The topology below is the one we will be using:

The lab has the following characteristics:

  • OSPF in the underlay.
  • Ingress replication for BUM traffic through the use of EVPN.
  • ARP suppression is enabled.
  • ARP cache is cleared on Server-1 and Server-4 before initating the packet capture.
  • Server-1 is the host sourcing traffic by pinging Server-4.

Server-1 clears the ARP entry for Server-4 and initiates the ping:

sudo ip neighbor del 198.51.100.44 dev ens160
ping 198.51.100.44
PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data.
64 bytes from 198.51.100.44: icmp_seq=1 ttl=64 time=6.38 ms
64 bytes from 198.51.100.44: icmp_seq=2 ttl=64 time=4.56 ms
64 bytes from 198.51.100.44: icmp_seq=3 ttl=64 time=4.60 ms

Below is packet capture showing the ARP request from Server-1:

Frame 7854: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 4
Ethernet II, Src: 00:50:56:ad:85:06, Dst: ff:ff:ff:ff:ff:ff
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: 00:50:56:ad:85:06
    Sender IP address: 198.51.100.11
    Target MAC address: 00:00:00:00:00:00
    Target IP address: 198.51.100.44

This is a broadcast as expected. This frame goes to Leaf-1 which checks the ARP suppression cache:

Leaf1# show ip arp suppression-cache detail 

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface
       PS - Added via L2RIB, Peer Sync
       RO - Dervied from L2RIB Peer Sync Entry

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags    Remote Vtep Addrs

198.51.100.11   00:02:25 0050.56ad.8506   10 Ethernet1/3         L
198.51.100.44       4w5d 0050.56ad.7d68   10 (null)              R        203.0.113.4

There is an entry for 198.51.100.44 (Server-4) so Leaf-1 responds on behalf of Server-4 (this frame is never flooded using ingress replication):

Frame 7855: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 4
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06
Address Resolution Protocol (reply)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (2)
    Sender MAC address: 00:50:56:ad:7d:68
    Sender IP address: 198.51.100.44
    Target MAC address: 00:50:56:ad:85:06
    Target IP address: 198.51.100.11

This is shown visually below:

Note that Leaf-1 generates a frame with Source MAC of Server-4.

Server-1 then generates the ICMP Echo request:

Frame 7856: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens257, id 4
Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68
Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xeabc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Response frame: 7857]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.001908472 seconds]
    Data (40 bytes)

Leaf-1 checks the MAC address table for 00:50:56:ad:7d:68 which is reachable via 203.0.113.4:

Leaf1# show mac address-table vlan 10 address 0050.56ad.7d68
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C   10     0050.56ad.7d68   dynamic  NA         F      F    nve1(203.0.113.4)

This packet needs to be VXLAN encapsulated and it needs to be routed towards a Spine. There are two available routes:

Leaf1# show ip route 203.0.113.4
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

203.0.113.4/32, ubest/mbest: 2/0
    *via 192.0.2.1, Eth1/1, [110/81], 6w5d, ospf-UNDERLAY, intra
    *via 192.0.2.2, Eth1/2, [110/81], 6w5d, ospf-UNDERLAY, intra

Leaf-1 forwards it towards Spine-1:

Frame 7848: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens192, id 1
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:b3:fd:1b:08
Internet Protocol Version 4, Src: 203.0.113.1, Dst: 203.0.113.4
User Datagram Protocol, Src Port: 62492, Dst Port: 4789
Virtual eXtensible Local Area Network
    Flags: 0x0800, VXLAN Network ID (VNI)
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 10000
    Reserved: 0
Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68
Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xeabc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Response frame: 7849]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.002592388 seconds]
    Data (40 bytes)

Spine-1 then forwards it towards Leaf-4:

Frame 7838: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens161, id 0
Ethernet II, Src: 00:ad:b3:fd:1b:08, Dst: 00:ad:70:83:1b:08
Internet Protocol Version 4, Src: 203.0.113.1, Dst: 203.0.113.4
User Datagram Protocol, Src Port: 62492, Dst Port: 4789
Virtual eXtensible Local Area Network
    Flags: 0x0800, VXLAN Network ID (VNI)
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 10000
    Reserved: 0
Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68
Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xeabc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Response frame: 7839]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.003668982 seconds]
    Data (40 bytes)

Leaf-4 forwards it towards Server-4:

Frame 7858: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens194, id 8
Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68
Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44
Internet Control Message Protocol
    Type: 8 (Echo (ping) request)
    Code: 0
    Checksum: 0xeabc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Response frame: 7861]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.003917144 seconds]
    Data (40 bytes)

This is shown visually below:

The ICMP Echo request has reached Server-4. It does not have an ARP entry for Server-1 so it sends an ARP Request:

Frame 7859: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens194, id 8
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: ff:ff:ff:ff:ff:ff
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: 00:50:56:ad:7d:68
    Sender IP address: 198.51.100.44
    Target MAC address: 00:00:00:00:00:00
    Target IP address: 198.51.100.11

Leaf-4 checks its ARP suppression cache:

Leaf4# show ip arp suppression-cache detail 

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface
       PS - Added via L2RIB, Peer Sync
       RO - Dervied from L2RIB Peer Sync Entry

Ip Address      Age      Mac Address    Vlan Physical-ifindex    Flags    Remote Vtep Addrs

198.51.100.44   0.816039 0050.56ad.7d68   10 Ethernet1/3         L
198.51.100.11       6w3d 0050.56ad.8506   10 (null)              R        203.0.113.1

As it has an entry for 198.51.100.11 (Server-1), it responds on behalf of it:

Frame 7860: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens194, id 8
Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68
Address Resolution Protocol (reply)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (2)
    Sender MAC address: 00:50:56:ad:85:06
    Sender IP address: 198.51.100.11
    Target MAC address: 00:50:56:ad:7d:68
    Target IP address: 198.51.100.44

This is shown visually below:

Server-4 can now generate the ICMP Echo reply and send it towards Leaf-4:

Frame 7861: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens194, id 8
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06
Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xf2bc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Request frame: 7858]
    [Response time: 1,184 ms]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.005100694 seconds]
    Data (40 bytes)

Leaf-4 checks the MAC address table for 00:50:56:ad:85:06 and finds it reachable via 203.0.113.1:

Leaf4# show mac address-table vlan 10 address 0050.56ad.8506
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C   10     0050.56ad.8506   dynamic  NA         F      F    nve1(203.0.113.1)

This packet needs to be VXLAN encapsulated and it needs to be routed towards a Spine. There are two available routes:

Leaf4# show ip route 203.0.113.1
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

203.0.113.1/32, ubest/mbest: 2/0
    *via 192.0.2.1, Eth1/1, [110/81], 6w5d, ospf-UNDERLAY, intra
    *via 192.0.2.2, Eth1/2, [110/81], 6w5d, ospf-UNDERLAY, intra

Leaf-4 forwards it towards Spine-1:

Frame 7839: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens161, id 0
Ethernet II, Src: 00:ad:70:83:1b:08, Dst: 00:ad:b3:fd:1b:08
Internet Protocol Version 4, Src: 203.0.113.4, Dst: 203.0.113.1
User Datagram Protocol, Src Port: 64411, Dst Port: 4789
Virtual eXtensible Local Area Network
    Flags: 0x0800, VXLAN Network ID (VNI)
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 10000
    Reserved: 0
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06
Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xf2bc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Request frame: 7838]
    [Response time: 2,376 ms]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.006045303 seconds]
    Data (40 bytes)

Spine-1 then forwards it towards Leaf-1:

Frame 7849: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens192, id 1
Ethernet II, Src: 00:ad:b3:fd:1b:08, Dst: 00:ad:e6:88:1b:08
Internet Protocol Version 4, Src: 203.0.113.4, Dst: 203.0.113.1
User Datagram Protocol, Src Port: 64411, Dst Port: 4789
Virtual eXtensible Local Area Network
    Flags: 0x0800, VXLAN Network ID (VNI)
    Group Policy ID: 0
    VXLAN Network Identifier (VNI): 10000
    Reserved: 0
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06
Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xf2bc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Request frame: 7848]
    [Response time: 3,931 ms]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.006523313 seconds]
    Data (40 bytes)

Leaf-1 forwards it towards Server-1:

Frame 7857: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens257, id 4
Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06
Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11
Internet Control Message Protocol
    Type: 0 (Echo (ping) reply)
    Code: 0
    Checksum: 0xf2bc [correct]
    [Checksum Status: Good]
    Identifier (BE): 60 (0x003c)
    Identifier (LE): 15360 (0x3c00)
    Sequence Number (BE): 1 (0x0001)
    Sequence Number (LE): 256 (0x0100)
    [Request frame: 7856]
    [Response time: 4,724 ms]
    Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time
    [Timestamp from icmp data (relative): 0.006632873 seconds]
    Data (40 bytes)

This is shown visually below:

In this post we did a packet walk when forwarding packets over a L2 VNI in VXLAN/EVPN network. We learned the following:

  • How ARP suppression is used to respond to ARP Request on behalf of another host.
  • How the Leaf does a lookup in the MAC address table to find where to forward the frame to.
  • How the packets get encapsulated with additional headers like VXLAN.
  • That underlay is using ECMP to forward packets towards spine.

I hope this has been informative! In the next post we’ll do a packet walk where forwarding between two different networks through the use of L3 VNI.

Bridging Packet Walk In VXLAN/EVPN Network
Tagged on:             

4 thoughts on “Bridging Packet Walk In VXLAN/EVPN Network”

  • Lucian
    February 26, 2024 at 5:51 pm
    Permalink

    Hi Daniel,
    Very nice explanation, thanks for your post. I am looking forward to reading the next one related to the L3 VNI forwarding.
    Thanks,
    Lucian

    Reply
    • ddibPost author
      February 27, 2024 at 9:31 am
      Permalink

      Thanks, Lucian!

      Reply
  • Sumit
    February 28, 2024 at 1:48 pm
    Permalink

    Thanks Daniel, Very nicely explained with each field at every hop.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *