In this post I walk you through all the steps and packets involved in two hosts communicating over a L2 VNI in a VXLAN/EVPN network. The topology below is the one we will be using:

The lab has the following characteristics:
- OSPF in the underlay.
- Ingress replication for BUM traffic through the use of EVPN.
- ARP suppression is enabled.
- ARP cache is cleared on Server-1 and Server-4 before initating the packet capture.
- Server-1 is the host sourcing traffic by pinging Server-4.
Server-1 clears the ARP entry for Server-4 and initiates the ping:
sudo ip neighbor del 198.51.100.44 dev ens160 ping 198.51.100.44 PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data. 64 bytes from 198.51.100.44: icmp_seq=1 ttl=64 time=6.38 ms 64 bytes from 198.51.100.44: icmp_seq=2 ttl=64 time=4.56 ms 64 bytes from 198.51.100.44: icmp_seq=3 ttl=64 time=4.60 ms
Below is packet capture showing the ARP request from Server-1:
Frame 7854: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 4 Ethernet II, Src: 00:50:56:ad:85:06, Dst: ff:ff:ff:ff:ff:ff Address Resolution Protocol (request) Hardware type: Ethernet (1) Protocol type: IPv4 (0x0800) Hardware size: 6 Protocol size: 4 Opcode: request (1) Sender MAC address: 00:50:56:ad:85:06 Sender IP address: 198.51.100.11 Target MAC address: 00:00:00:00:00:00 Target IP address: 198.51.100.44
This is a broadcast as expected. This frame goes to Leaf-1 which checks the ARP suppression cache:
Leaf1# show ip arp suppression-cache detail Flags: + - Adjacencies synced via CFSoE L - Local Adjacency R - Remote Adjacency L2 - Learnt over L2 interface PS - Added via L2RIB, Peer Sync RO - Dervied from L2RIB Peer Sync Entry Ip Address Age Mac Address Vlan Physical-ifindex Flags Remote Vtep Addrs 198.51.100.11 00:02:25 0050.56ad.8506 10 Ethernet1/3 L 198.51.100.44 4w5d 0050.56ad.7d68 10 (null) R 203.0.113.4
There is an entry for 198.51.100.44 (Server-4) so Leaf-1 responds on behalf of Server-4 (this frame is never flooded using ingress replication):
Frame 7855: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 4 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06 Address Resolution Protocol (reply) Hardware type: Ethernet (1) Protocol type: IPv4 (0x0800) Hardware size: 6 Protocol size: 4 Opcode: reply (2) Sender MAC address: 00:50:56:ad:7d:68 Sender IP address: 198.51.100.44 Target MAC address: 00:50:56:ad:85:06 Target IP address: 198.51.100.11
This is shown visually below:

Note that Leaf-1 generates a frame with Source MAC of Server-4.
Server-1 then generates the ICMP Echo request:
Frame 7856: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens257, id 4 Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68 Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xeabc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Response frame: 7857] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.001908472 seconds] Data (40 bytes)
Leaf-1 checks the MAC address table for 00:50:56:ad:7d:68 which is reachable via 203.0.113.4:
Leaf1# show mac address-table vlan 10 address 0050.56ad.7d68 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan, (NA)- Not Applicable VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ C 10 0050.56ad.7d68 dynamic NA F F nve1(203.0.113.4)
This packet needs to be VXLAN encapsulated and it needs to be routed towards a Spine. There are two available routes:
Leaf1# show ip route 203.0.113.4 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 203.0.113.4/32, ubest/mbest: 2/0 *via 192.0.2.1, Eth1/1, [110/81], 6w5d, ospf-UNDERLAY, intra *via 192.0.2.2, Eth1/2, [110/81], 6w5d, ospf-UNDERLAY, intra
Leaf-1 forwards it towards Spine-1:
Frame 7848: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens192, id 1 Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:b3:fd:1b:08 Internet Protocol Version 4, Src: 203.0.113.1, Dst: 203.0.113.4 User Datagram Protocol, Src Port: 62492, Dst Port: 4789 Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) Group Policy ID: 0 VXLAN Network Identifier (VNI): 10000 Reserved: 0 Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68 Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xeabc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Response frame: 7849] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.002592388 seconds] Data (40 bytes)
Spine-1 then forwards it towards Leaf-4:
Frame 7838: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens161, id 0 Ethernet II, Src: 00:ad:b3:fd:1b:08, Dst: 00:ad:70:83:1b:08 Internet Protocol Version 4, Src: 203.0.113.1, Dst: 203.0.113.4 User Datagram Protocol, Src Port: 62492, Dst Port: 4789 Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) Group Policy ID: 0 VXLAN Network Identifier (VNI): 10000 Reserved: 0 Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68 Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xeabc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Response frame: 7839] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.003668982 seconds] Data (40 bytes)
Leaf-4 forwards it towards Server-4:
Frame 7858: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens194, id 8 Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68 Internet Protocol Version 4, Src: 198.51.100.11, Dst: 198.51.100.44 Internet Control Message Protocol Type: 8 (Echo (ping) request) Code: 0 Checksum: 0xeabc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Response frame: 7861] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.003917144 seconds] Data (40 bytes)
This is shown visually below:

The ICMP Echo request has reached Server-4. It does not have an ARP entry for Server-1 so it sends an ARP Request:
Frame 7859: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens194, id 8 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: ff:ff:ff:ff:ff:ff Address Resolution Protocol (request) Hardware type: Ethernet (1) Protocol type: IPv4 (0x0800) Hardware size: 6 Protocol size: 4 Opcode: request (1) Sender MAC address: 00:50:56:ad:7d:68 Sender IP address: 198.51.100.44 Target MAC address: 00:00:00:00:00:00 Target IP address: 198.51.100.11
Leaf-4 checks its ARP suppression cache:
Leaf4# show ip arp suppression-cache detail Flags: + - Adjacencies synced via CFSoE L - Local Adjacency R - Remote Adjacency L2 - Learnt over L2 interface PS - Added via L2RIB, Peer Sync RO - Dervied from L2RIB Peer Sync Entry Ip Address Age Mac Address Vlan Physical-ifindex Flags Remote Vtep Addrs 198.51.100.44 0.816039 0050.56ad.7d68 10 Ethernet1/3 L 198.51.100.11 6w3d 0050.56ad.8506 10 (null) R 203.0.113.1
As it has an entry for 198.51.100.11 (Server-1), it responds on behalf of it:
Frame 7860: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens194, id 8 Ethernet II, Src: 00:50:56:ad:85:06, Dst: 00:50:56:ad:7d:68 Address Resolution Protocol (reply) Hardware type: Ethernet (1) Protocol type: IPv4 (0x0800) Hardware size: 6 Protocol size: 4 Opcode: reply (2) Sender MAC address: 00:50:56:ad:85:06 Sender IP address: 198.51.100.11 Target MAC address: 00:50:56:ad:7d:68 Target IP address: 198.51.100.44
This is shown visually below:

Server-4 can now generate the ICMP Echo reply and send it towards Leaf-4:
Frame 7861: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens194, id 8 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06 Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11 Internet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 Checksum: 0xf2bc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Request frame: 7858] [Response time: 1,184 ms] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.005100694 seconds] Data (40 bytes)
Leaf-4 checks the MAC address table for 00:50:56:ad:85:06 and finds it reachable via 203.0.113.1:
Leaf4# show mac address-table vlan 10 address 0050.56ad.8506 Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan, (NA)- Not Applicable VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ C 10 0050.56ad.8506 dynamic NA F F nve1(203.0.113.1)
This packet needs to be VXLAN encapsulated and it needs to be routed towards a Spine. There are two available routes:
Leaf4# show ip route 203.0.113.1 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 203.0.113.1/32, ubest/mbest: 2/0 *via 192.0.2.1, Eth1/1, [110/81], 6w5d, ospf-UNDERLAY, intra *via 192.0.2.2, Eth1/2, [110/81], 6w5d, ospf-UNDERLAY, intra
Leaf-4 forwards it towards Spine-1:
Frame 7839: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens161, id 0 Ethernet II, Src: 00:ad:70:83:1b:08, Dst: 00:ad:b3:fd:1b:08 Internet Protocol Version 4, Src: 203.0.113.4, Dst: 203.0.113.1 User Datagram Protocol, Src Port: 64411, Dst Port: 4789 Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) Group Policy ID: 0 VXLAN Network Identifier (VNI): 10000 Reserved: 0 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06 Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11 Internet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 Checksum: 0xf2bc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Request frame: 7838] [Response time: 2,376 ms] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.006045303 seconds] Data (40 bytes)
Spine-1 then forwards it towards Leaf-1:
Frame 7849: 148 bytes on wire (1184 bits), 148 bytes captured (1184 bits) on interface ens192, id 1 Ethernet II, Src: 00:ad:b3:fd:1b:08, Dst: 00:ad:e6:88:1b:08 Internet Protocol Version 4, Src: 203.0.113.4, Dst: 203.0.113.1 User Datagram Protocol, Src Port: 64411, Dst Port: 4789 Virtual eXtensible Local Area Network Flags: 0x0800, VXLAN Network ID (VNI) Group Policy ID: 0 VXLAN Network Identifier (VNI): 10000 Reserved: 0 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06 Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11 Internet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 Checksum: 0xf2bc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Request frame: 7848] [Response time: 3,931 ms] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.006523313 seconds] Data (40 bytes)
Leaf-1 forwards it towards Server-1:
Frame 7857: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) on interface ens257, id 4 Ethernet II, Src: 00:50:56:ad:7d:68, Dst: 00:50:56:ad:85:06 Internet Protocol Version 4, Src: 198.51.100.44, Dst: 198.51.100.11 Internet Control Message Protocol Type: 0 (Echo (ping) reply) Code: 0 Checksum: 0xf2bc [correct] [Checksum Status: Good] Identifier (BE): 60 (0x003c) Identifier (LE): 15360 (0x3c00) Sequence Number (BE): 1 (0x0001) Sequence Number (LE): 256 (0x0100) [Request frame: 7856] [Response time: 4,724 ms] Timestamp from icmp data: Feb 24, 2024 08:12:54.931440000 Romance Standard Time [Timestamp from icmp data (relative): 0.006632873 seconds] Data (40 bytes)
This is shown visually below:

In this post we did a packet walk when forwarding packets over a L2 VNI in VXLAN/EVPN network. We learned the following:
- How ARP suppression is used to respond to ARP Request on behalf of another host.
- How the Leaf does a lookup in the MAC address table to find where to forward the frame to.
- How the packets get encapsulated with additional headers like VXLAN.
- That underlay is using ECMP to forward packets towards spine.
I hope this has been informative! In the next post we’ll do a packet walk where forwarding between two different networks through the use of L3 VNI.
Hi Daniel,
Very nice explanation, thanks for your post. I am looking forward to reading the next one related to the L3 VNI forwarding.
Thanks,
Lucian
Thanks, Lucian!
Thanks Daniel, Very nicely explained with each field at every hop.
Thanks!
Thanks Daniel, Clearly explained