Forwarding BUM Frames in VXLAN Network With Multicast in Underlay

There are two main methods that can be used to forward broadcast, unknown unicast, and multicast (BUM) frames in a VXLAN network:

• Ingress replication.
• Multicast in underlay.

In this post, we take a detailed look at how multicast can be used to forward BUM frames by running multicast in the underlay. We are using the topology from my Building a VXLAN Lab Using Nexus9000v post. The Spine switches are configured using the Nexus feature Anycast RP. That is, no MSDP is used between the RPs. To be able to forward broadcast frames such as ARP in our topology, the following is required:

• The VTEPs must signal that they want to join the shared tree for 239.0.0.1 (to receive multicast) using a PIM Join.
• The VTEPs must signal that they intend to send multicast for 239.0.0.1 on the source tree using a PIM Register.
• The RPs must share information about the sources that they know of by forwarding PIM Register messages.
• The VTEP must encapsulate ARP packets in VXLAN and forward in the underlay using multicast.

The Leaf switches have the following configuration to enable multicast:

ip pim rp-address 192.0.2.255 group-list 224.0.0.0/4

interface nve1
  source-interface loopback1
  member vni 1000
    mcast-group 239.0.0.1

interface Ethernet1/1
  no switchport
  mtu 9216
  medium p2p
  ip address 198.51.100.1/31
  ip ospf network point-to-point
  no ip ospf passive-interface
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet1/2
  no switchport
  mtu 9216
  medium p2p
  ip address 198.51.100.9/31
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  ip pim sparse-mode
  no shutdown

Let’s first verify that the Leaf has PIM neighbors and is aware of the RP:

Leaf1# show ip pim nei
PIM Neighbor Status for VRF "default"
Neighbor        Interface            Uptime    Expires   DR       Bidir-  BFD    ECMP Redirect
                                                         Priority Capable State     Capable
198.51.100.0    Ethernet1/1          3d21h     00:01:19  1        yes     n/a     no
198.51.100.8    Ethernet1/2          3d22h     00:01:31  1        yes     n/a     no
Leaf1# show ip pim rp vrf default
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None

RP: 192.0.2.255, (0), 
 uptime: 3d22h   priority: 255, 
 RP-source: (local),  
 group ranges:
 224.0.0.0/4   

Let’s also verify that the Spines are aware of each other as RPs:

Spine1# show ip pim rp vrf default
PIM RP Status Information for VRF "default"
BSR disabled
Auto-RP disabled
BSR RP Candidate policy: None
BSR RP policy: None
Auto-RP Announce policy: None
Auto-RP Discovery policy: None

Anycast-RP 192.0.2.255 members:
  192.0.2.1*  192.0.2.2  

RP: 192.0.2.255*, (0), 
 uptime: 10w0d   priority: 255, 
 RP-source: (local),  
 group ranges:
 224.0.0.0/4   

That is all looking as expected.

Currently, Leafs have their NVE interface shutdown to enable us to do packet captures for the entire interaction. I will enable packet capture in Wireshark on my sniffer to capture the PIM packets.

Currently Leaf1 has no outgoing interface for 239.0.0.1:

Leaf1# show ip mroute
IP Multicast Routing Table for VRF "default"

(*, 232.0.0.0/8), uptime: 3d22h, pim ip 
  Incoming interface: Null, RPF nbr: 0.0.0.0
  Outgoing interface list: (count: 0)

The NVE interface is brought up and Leaf1 sends a PIM Join towards the RP:

Leaf1 also sends a PIM Register:

The RP (Spine1) then sends a PIM Register Stop:

Spine1 also forwards the PIM Register to Spine2 as they are part of the same Anycast RP set:

At this point, Leaf1 is both on the shared and the source tree:

Leaf1# show ip mroute 239.0.0.1
IP Multicast Routing Table for VRF "default"

(*, 239.0.0.1/32), uptime: 00:27:06, nve ip pim 
  Incoming interface: Ethernet1/1, RPF nbr: 198.51.100.0
  Outgoing interface list: (count: 1)
    nve1, uptime: 00:27:06, nve


(203.0.113.1/32, 239.0.0.1/32), uptime: 00:27:06, nve mrib ip pim 
  Incoming interface: loopback1, RPF nbr: 203.0.113.1
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:26:08, pim

For the shared tree, the incoming interface is Eth1/1 which is the interface towards the Spine and the outgoing interface is the VXLAN interface. For the source tree, the source is loopback1 which is the source interface for the NVE and the outgoing interface is Eth1/1 towards Spine1.

The RP is also aware of Leaf1:

Spine1# show ip mroute 239.0.0.1
IP Multicast Routing Table for VRF "default"

(*, 239.0.0.1/32), uptime: 00:30:17, pim ip 
  Incoming interface: loopback1, RPF nbr: 192.0.2.255
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:30:17, pim


(203.0.113.1/32, 239.0.0.1/32), uptime: 00:29:20, pim mrib ip 
  Incoming interface: Ethernet1/1, RPF nbr: 198.51.100.1, internal
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:26:20, pim, (RPF)

Now Leaf4 will have its NVE brought up and we will verify the mroutes again on the Spines:

Spine1# show ip mroute 239.0.0.1
IP Multicast Routing Table for VRF "default"

(*, 239.0.0.1/32), uptime: 00:37:00, pim ip 
  Incoming interface: loopback1, RPF nbr: 192.0.2.255
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:37:00, pim


(203.0.113.1/32, 239.0.0.1/32), uptime: 00:36:03, pim mrib ip 
  Incoming interface: Ethernet1/1, RPF nbr: 198.51.100.1, internal
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:33:03, pim, (RPF)


(203.0.113.4/32, 239.0.0.1/32), uptime: 00:04:32, pim mrib ip 
  Incoming interface: Ethernet1/4, RPF nbr: 198.51.100.7, internal
  Outgoing interface list: (count: 1)
    Ethernet1/1, uptime: 00:04:32, pim

Spine2# show ip mroute 239.0.0.1
IP Multicast Routing Table for VRF "default"

(*, 239.0.0.1/32), uptime: 00:02:08, pim ip 
  Incoming interface: loopback1, RPF nbr: 192.0.2.255
  Outgoing interface list: (count: 1)
    Ethernet1/4, uptime: 00:02:08, pim


(203.0.113.1/32, 239.0.0.1/32), uptime: 00:33:04, pim ip 
  Incoming interface: Ethernet1/1, RPF nbr: 198.51.100.9, internal
  Outgoing interface list: (count: 1)
    Ethernet1/4, uptime: 00:02:08, pim


(203.0.113.4/32, 239.0.0.1/32), uptime: 00:01:33, pim mrib ip 
  Incoming interface: Ethernet1/4, RPF nbr: 198.51.100.15, internal
  Outgoing interface list: (count: 0)

Now let’s clear the ARP cache on Server1 (198.51.100.11) and ping Server4 (198.51.100.44):

cisco@server1:~$ sudo ip neighbor flush dev ens160
cisco@server1:~$ ping 198.51.100.44
PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data.
64 bytes from 198.51.100.44: icmp_seq=1 ttl=64 time=9.46 ms
64 bytes from 198.51.100.44: icmp_seq=2 ttl=64 time=5.03 ms
64 bytes from 198.51.100.44: icmp_seq=3 ttl=64 time=4.19 ms
64 bytes from 198.51.100.44: icmp_seq=4 ttl=64 time=4.35 ms

Server1 sends ARP request that is a broadcast frame as the MAC address of Server4 is unknown:

Note that the frame is sent to 239.0.0.1 using multicast but that it is VXLAN encapsulated. The destination port is 4789 (VXLAN) and the source port is determined based on an algorithm.

Note that when the ARP response comes back, Server4 has learned the MAC address of Server1 so there is no need for a broadcast. The ARP response is unicast and is also sent as unicast between Leaf4 and Leaf1:

There you have it. VXLAN with flood and learn using multicast in the underlay for BUM frames. In another post we will take a look at ingress replication. See you next time!