Forwarding BUM Frames in VXLAN Network With Static Ingress Replication

In the last post we used multicast to forward BUM frames. In this post, static ingress replication is used which removes the requirement for multicast in the underlay. Before setting this up, let’s do a comparison of multicast vs ingress replication. To make this interesting, let’s use a larger topology consisting of 2 Spines and 32 Leafs:

Assume that all Leafs have the same VNIs. In a scenario with multicast, Leaf-1 sends for example an ARP request encapsulated in multicast towards Spine-1. This is a single frame. Spine-1 then replicates this frame and sends it on all its links towards the Leafs:

This is very efficient as a single frame is sent by Leaf-1 and then 31 copies are sent on the other links to the Leafs. If the ARP request is 110 bytes, then a total of 32 x 110 = 3520 bytes have been consumed to send this ARP request. This is optimal forwarding from a resource consumption perspective and one of the strengths of multicast. Now let’s compare that to ingress replication. With ingress replication, Leaf-1 sends 31x copies of the frame (with different destination IP) towards Spine-1. Spine-1 then forwards those frames on each link towards the other Leafs:

In our second example, frames are sent as unicast. Instead of 32 frames being sent in total, there’s now 62 frames meaning that 6820 bytes have been consumed for this one ARP request. Now let’s say that Leaf-1 needs to generate 100 BUM frames per second. Using multicast, this would be around 88 kbit/s of traffic on the link between Leaf-1 and Spine-1. Using unicast, it would be around 2.7 Mbit/s. That’s still a trivial amount of BW, especially in a DC, but shows the inefficiency of ingress replication.

Now let’s look at setting up ingress replication using static configuration. The configuration is very straightforward:

Leaf1# sh run int nve1

!Command: show running-config interface nve1
!Running configuration last done at: Mon Oct 30 11:38:37 2023
!Time: Mon Oct 30 11:38:41 2023

version 10.2(5) Bios:version  

interface nve1
  no shutdown
  source-interface loopback1
  member vni 1000
    ingress-replication protocol static
      peer-ip 203.0.113.2
      peer-ip 203.0.113.3
      peer-ip 203.0.113.4

Under the VNI, the ingress-replication protocol static command is added and then every NVE is identified with the peer-ip command. Running a packet capture, we can now see that Leaf-1 is sending multiple frames to the different NVEs:

There’s not much more to it than that. Ingress replication is not as elegant as multicast and it does consume more bandwidth but the advantage is that the configuration is really simple and that there is no state required in the other switches. It uses unicast so forwarding is straightforward. Ingress replication can also be used with EVPN for discovery of the NVEs. We’ll take a look at that in future posts on VXLAN.