In the last post we used multicast to forward BUM frames. In this post, static ingress replication is used which removes the requirement for multicast in the underlay. Before setting this up, let’s do a comparison of multicast vs ingress replication. To make this interesting, let’s use a larger topology consisting of 2 Spines and 32 Leafs:
Assume that all Leafs have the same VNIs. In a scenario with multicast, Leaf-1 sends for example an ARP request encapsulated in multicast towards Spine-1. This is a single frame. Spine-1 then replicates this frame and sends it on all its links towards the Leafs:
This is very efficient as a single frame is sent by Leaf-1 and then 31 copies are sent on the other links to the Leafs. If the ARP request is 110 bytes, then a total of 32 x 110 = 3520 bytes have been consumed to send this ARP request. This is optimal forwarding from a resource consumption perspective and one of the strengths of multicast. Now let’s compare that to ingress replication. With ingress replication, Leaf-1 sends 31x copies of the frame (with different destination IP) towards Spine-1. Spine-1 then forwards those frames on each link towards the other Leafs:
In our second example, frames are sent as unicast. Instead of 32 frames being sent in total, there’s now 62 frames meaning that 6820 bytes have been consumed for this one ARP request. Now let’s say that Leaf-1 needs to generate 100 BUM frames per second. Using multicast, this would be around 88 kbit/s of traffic on the link between Leaf-1 and Spine-1. Using unicast, it would be around 2.7 Mbit/s. That’s still a trivial amount of BW, especially in a DC, but shows the inefficiency of ingress replication.
Now let’s look at setting up ingress replication using static configuration. The configuration is very straightforward:
Leaf1# sh run int nve1 !Command: show running-config interface nve1 !Running configuration last done at: Mon Oct 30 11:38:37 2023 !Time: Mon Oct 30 11:38:41 2023 version 10.2(5) Bios:version interface nve1 no shutdown source-interface loopback1 member vni 1000 ingress-replication protocol static peer-ip 203.0.113.2 peer-ip 203.0.113.3 peer-ip 203.0.113.4
Under the VNI, the
command is added and then every NVE is identified with the
ingress-replication protocol static
peer-ip command. Running a packet capture, we can now see that Leaf-1 is sending multiple frames to the different NVEs:
There’s not much more to it than that. Ingress replication is not as elegant as multicast and it does consume more bandwidth but the advantage is that the configuration is really simple and that there is no state required in the other switches. It uses unicast so forwarding is straightforward. Ingress replication can also be used with EVPN for discovery of the NVEs. We’ll take a look at that in future posts on VXLAN.