In my last post EVPN Deepdive Route Types 2 and 3, we took a deepdive into these two route types. I mentioned that the IP address of a host, a /32 or /128 address, could optionally be advertised. I also mentioned that this is mainly to facilitate features such as ARP suppression where a VTEP will be aware of the MAC/IP mapping and not have to flood BUM frames. However, in my last lab no IP addresses were advertised. Why is that? How do we get them advertised?
Currently, I have only setup a L2 VNI in the lab. This provides connectivity for the VLAN that my hosts are in, but it does not provide any L3 services. There is no SVI configured and there is also no L3 service configured that can route between different VNIs. The “standard” way of setting this up would be to configure anycast gateway on the leafs where every leaf that hosts the VNI has the same IP/MAC, but I consider this to be an optimization that I want to cover in a future posts. I prefer to break things down into their components and focus on the configuration needed for each component as opposed to bundling them together. Therefore, let’s see if we can get IP addresses advertised in EVPN type 2 routes simply by configuring a SVI on our leafs.
Before doing any configuration, there is one thing I want to point out. Look at the output below:
Leaf1# show nve interface Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [not-notified] Local Router MAC: 00ad.e688.1b08 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 203.0.113.1, secondary: 0.0.0.0)
Notice Local Router MAC. Then take a look at the following output:
Leaf1# show nve peers det
Details of nve Peers:
----------------------------------------
Peer-Ip: 203.0.113.4
NVE Interface : nve1
Peer State : Up
Peer Uptime : 3d23h
Router-Mac : n/a
Peer First VNI : 10000
Time since Create : 3d23h
Configured VNIs : 10000
Provision State : peer-add-complete
Learnt CP VNIs : 10000
vni assignment mode : SYMMETRIC
Peer Location : N/A
Notice that Router-Mac is n/a.
What is a Router MAC and why do we need it? RFC 9135 defines the Router MAC:
A new EVPN BGP Extended Community called “EVPN Router’s MAC” is introduced here. This new extended community is a transitive extended community with a Type field of 0x06 (EVPN) and a Sub-Type field of 0x03. It may be advertised along with the Encapsulation Extended Community defined in Section 4.1 of [RFC9012].
The EVPN Router’s MAC Extended Community is encoded as an 8-octet value as follows:
This extended community is used to carry the PE’s MAC address for symmetric IRB scenarios, and it is sent with EVPN RT-2. The advertising PE SHALL only attach a single EVPN Router’s MAC Extended Community to a route. In case the receiving PE receives more than one EVPN Router’s MAC Extended Community with a route, it SHALL process the first one in the list and not store and propagate the others.
The Router MAC is used for inter-VNI traffic flows. I will describe symmetric IRB in detail in a future post but for now let’s get back to our task of getting to advertise an IP in combination with MAC for route type 2. For this advertisement, we are expecting:
- One route type 2 route for MAC.
- One route type 2 route for MAC/IP.
- The second route will use MPLS Label2 set to L3 VNI.
- Separate RT for L3 VNI as opposed to L2 VNI.
- Router MAC extended community attached to route with MAC address of SVI used as transit VNI.
To get symmetric IRB working we will need the following:
- Define a VLAN for transit VNI and assign a VNI to it.
- Map the L3 VNI to a VRF.
- Define a SVI for the VLAN created for the transit VNI. This SVI will be used for inter-VNI routing.
- Associate L3VNI to a VRF under the NVE interface.
We will configure this shortly but I find it useful to break things down into their components and sometimes also to provide a faulty or partially complete configuration to verify what each command does. What happens if we just configure a SVI in a VRF?
vrf context Tenant1
vni 10001
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
!
interface Vlan10
no shutdown
vrf member Tenant1
ip address 198.51.100.1/24
After configuring this, there was no IP address in the type 2 routes:
Leaf1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 11, Local Router ID is 192.0.2.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.0.2.3:32777 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
203.0.113.1 100 32768 i
*>l[3]:[0]:[32]:[203.0.113.1]/88
203.0.113.1 100 32768 i
*>i[3]:[0]:[32]:[203.0.113.4]/88
203.0.113.4 100 0 i
Route Distinguisher: 192.0.2.6:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>i 203.0.113.4 100 0 i
* i[3]:[0]:[32]:[203.0.113.4]/88
203.0.113.4 100 0 i
*>i 203.0.113.4 100 0 i
Could this be related to the ARP cache being empty?
Leaf1# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 0
To verify this, I initiated a ping from a host towards the SVI of Leaf1. The ARP cache is now populated:
Leaf1# show ip arp vrf Tenant1
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies
PS - Added via L2RIB, Peer Sync
RO - Re-Originated Peer Sync Entry
D - Static Adjacencies attached to down interface
IP ARP Table for context Tenant1
Total number of entries: 1
Address Age MAC Address Interface Flags
198.51.100.11 00:00:08 0050.56ad.8506 Vlan10
There was no change in perspective of EVPN, though. Still no IP being advertised in route type 2. It would seem that a SVI can’t be host facing for L2 VNI if it is not configured as anycast gateway. I have not been able to verify this in configuration guides, though. Configuring anycast gateway is the standard setup of using host facing SVIs. Let’s enable this:
Leaf1(config)# fabric forwarding anycast-gateway-mac 0001.0001.0001 Leaf1(config)# int vlan10 Leaf1(config-if)# fabric forwarding mode anycast-gateway
At this point I started to see IP addresses in route type 2:
Leaf1# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 12, Local Router ID is 192.0.2.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.0.2.3:32777 (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
203.0.113.1 100 32768 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272
203.0.113.1 100 32768 i
*>l[3]:[0]:[32]:[203.0.113.1]/88
203.0.113.1 100 32768 i
*>i[3]:[0]:[32]:[203.0.113.4]/88
203.0.113.4 100 0 i
Route Distinguisher: 192.0.2.6:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
203.0.113.4 100 0 i
*>i 203.0.113.4 100 0 i
* i[3]:[0]:[32]:[203.0.113.4]/88
203.0.113.4 100 0 i
*>i 203.0.113.4 100 0 i
We can check a route in more detail:
Leaf1# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777 (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 12
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path locally originated
203.0.113.1 (metric 0) from 0.0.0.0 (192.0.2.3)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8
Path-id 1 advertised to peers:
192.0.2.11 192.0.2.12
This route is now being advertised to the RRs (spines), but not making it to other leafs. Why? Let’s take a look at the spine:
Spine1# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/248, version 14
Paths: (1 available, best #0)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is invalid(no RMAC), no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 41) from 192.0.2.101 (192.0.2.3)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8
This route now has the following information in it that was not there until IP got advertised:
- IP Address Length set to 32 instead of zero.
- IP Address set to 198.51.100.11 instead of quad zero.
- Additional route target (extended community) set to 65000:10001 (10001 is the L3 VNI).
- MPLS Label2 set to 10001 (10001 is the L3 VNI).
However, one piece of information is missing. The spine is giving it to us showing “invalid(no RMAC)”. RFC 9135 has the following passage on RMAC:
Each NVE advertises an EVPN RT-2 route with two Route Targets (one corresponding to its MAC-VRF and the other corresponding to its IP-VRF). Furthermore, the EVPN RT-2 is advertised with two BGP Extended Communities. The first BGP Extended Community identifies the tunnel type, and it is called “Encapsulation Extended Community” as defined in [RFC9012], and the second BGP Extended Community includes the MAC address of the NVE (e.g., MACx for NVE1 or MACy for NVE2) as defined in Section 8.1. The EVPN Router’s MAC Extended Community MUST be added when the Ethernet NVO tunnel is used. If the IP NVO tunnel type is used, then there is no need to send this second Extended Community. It should be noted that the IP NVO tunnel type is only applicable to symmetric IRB procedures.
Since we are now advertising IP, and hence two route targets, the Router MAC must be advertised.
For completeness, we can also see this route in our packet capture:
Frame 363: 185 bytes on wire (1480 bits), 185 bytes captured (1480 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 51368, Dst Port: 179, Seq: 233, Ack: 490, Len: 119
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 119
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 96
Path attributes
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop: 203.0.113.1
Number of Subnetwork points of attachment (SNPA): 0
Network Layer Reachability Information (NLRI)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: 00:50:56:ad:85:06
IP Address Length: 32
IPv4 address: 198.51.100.11
VNI: 10000
VNI: 10001
Path Attribute - ORIGIN: IGP
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 24
Carried extended communities: (3 communities)
Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
Now that we’ve got this far, let’s supply a fully functional configuration and walk through all of the commands:
There’s a few interesting things going on here. Notice that the SVI for VLAN 100 has no IP address. This is because we only need the MAC address for data plane forwarding. This is the Router MAC that will be advertised as extended community in route type 2. The command ip forward is needed on the interface to be able to process IP packets without having an IP address configured. The member vni 10001 associate-vrf command is also interesting. Because the L3 VNI is associated with a VRF, it is not enough to simply configure member vni 10001.
With this configured, we now have valid routes. On Leaf4 we can see the route for the host that is connected to Leaf1:
Leaf4# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 13
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 3 destination(s)
Imported paths list: Tenant1 L3-10001 L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
Originator: 192.0.2.3 Cluster list: 192.0.2.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
Originator: 192.0.2.3 Cluster list: 192.0.2.2
Path-id 1 not advertised to any peer
Notice that the Router MAC is now included. We can also see this in a packet capture:
Frame 903: 193 bytes on wire (1544 bits), 193 bytes captured (1544 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 51368, Dst Port: 179, Seq: 656, Ack: 794, Len: 127
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 127
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 104
Path attributes
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop: 203.0.113.1
Number of Subnetwork points of attachment (SNPA): 0
Network Layer Reachability Information (NLRI)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: 00:50:56:ad:85:06
IP Address Length: 32
IPv4 address: 198.51.100.11
VNI: 10000
VNI: 10001
Path Attribute - ORIGIN: IGP
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 32
Carried extended communities: (4 communities)
Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
EVPN Router's MAC: Router's MAC: 00:ad:e6:88:1b:08 [Transitive EVPN]
This is the MAC of SVI for VLAN 100 on Leaf1:
Leaf1# show int vlan 100 | i Ether Hardware is EtherSVI, address is 00ad.e688.1b08
With this there is now a Router MAC populated for NVE peers:
Leaf4# show nve peers det
Details of nve Peers:
----------------------------------------
Peer-Ip: 203.0.113.1
NVE Interface : nve1
Peer State : Up
Peer Uptime : 4d23h
Router-Mac : 00ad.e688.1b08
Peer First VNI : 10000
Time since Create : 4d23h
Configured VNIs : 10000-10001
Provision State : peer-add-complete
Learnt CP VNIs : 10000-10001
vni assignment mode : SYMMETRIC
Peer Location : N/A
Finally, it’s worth highlighting that there are now two route type 2 for MAC address of host connected to Leaf1:
Leaf4# show bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 17, Local Router ID is 192.0.2.6
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.0.2.3:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
203.0.113.1 100 0 i
*>i 203.0.113.1 100 0 i
*>i[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272
203.0.113.1 100 0 i
* i 203.0.113.1 100 0 i
That was a lot of information to digest! What did we learn?
- To advertise IPs into route type 2, the following is required:
- Anycast gateway configured and enabled on SVI toward hosts.
- SVI used for inter-VNI routing.
- A L3 VNI.
- VLAN that is assigned the L3 VNI.
- VRF that is assigned a L3 VNI.
- L3 VNI associated with NVE.
In a future post we will look at what the IP address in route type 2 can be used for and how it ties into route type 5. Stay tuned!
Hello Daniel,
Great post with lots of great outputs, thankyou.
Question I had, if for whatever reason you don’t want to have the vlan communicate with other vlan, so no inter-vni routing, could you leave off the following commands to achieve this.
Leaf1(config)# int vlan10
Leaf1(config-if)# fabric forwarding mode anycast-gateway
Regards
Cem Akilli
Hi Cem,
If you want to isolate at L2, you could remove the SVI. Then there is only L2 connectivity.
When creating VNI for EVPN VXLAN , why we have to define route target different for each L2 VNI? Why not same RT for all L2 VNI? What role does RT play here for Type 2 and Type 3 routes?
The RT is used to know into what MAC VRF (for L2 VNI) and IP VRF (for L3 VNI) that routes should get imported to. Using the same RT for L2 VNI would mean that they are all on the same L2 segment.
Hi Daniel
Thanks for sharing, it really help.
May I ask, if there are two clients need to communicate with each other(same vlan), and the clients are allocated to different vtep, so L3 vni is no nesscery, only use l2 vni to communicate, right?
Yes, that’s correct.
Hey,
You mentioned: “Router MAC extended community attached to route with MAC address of SVI used as transit VNI.” This should be probably MAC address of NVE interface not transit SVI, or?
Thanks