Advertising IPs In EVPN Route Type 2

In my last post EVPN Deepdive Route Types 2 and 3, we took a deepdive into these two route types. I mentioned that the IP address of a host, a /32 or /128 address, could optionally be advertised. I also mentioned that this is mainly to facilitate features such as ARP suppression where a VTEP will be aware of the MAC/IP mapping and not have to flood BUM frames. However, in my last lab no IP addresses were advertised. Why is that? How do we get them advertised?

Currently, I have only setup a L2 VNI in the lab. This provides connectivity for the VLAN that my hosts are in, but it does not provide any L3 services. There is no SVI configured and there is also no L3 service configured that can route between different VNIs. The “standard” way of setting this up would be to configure anycast gateway on the leafs where every leaf that hosts the VNI has the same IP/MAC, but I consider this to be an optimization that I want to cover in a future posts. I prefer to break things down into their components and focus on the configuration needed for each component as opposed to bundling them together. Therefore, let’s see if we can get IP addresses advertised in EVPN type 2 routes simply by configuring a SVI on our leafs.

Before doing any configuration, there is one thing I want to point out. Look at the output below:

Leaf1# show nve interface 
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [not-notified]
 Local Router MAC: 00ad.e688.1b08
 Host Learning Mode: Control-Plane
 Source-Interface: loopback1 (primary: 203.0.113.1, secondary: 0.0.0.0)

Notice

Leaf1# show nve peers det
Details of nve Peers:
----------------------------------------
Peer-Ip: 203.0.113.4
    NVE Interface       : nve1
    Peer State          : Up
    Peer Uptime         : 3d23h
    Router-Mac          : n/a
    Peer First VNI      : 10000
    Time since Create   : 3d23h
    Configured VNIs     : 10000
    Provision State     : peer-add-complete
    Learnt CP VNIs      : 10000
    vni assignment mode : SYMMETRIC
    Peer Location       : N/A

Notice that

What is a Router MAC and why do we need it? RFC 9135 defines the Router MAC:

A new EVPN BGP Extended Community called “EVPN Router’s MAC” is introduced here. This new extended community is a transitive extended community with a Type field of 0x06 (EVPN) and a Sub-Type field of 0x03. It may be advertised along with the Encapsulation Extended Community defined in Section 4.1 of [RFC9012].

The EVPN Router’s MAC Extended Community is encoded as an 8-octet value as follows:

This extended community is used to carry the PE’s MAC address for symmetric IRB scenarios, and it is sent with EVPN RT-2. The advertising PE SHALL only attach a single EVPN Router’s MAC Extended Community to a route. In case the receiving PE receives more than one EVPN Router’s MAC Extended Community with a route, it SHALL process the first one in the list and not store and propagate the others.

The Router MAC is used for inter-VNI traffic flows. I will describe symmetric IRB in detail in a future post but for now let’s get back to our task of getting to advertise an IP in combination with MAC for route type 2. For this advertisement, we are expecting:

• One route type 2 route for MAC.
• One route type 2 route for MAC/IP.
• The second route will use MPLS Label2 set to L3 VNI.
• Separate RT for L3 VNI as opposed to L2 VNI.
• Router MAC extended community attached to route with MAC address of SVI used as transit VNI.

To get symmetric IRB working we will need the following:

• Define a VLAN for transit VNI and assign a VNI to it.
• Map the L3 VNI to a VRF.
• Define a SVI for the VLAN created for the transit VNI. This SVI will be used for inter-VNI routing.
• Associate L3VNI to a VRF under the NVE interface.

We will configure this shortly but I find it useful to break things down into their components and sometimes also to provide a faulty or partially complete configuration to verify what each command does. What happens if we just configure a SVI in a VRF?

vrf context Tenant1
  vni 10001
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
!
interface Vlan10
  no shutdown
  vrf member Tenant1
  ip address 198.51.100.1/24

After configuring this, there was no IP address in the type 2 routes:

Leaf1# show bgp l2vpn evpn 
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 11, Local Router ID is 192.0.2.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 192.0.2.3:32777    (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
                      203.0.113.4                       100          0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
                      203.0.113.1                       100      32768 i
*>l[3]:[0]:[32]:[203.0.113.1]/88
                      203.0.113.1                       100      32768 i
*>i[3]:[0]:[32]:[203.0.113.4]/88
                      203.0.113.4                       100          0 i

Route Distinguisher: 192.0.2.6:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
                      203.0.113.4                       100          0 i
*>i                   203.0.113.4                       100          0 i
* i[3]:[0]:[32]:[203.0.113.4]/88
                      203.0.113.4                       100          0 i
*>i                   203.0.113.4                       100          0 i

Could this be related to the ARP cache being empty?

Leaf1# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 0

To verify this, I initiated a ping from a host towards the SVI of Leaf1. The ARP cache is now populated:

Leaf1# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 1
Address         Age       MAC Address     Interface       Flags
198.51.100.11   00:00:08  0050.56ad.8506  Vlan10                

There was no change in perspective of EVPN, though. Still no IP being advertised in route type 2. It would seem that a SVI can’t be host facing for L2 VNI if it is not configured as anycast gateway. I have not been able to verify this in configuration guides, though. Configuring anycast gateway is the standard setup of using host facing SVIs. Let’s enable this:

Leaf1(config)# fabric forwarding anycast-gateway-mac 0001.0001.0001
Leaf1(config)# int vlan10
Leaf1(config-if)# fabric forwarding mode anycast-gateway 

At this point I started to see IP addresses in route type 2:

Leaf1# show bgp l2vpn evpn 
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 12, Local Router ID is 192.0.2.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 192.0.2.3:32777    (L2VNI 10000)
*>i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
                      203.0.113.4                       100          0 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
                      203.0.113.1                       100      32768 i
*>l[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272
                      203.0.113.1                       100      32768 i
*>l[3]:[0]:[32]:[203.0.113.1]/88
                      203.0.113.1                       100      32768 i
*>i[3]:[0]:[32]:[203.0.113.4]/88
                      203.0.113.4                       100          0 i

Route Distinguisher: 192.0.2.6:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.7d68]:[0]:[0.0.0.0]/216
                      203.0.113.4                       100          0 i
*>i                   203.0.113.4                       100          0 i
* i[3]:[0]:[32]:[203.0.113.4]/88
                      203.0.113.4                       100          0 i
*>i                   203.0.113.4                       100          0 i

We can check a route in more detail:

Leaf1# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777    (L2VNI 10000)
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 12
Paths: (1 available, best #1)
Flags: (0x000102) (high32 00000000) on xmit-list, is not in l2rib/evpn

  Advertised path-id 1
  Path type: local, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    203.0.113.1 (metric 0) from 0.0.0.0 (192.0.2.3)
      Origin IGP, MED not set, localpref 100, weight 32768
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8

  Path-id 1 advertised to peers:
    192.0.2.11         192.0.2.12

This route is now being advertised to the RRs (spines), but not making it to other leafs. Why? Let’s take a look at the spine:

Spine1# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/248, version 14
Paths: (1 available, best #0)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is invalid(no RMAC), no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 41) from 192.0.2.101 (192.0.2.3)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8

This route now has the following information in it that was not there until IP got advertised:

• IP Address Length set to 32 instead of zero.
• IP Address set to 198.51.100.11 instead of quad zero.
• Additional route target (extended community) set to 65000:10001 (10001 is the L3 VNI).
• MPLS Label2 set to 10001 (10001 is the L3 VNI).

However, one piece of information is missing. The spine is giving it to us showing “invalid(no RMAC)”. RFC 9135 has the following passage on RMAC:

Each NVE advertises an EVPN RT-2 route with two Route Targets (one corresponding to its MAC-VRF and the other corresponding to its IP-VRF). Furthermore, the EVPN RT-2 is advertised with two BGP Extended Communities. The first BGP Extended Community identifies the tunnel type, and it is called “Encapsulation Extended Community” as defined in [RFC9012], and the second BGP Extended Community includes the MAC address of the NVE (e.g., MACx for NVE1 or MACy for NVE2) as defined in Section 8.1. The EVPN Router’s MAC Extended Community MUST be added when the Ethernet NVO tunnel is used. If the IP NVO tunnel type is used, then there is no need to send this second Extended Community. It should be noted that the IP NVO tunnel type is only applicable to symmetric IRB procedures.

Since we are now advertising IP, and hence two route targets, the Router MAC must be advertised.

For completeness, we can also see this route in our packet capture:

Frame 363: 185 bytes on wire (1480 bits), 185 bytes captured (1480 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 51368, Dst Port: 179, Seq: 233, Ack: 490, Len: 119
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 119
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 96
    Path attributes
        Path Attribute - MP_REACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
            Type Code: MP_REACH_NLRI (14)
            Length: 51
            Address family identifier (AFI): Layer-2 VPN (25)
            Subsequent address family identifier (SAFI): EVPN (70)
            Next hop: 203.0.113.1
            Number of Subnetwork points of attachment (SNPA): 0
            Network Layer Reachability Information (NLRI)
                EVPN NLRI: MAC Advertisement Route
                    Route Type: MAC Advertisement Route (2)
                    Length: 40
                    Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
                    ESI: 00:00:00:00:00:00:00:00:00:00
                    Ethernet Tag ID: 0
                    MAC Address Length: 48
                    MAC Address: 00:50:56:ad:85:06
                    IP Address Length: 32
                    IPv4 address: 198.51.100.11
                    VNI: 10000
                    VNI: 10001
        Path Attribute - ORIGIN: IGP
        Path Attribute - AS_PATH: empty
        Path Attribute - LOCAL_PREF: 100
        Path Attribute - EXTENDED_COMMUNITIES
            Flags: 0xc0, Optional, Transitive, Complete
            Type Code: EXTENDED_COMMUNITIES (16)
            Length: 24
            Carried extended communities: (3 communities)
                Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
                Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
                Encapsulation: VXLAN Encapsulation [Transitive Opaque]

Now that we’ve got this far, let’s supply a fully functional configuration and walk through all of the commands:

There’s a few interesting things going on here. Notice that the SVI for VLAN 100 has no IP address. This is because we only need the MAC address for data plane forwarding. This is the Router MAC that will be advertised as extended community in route type 2. The command

With this configured, we now have valid routes. On Leaf4 we can see the route for the host that is connected to Leaf1:

Leaf4# show bgp l2vpn evpn route-type 2
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 13
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: Tenant1 L3-10001 L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
      Originator: 192.0.2.3 Cluster list: 192.0.2.1 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
      Originator: 192.0.2.3 Cluster list: 192.0.2.2 

  Path-id 1 not advertised to any peer

Notice that the Router MAC is now included. We can also see this in a packet capture:

Frame 903: 193 bytes on wire (1544 bits), 193 bytes captured (1544 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 51368, Dst Port: 179, Seq: 656, Ack: 794, Len: 127
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 127
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 104
    Path attributes
        Path Attribute - MP_REACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
            Type Code: MP_REACH_NLRI (14)
            Length: 51
            Address family identifier (AFI): Layer-2 VPN (25)
            Subsequent address family identifier (SAFI): EVPN (70)
            Next hop: 203.0.113.1
            Number of Subnetwork points of attachment (SNPA): 0
            Network Layer Reachability Information (NLRI)
                EVPN NLRI: MAC Advertisement Route
                    Route Type: MAC Advertisement Route (2)
                    Length: 40
                    Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
                    ESI: 00:00:00:00:00:00:00:00:00:00
                    Ethernet Tag ID: 0
                    MAC Address Length: 48
                    MAC Address: 00:50:56:ad:85:06
                    IP Address Length: 32
                    IPv4 address: 198.51.100.11
                    VNI: 10000
                    VNI: 10001
        Path Attribute - ORIGIN: IGP
        Path Attribute - AS_PATH: empty
        Path Attribute - LOCAL_PREF: 100
        Path Attribute - EXTENDED_COMMUNITIES
            Flags: 0xc0, Optional, Transitive, Complete
            Type Code: EXTENDED_COMMUNITIES (16)
            Length: 32
            Carried extended communities: (4 communities)
                Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
                Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
                Encapsulation: VXLAN Encapsulation [Transitive Opaque]
                EVPN Router's MAC: Router's MAC: 00:ad:e6:88:1b:08 [Transitive EVPN]

This is the MAC of SVI for VLAN 100 on Leaf1:

Leaf1# show int vlan 100 | i Ether
  Hardware is EtherSVI, address is  00ad.e688.1b08

With this there is now a Router MAC populated for NVE peers:

Leaf4# show nve peers det
Details of nve Peers:
----------------------------------------
Peer-Ip: 203.0.113.1
    NVE Interface       : nve1
    Peer State          : Up
    Peer Uptime         : 4d23h
    Router-Mac          : 00ad.e688.1b08
    Peer First VNI      : 10000
    Time since Create   : 4d23h
    Configured VNIs     : 10000-10001
    Provision State     : peer-add-complete
    Learnt CP VNIs      : 10000-10001
    vni assignment mode : SYMMETRIC
    Peer Location       : N/A

Finally, it’s worth highlighting that there are now two route type 2 for MAC address of host connected to Leaf1:

Leaf4# show bgp l2vpn evpn 
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 17, Local Router ID is 192.0.2.6
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
Route Distinguisher: 192.0.2.3:32777
* i[2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216
                      203.0.113.1                       100          0 i
*>i                   203.0.113.1                       100          0 i
*>i[2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272
                      203.0.113.1                       100          0 i
* i                   203.0.113.1                       100          0 i

That was a lot of information to digest! What did we learn?

• To advertise IPs into route type 2, the following is required:
  • Anycast gateway configured and enabled on SVI toward hosts.
  • SVI used for inter-VNI routing.
  • A L3 VNI.
  • VLAN that is assigned the L3 VNI.
  • VRF that is assigned a L3 VNI.
  • L3 VNI associated with NVE.

In a future post we will look at what the IP address in route type 2 can be used for and how it ties into route type 5. Stay tuned!