In the previous post VXLAN/EVPN – Host ARP, I talked about how knowing the MAC/IP of endpoints allows for ARP suppression. In this post we’ll take a look at host mobility. The topology used is the same as in the previous post:
Currently SERVER-1 is connected to LEAF-1. What happens if SERVER-1 moves to LEAF-2? This would be a common scenario for a virtual infrastructure. First let’s take a look at LEAF-4 on what routes we have for SERVER-1:
Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 662
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8
Originator: 192.0.2.3 Cluster list: 192.0.2.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8
Originator: 192.0.2.3 Cluster list: 192.0.2.1
Path-id 1 not advertised to any peer
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 25
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
Originator: 192.0.2.3 Cluster list: 192.0.2.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 3 destination(s)
Imported paths list: Tenant1 L3-10001 L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
Originator: 192.0.2.3 Cluster list: 192.0.2.1
Path-id 1 not advertised to any peer
There are two route type 2 routes here as expected. One with MAC and one with MAC/IP. However, there is currently no attribute in this route that signals if a host has moved. This is expected as this host hasn’t moved but what if it did? I will move SERVER-1 in my lab to LEAF-2 and we will try to capture the BGP updates. Before I do that, let’s just verify the MAC address table on LEAF-1 and LEAF-4:
Leaf1# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 0050.56ad.040e dynamic NA F F Eth1/3
C 10 0050.56ad.7d68 dynamic NA F F nve1(203.0.113.4)
* 10 0050.56ad.8506 dynamic NA F F Eth1/3
G 10 00ad.e688.1b08 static - F F sup-eth1(R)
0050.56ad.8506 is known locally on LEAF-1. This is SERVER-1. It is known via 203.0.113.1 on LEAF-4:
Leaf4# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 10 0050.56ad.040e dynamic NA F F nve1(203.0.113.1)
* 10 0050.56ad.7d68 dynamic NA F F Eth1/3
C 10 0050.56ad.8506 dynamic NA F F nve1(203.0.113.1)
G 10 00ad.7083.1b08 static - F F sup-eth1(R)
To test if the move would be seamless, I initiated a ping on SERVER-4 towards SERVER-1:
ping 198.51.100.11 PING 198.51.100.11 (198.51.100.11) 56(84) bytes of data. 64 bytes from 198.51.100.11: icmp_seq=1 ttl=64 time=4.40 ms <SNIP> 64 bytes from 198.51.100.11: icmp_seq=51 ttl=64 time=4.92 ms 64 bytes from 198.51.100.11: icmp_seq=148 ttl=64 time=3.70 ms
The move was not seamless. The hypervisor did not generate any GARP or RARP frames so LEAF-2 was not aware of SERVER-1 moving there until I generated some traffic on SERVER-1.
This is what the update looks like on the CLI:
Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.4:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 682
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.2 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:1
Originator: 192.0.2.4 Cluster list: 192.0.2.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.2 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:1
Originator: 192.0.2.4 Cluster list: 192.0.2.1
Path-id 1 not advertised to any peer
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 676
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 3 destination(s)
Imported paths list: Tenant1 L3-10001 L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.2 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:1
Router MAC:00ad.f3bb.1b08
Originator: 192.0.2.4 Cluster list: 192.0.2.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.2 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000 10001
Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:1
Router MAC:00ad.f3bb.1b08
Originator: 192.0.2.4 Cluster list: 192.0.2.2
Path-id 1 not advertised to any peer
A packet capture of the BGP update that LEAF-2 sent looks like this:
Frame 1862: 201 bytes on wire (1608 bits), 201 bytes captured (1608 bits) on interface ens224, id 2
Ethernet II, Src: 00:ad:f3:bb:1b:08, Dst: 00:ad:b3:fd:1b:08
Internet Protocol Version 4, Src: 192.0.2.102, Dst: 192.0.2.11
Transmission Control Protocol, Src Port: 38924, Dst Port: 179, Seq: 58, Ack: 58, Len: 135
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 135
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 112
Path attributes
Path Attribute - MP_REACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_REACH_NLRI (14)
Length: 51
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Next hop: 203.0.113.2
Number of Subnetwork points of attachment (SNPA): 0
Network Layer Reachability Information (NLRI)
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 40
Route Distinguisher: 0001c00002048009 (192.0.2.4:32777)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: 00:50:56:ad:85:06
IP Address Length: 32
IPv4 address: 198.51.100.11
VNI: 10000
VNI: 10001
Path Attribute - ORIGIN: IGP
Path Attribute - AS_PATH: empty
Path Attribute - LOCAL_PREF: 100
Path Attribute - EXTENDED_COMMUNITIES
Flags: 0xc0, Optional, Transitive, Complete
Type Code: EXTENDED_COMMUNITIES (16)
Length: 40
Carried extended communities: (5 communities)
Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
Encapsulation: VXLAN Encapsulation [Transitive Opaque]
MAC Mobility: Movable MAC [Transitive EVPN]
Type: Transitive EVPN (0x06)
Subtype (EVPN): MAC Mobility (0x00)
Flags: 0x00
Sequence number: 1
EVPN Router's MAC: Router's MAC: 00:ad:f3:bb:1b:08 [Transitive EVPN]
There is an additional community in this update that we haven’t seen before, MAC Mobility, and the sequence number is set to 1. What is this community and why haven’t we seen it before? RFC 7432 section 15 describes MAC mobility:
It is possible for a given host or end-station (as defined by its MAC address) to move from one Ethernet segment to another; this is referred to as ‘MAC Mobility’ or ‘MAC move’, and it is different from the multihoming situation in which a given MAC address is reachable via multiple PEs for the same Ethernet segment.
<SNIP>If local learning is performed using the data plane, these PEs will not be able to detect that the MAC address has moved to another Ethernet segment, and the receipt of MAC/IP Advertisement routes, with the MAC Mobility extended community attribute, from other PEs serves as the trigger for these PEs to withdraw their advertisements. If local learning is performed using the control or management planes, these interactions serve as the trigger for these PEs to withdraw their advertisements. In a situation where there are multiple moves of a given MAC, possibly between the same two Ethernet segments, there may be multiple withdrawals and re-advertisements. In order to ensure that all PEs in the EVPN instance receive all of these correctly through the intervening BGP infrastructure, introducing a sequence number into the MAC Mobility extended community attribute is necessary.
To summarize the RFC, the MAC Mobility community works like this:
- The first time a MAC address is advertised in route type 2, the MAC Mobility community is not set.
- When host moves to a new switch, the switch learns MAC locally and then generates route type 2 with MAC Mobility community and sequence number set to 1.
- The switch where host was previously located sends update to withdraw its route.
- If host moves again, sequence number is increased to 2 and the same process repeats.
If there are several routes, then the route with the highest sequence number is selected as best. Note that a switch will also monitor how many times a MAC has moved to avoid “MAC flapping”. On NX-OS, a if a host moves 5 times or more within 180 seconds, the switch starts a 30 second lock before checking again. The 30-second lock can happen 5 times within 24 hours before the MAC becomes permanently frozen and will require user intervention.
Below we can see that LEAF-1 is withdrawing its route:
Frame 1828: 135 bytes on wire (1080 bits), 135 bytes captured (1080 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 179, Dst Port: 18728, Seq: 39, Ack: 207, Len: 69
Border Gateway Protocol - UPDATE Message
Marker: ffffffffffffffffffffffffffffffff
Length: 69
Type: UPDATE Message (2)
Withdrawn Routes Length: 0
Total Path Attribute Length: 46
Path attributes
Path Attribute - MP_UNREACH_NLRI
Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
Type Code: MP_UNREACH_NLRI (15)
Length: 42
Address family identifier (AFI): Layer-2 VPN (25)
Subsequent address family identifier (SAFI): EVPN (70)
Withdrawn Routes
EVPN NLRI: MAC Advertisement Route
Route Type: MAC Advertisement Route (2)
Length: 37
Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
ESI: 00:00:00:00:00:00:00:00:00:00
Ethernet Tag ID: 0
MAC Address Length: 48
MAC Address: 00:50:56:ad:85:06
IP Address Length: 32
IPv4 address: 198.51.100.11
1000 0000 0000 0000 0000 .... = MPLS Label 1: 524288
LEAF-2 now has SERVER-1 MAC locally:
Leaf2# show mac address-table vlan 10
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C 10 0050.56ad.040e dynamic NA F F nve1(203.0.113.1)
C 10 0050.56ad.7d68 dynamic NA F F nve1(203.0.113.4)
* 10 0050.56ad.8506 dynamic NA F F Eth1/3
* 10 0050.56ad.b4a4 dynamic NA F F Eth1/3
G 10 00ad.f3bb.1b08 static - F F sup-eth1(R)
After this I moved SERVER-1 back to LEAF-1. Note that the sequence number then has been increased to 2:
Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 698
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:2
Originator: 192.0.2.3 Cluster list: 192.0.2.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L2-10000
AS-Path: NONE, path sourced internal to AS
203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 10000
Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:2
Originator: 192.0.2.3 Cluster list: 192.0.2.1
Path-id 1 not advertised to any peer
That’s how MAC mobility works! You may be wondering, though. What’s the maximum sequence number? The sequence number is 32 bits so you do the math 🙂 What happens if a host disappears and then comes back again? If a host has timed out from MAC address tables, ARP cache, and BGP tables, when the host comes back it will be advertised without the MAC mobility community. It is treated as a new host.
In this post we learned that hosts can and are expected to move in a VXLAN network. The extended community MAC Mobility assists in keeping track of the most up to date information.
This is one the best posts you’ve ever done on EVPN; Very well done!
Beautiful presentation as always.
Thanks, Mark!
Informative and great posts as usual. Thanks Daniel !
Thank you!
very well explained
Does MAC Mobility decide PATH selection? because when I insert BGP NULL route in route-map with BGP community I found it said this is not a best path because of MAC Mobility. How do I override and make it best PATH ?