In the previous post VXLAN/EVPN – Host ARP, I talked about how knowing the MAC/IP of endpoints allows for ARP suppression. In this post we’ll take a look at host mobility. The topology used is the same as in the previous post:

Currently SERVER-1 is connected to LEAF-1. What happens if SERVER-1 moves to LEAF-2? This would be a common scenario for a virtual infrastructure. First let’s take a look at LEAF-4 on what routes we have for SERVER-1:

Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 662
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8
      Originator: 192.0.2.3 Cluster list: 192.0.2.2 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8
      Originator: 192.0.2.3 Cluster list: 192.0.2.1 

  Path-id 1 not advertised to any peer
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 25
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
      Originator: 192.0.2.3 Cluster list: 192.0.2.2 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: Tenant1 L3-10001 L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 Router MAC:00ad.e688.1b08
      Originator: 192.0.2.3 Cluster list: 192.0.2.1 

  Path-id 1 not advertised to any peer

There are two route type 2 routes here as expected. One with MAC and one with MAC/IP. However, there is currently no attribute in this route that signals if a host has moved. This is expected as this host hasn’t moved but what if it did? I will move SERVER-1 in my lab to LEAF-2 and we will try to capture the BGP updates. Before I do that, let’s just verify the MAC address table on LEAF-1 and LEAF-4:

Leaf1# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   10     0050.56ad.040e   dynamic  NA         F      F    Eth1/3
C   10     0050.56ad.7d68   dynamic  NA         F      F    nve1(203.0.113.4)
*   10     0050.56ad.8506   dynamic  NA         F      F    Eth1/3
G   10     00ad.e688.1b08   static   -         F      F    sup-eth1(R)

0050.56ad.8506 is known locally on LEAF-1. This is SERVER-1. It is known via 203.0.113.1 on LEAF-4:

Leaf4# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C   10     0050.56ad.040e   dynamic  NA         F      F    nve1(203.0.113.1)
*   10     0050.56ad.7d68   dynamic  NA         F      F    Eth1/3
C   10     0050.56ad.8506   dynamic  NA         F      F    nve1(203.0.113.1)
G   10     00ad.7083.1b08   static   -         F      F    sup-eth1(R)

To test if the move would be seamless, I initiated a ping on SERVER-4 towards SERVER-1:

ping 198.51.100.11
PING 198.51.100.11 (198.51.100.11) 56(84) bytes of data.
64 bytes from 198.51.100.11: icmp_seq=1 ttl=64 time=4.40 ms
<SNIP>
64 bytes from 198.51.100.11: icmp_seq=51 ttl=64 time=4.92 ms
64 bytes from 198.51.100.11: icmp_seq=148 ttl=64 time=3.70 ms

The move was not seamless. The hypervisor did not generate any GARP or RARP frames so LEAF-2 was not aware of SERVER-1 moving there until I generated some traffic on SERVER-1.

This is what the update looks like on the CLI:

Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.4:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 682
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.2 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:1
      Originator: 192.0.2.4 Cluster list: 192.0.2.2 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.2 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:1
      Originator: 192.0.2.4 Cluster list: 192.0.2.1 

  Path-id 1 not advertised to any peer
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[32]:[198.51.100.11]/272, version 676
Paths: (2 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 3 destination(s)
             Imported paths list: Tenant1 L3-10001 L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.2 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:1
          Router MAC:00ad.f3bb.1b08
      Originator: 192.0.2.4 Cluster list: 192.0.2.1 

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.2 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000 10001
      Extcommunity: RT:65000:10000 RT:65000:10001 ENCAP:8 MAC Mobility Sequence:00:1
          Router MAC:00ad.f3bb.1b08
      Originator: 192.0.2.4 Cluster list: 192.0.2.2 

  Path-id 1 not advertised to any peer

A packet capture of the BGP update that LEAF-2 sent looks like this:

Frame 1862: 201 bytes on wire (1608 bits), 201 bytes captured (1608 bits) on interface ens224, id 2
Ethernet II, Src: 00:ad:f3:bb:1b:08, Dst: 00:ad:b3:fd:1b:08
Internet Protocol Version 4, Src: 192.0.2.102, Dst: 192.0.2.11
Transmission Control Protocol, Src Port: 38924, Dst Port: 179, Seq: 58, Ack: 58, Len: 135
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 135
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 112
    Path attributes
        Path Attribute - MP_REACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
            Type Code: MP_REACH_NLRI (14)
            Length: 51
            Address family identifier (AFI): Layer-2 VPN (25)
            Subsequent address family identifier (SAFI): EVPN (70)
            Next hop: 203.0.113.2
            Number of Subnetwork points of attachment (SNPA): 0
            Network Layer Reachability Information (NLRI)
                EVPN NLRI: MAC Advertisement Route
                    Route Type: MAC Advertisement Route (2)
                    Length: 40
                    Route Distinguisher: 0001c00002048009 (192.0.2.4:32777)
                    ESI: 00:00:00:00:00:00:00:00:00:00
                    Ethernet Tag ID: 0
                    MAC Address Length: 48
                    MAC Address: 00:50:56:ad:85:06
                    IP Address Length: 32
                    IPv4 address: 198.51.100.11
                    VNI: 10000
                    VNI: 10001
        Path Attribute - ORIGIN: IGP
        Path Attribute - AS_PATH: empty
        Path Attribute - LOCAL_PREF: 100
        Path Attribute - EXTENDED_COMMUNITIES
            Flags: 0xc0, Optional, Transitive, Complete
            Type Code: EXTENDED_COMMUNITIES (16)
            Length: 40
            Carried extended communities: (5 communities)
                Route Target: 65000:10000 [Transitive 2-Octet AS-Specific]
                Route Target: 65000:10001 [Transitive 2-Octet AS-Specific]
                Encapsulation: VXLAN Encapsulation [Transitive Opaque]
                MAC Mobility: Movable MAC [Transitive EVPN]
                    Type: Transitive EVPN (0x06)
                    Subtype (EVPN): MAC Mobility (0x00)
                    Flags: 0x00
                    Sequence number: 1
                EVPN Router's MAC: Router's MAC: 00:ad:f3:bb:1b:08 [Transitive EVPN]

There is an additional community in this update that we haven’t seen before, MAC Mobility, and the sequence number is set to 1. What is this community and why haven’t we seen it before? RFC 7432 section 15 describes MAC mobility:

It is possible for a given host or end-station (as defined by its MAC address) to move from one Ethernet segment to another; this is referred to as ‘MAC Mobility’ or ‘MAC move’, and it is different from the multihoming situation in which a given MAC address is reachable via multiple PEs for the same Ethernet segment.

<SNIP>

If local learning is performed using the data plane, these PEs will not be able to detect that the MAC address has moved to another Ethernet segment, and the receipt of MAC/IP Advertisement routes, with the MAC Mobility extended community attribute, from other PEs serves as the trigger for these PEs to withdraw their advertisements. If local learning is performed using the control or management planes, these interactions serve as the trigger for these PEs to withdraw their advertisements. In a situation where there are multiple moves of a given MAC, possibly between the same two Ethernet segments, there may be multiple withdrawals and re-advertisements. In order to ensure that all PEs in the EVPN instance receive all of these correctly through the intervening BGP infrastructure, introducing a sequence number into the MAC Mobility extended community attribute is necessary.

To summarize the RFC, the MAC Mobility community works like this:

  • The first time a MAC address is advertised in route type 2, the MAC Mobility community is not set.
  • When host moves to a new switch, the switch learns MAC locally and then generates route type 2 with MAC Mobility community and sequence number set to 1.
  • The switch where host was previously located sends update to withdraw its route.
  • If host moves again, sequence number is increased to 2 and the same process repeats.

If there are several routes, then the route with the highest sequence number is selected as best. Note that a switch will also monitor how many times a MAC has moved to avoid “MAC flapping”. On NX-OS, a if a host moves 5 times or more within 180 seconds, the switch starts a 30 second lock before checking again. The 30-second lock can happen 5 times within 24 hours before the MAC becomes permanently frozen and will require user intervention.

Below we can see that LEAF-1 is withdrawing its route:

Frame 1828: 135 bytes on wire (1080 bits), 135 bytes captured (1080 bits) on interface ens193, id 6
Ethernet II, Src: 00:ad:e6:88:1b:08, Dst: 00:ad:7b:30:1b:08
Internet Protocol Version 4, Src: 192.0.2.101, Dst: 192.0.2.12
Transmission Control Protocol, Src Port: 179, Dst Port: 18728, Seq: 39, Ack: 207, Len: 69
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 69
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 46
    Path attributes
        Path Attribute - MP_UNREACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
            Type Code: MP_UNREACH_NLRI (15)
            Length: 42
            Address family identifier (AFI): Layer-2 VPN (25)
            Subsequent address family identifier (SAFI): EVPN (70)
            Withdrawn Routes
                EVPN NLRI: MAC Advertisement Route
                    Route Type: MAC Advertisement Route (2)
                    Length: 37
                    Route Distinguisher: 0001c00002038009 (192.0.2.3:32777)
                    ESI: 00:00:00:00:00:00:00:00:00:00
                    Ethernet Tag ID: 0
                    MAC Address Length: 48
                    MAC Address: 00:50:56:ad:85:06
                    IP Address Length: 32
                    IPv4 address: 198.51.100.11
                    1000 0000 0000 0000 0000 .... = MPLS Label 1: 524288

LEAF-2 now has SERVER-1 MAC locally:

Leaf2# show mac address-table vlan 10
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
C   10     0050.56ad.040e   dynamic  NA         F      F    nve1(203.0.113.1)
C   10     0050.56ad.7d68   dynamic  NA         F      F    nve1(203.0.113.4)
*   10     0050.56ad.8506   dynamic  NA         F      F    Eth1/3
*   10     0050.56ad.b4a4   dynamic  NA         F      F    Eth1/3
G   10     00ad.f3bb.1b08   static   -         F      F    sup-eth1(R)

After this I moved SERVER-1 back to LEAF-1. Note that the sequence number then has been increased to 2:

Leaf4# show bgp l2vpn evpn 0050.56ad.8506
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 192.0.2.3:32777
BGP routing table entry for [2]:[0]:[0]:[48]:[0050.56ad.8506]:[0]:[0.0.0.0]/216, version 698
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.12 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:2
      Originator: 192.0.2.3 Cluster list: 192.0.2.2 

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 1 destination(s)
             Imported paths list: L2-10000
  AS-Path: NONE, path sourced internal to AS
    203.0.113.1 (metric 81) from 192.0.2.11 (192.0.2.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10000
      Extcommunity: RT:65000:10000 ENCAP:8 MAC Mobility Sequence:00:2
      Originator: 192.0.2.3 Cluster list: 192.0.2.1 

  Path-id 1 not advertised to any peer

That’s how MAC mobility works! You may be wondering, though. What’s the maximum sequence number? The sequence number is 32 bits so you do the math 🙂 What happens if a host disappears and then comes back again? If a host has timed out from MAC address tables, ARP cache, and BGP tables, when the host comes back it will be advertised without the MAC mobility community. It is treated as a new host.

In this post we learned that hosts can and are expected to move in a VXLAN network. The extended community MAC Mobility assists in keeping track of the most up to date information.

VXLAN/EVPN – Host mobility
Tagged on:     

5 thoughts on “VXLAN/EVPN – Host mobility

  • January 11, 2024 at 5:32 am
    Permalink

    This is one the best posts you’ve ever done on EVPN; Very well done!

    Beautiful presentation as always.

    Reply
    • January 12, 2024 at 9:29 am
      Permalink

      Thanks, Mark!

      Reply
  • January 13, 2024 at 11:19 am
    Permalink

    Informative and great posts as usual. Thanks Daniel !

    Reply
    • January 17, 2024 at 4:44 pm
      Permalink

      Thank you!

      Reply
  • February 13, 2024 at 2:48 am
    Permalink

    very well explained

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *