Modifying Administrative Distance of Specific BGP Route

In one of the Discords that I’m in there was a user with a complex network consisting of a mix of DMVPN, BGP over MPLS VPN circuits, and SD-WAN. For some prefixes, the path via the private MPLS is preferred, for others, the SD-WAN path. Now, if a prefix is available in two different protocols, BGP vs Overlay Management Protocol (OMP), there is nothing we can do in BGP or OMP to modify which one gets installed into the Routing Information Base (RIB). This is no different than if EIGRP and OSPF were competing to install a prefix into the RIB, the protocol with the lower Administrative Distance (AD) would have its route installed.

The default AD values used on a Cisco device for these protocols are:

• eBGP – 20
• iBGP – 200
• OMP – 251

Based on the AD, OMP will always lose out. It is of course possible to change the AD of BGP, but that would have an effect of all prefixes and we lose the ability to have some prefixes preferred via BGP and others via OMP. I had never changed the AD of a specific BGP prefix before, so I turned to Twitter to see what my options were and got some good responses back. It turns out that there is a way of modifying AD of prefixes from a neighbor and that it’s not very well documented. To test this, I created a simple lab with three routers, R1, R2, and R3, that are Catalyst8000v devices running IOS-XE 17.6.3. To save time, I have not setup SD-WAN in the lab but I will use a static route with AD of 251 to simulate OMP. R1 will point this static route towards R3. R1 will also run BGP towards R2. R3 will advertise two routes via BGP:

• 198.51.100.0/24
• 203.0.113.0/24

R1 is receiving the routes via BGP:

R1#sh bgp ipv4 uni
BGP table version is 3, local router ID is 192.0.2.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, 
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, 
              x best-external, a additional-path, c RIB-compressed, 
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>   198.51.100.0     192.0.2.2                0             0 64512 i
 *>   203.0.113.0      192.0.2.2                0             0 64512 i

These routes are installed in the RIB:

R1#show ip route 198.51.100.0
Routing entry for 198.51.100.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 64512, type external
  Last update from 192.0.2.2 00:03:14 ago
  Routing Descriptor Blocks:
  * 192.0.2.2, from 192.0.2.2, 00:03:14 ago
      opaque_ptr 0x7FC3B83EC038 
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 64512
      MPLS label: none
R1#show ip route 203.0.113.0 
Routing entry for 203.0.113.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 64512, type external
  Last update from 192.0.2.2 00:02:54 ago
  Routing Descriptor Blocks:
  * 192.0.2.2, from 192.0.2.2, 00:02:54 ago
      opaque_ptr 0x7FC3B83EC038 
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 64512
      MPLS label: none

Notice that the distance is set to 20 which is the AD of eBGP. We can traceroute to confirm that traffic is going through R2 and then to R3:

R1#traceroute 198.51.100.1
Type escape sequence to abort.
Tracing the route to 198.51.100.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.0.2.2 1 msec 1 msec 0 msec
  2 192.0.2.10 2 msec *  1 msec

Now, let’s see if we can modify just one of these routes to go directly via R3 instead of via R2. The first step is to create an access-list:

R1(config)#ip access-list standard MODIFY_AD_R2_198-NET
R1(config-std-nacl)#permit 198.51.100.0 0.0.0.255

This ACL will match any prefix within 198.51.100.0/24, regardless of mask. Now here comes the tricky part, to modify AD we do it under the IPv4 address family in BGP:

R1(config)#router bgp 65000
R1(config-router)#address-family ipv4
R1(config-router-af)#distance ?
  <1-255>  Administrative distance
  bgp      BGP distance

The trick here is to NOT use the distance bgp command but rather the distance command. Let’s have a look:

R1(config)#router bgp 65000
R1(config-router)#address-family ipv4
R1(config-router-af)#distance ?
  <1-255>  Administrative distance
  bgp      BGP distance

R1(config-router-af)#distance 252 ?
  A.B.C.D  IP Source address

R1(config-router-af)#distance 252 192.0.2.2 ?
  A.B.C.D  Wildcard bits

R1(config-router-af)#distance 252 192.0.2.2 0.0.0.0 ?
  <1-99>       IP Standard access list number
  <1300-1999>  IP Standard expanded access list number
  WORD         Standard access-list name
  <cr>         <cr>

R1(config-router-af)#distance 252 192.0.2.2 0.0.0.0 MODIFY_AD_R2_198-NET

The syntax here is as follows:

• AD to set on prefixes matching
• Neighbor IP of BGP peer(s)
• Wild card to match one or more peers
• Access-list for the prefixes to be modified

Did this have any effect? Let’s take a look:

R1#show ip route 198.51.100.0   
Routing entry for 198.51.100.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 64512, type external
  Last update from 192.0.2.2 00:00:09 ago
  Routing Descriptor Blocks:
  * 192.0.2.2, from 192.0.2.2, 00:00:09 ago
      opaque_ptr 0x7FC3B83EC038 
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 64512
      MPLS label: none

Now this doesn’t look very promising, does it? What’s the problem here? It seems that this mechanism perhaps is only triggered when peer comes up so clearing the BGP session may be needed to trigger it:

R1#clear bgp ipv4 uni 192.0.2.2

Let’s check the routing table again:

R1#show ip route 198.51.100.0  
Routing entry for 198.51.100.0/24
  Known via "static", distance 251, metric 0
  Routing Descriptor Blocks:
  * 192.0.2.6
      Route metric is 0, traffic share count is 1
R1#show ip route 203.0.113.0 
Routing entry for 203.0.113.0/24
  Known via "bgp 65000", distance 20, metric 0
  Tag 64512, type external
  Last update from 192.0.2.2 00:00:37 ago
  Routing Descriptor Blocks:
  * 192.0.2.2, from 192.0.2.2, 00:00:37 ago
      opaque_ptr 0x7FC3B83EC038 
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 64512
      MPLS label: none

Now this looks better! Finally, let’s confirm with traceroute:

R1#traceroute 198.51.100.1
Type escape sequence to abort.
Tracing the route to 198.51.100.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.0.2.6 2 msec *  1 msec
R1#traceroute 203.0.113.1 
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 192.0.2.2 1 msec 1 msec 1 msec
  2 192.0.2.10 2 msec *  1 msec

Traffic to the first prefix is direct and for the other it goes via R2 first. This feature does not seem to be well known and poorly documented so I hope this post can help those that have run into a scenario where they need to modify AD of specific prefixes. Thanks to all the people at Twitter that responded to my query.