A Quick Look at Cisco FabricPath

Cisco FabricPath is a proprietary protocol that uses ISIS to populate a “routing table” that is used for layer 2 forwarding.

Whether we like or not, there is often a need for layer 2 in the Datacenter for the following reasons:

• Some applications or protocols require to be layer 2 adjacent
• It allows for virtual machine/workload mobility
• Systems administrators are more familiar with switching than routing

A traditional network with layer 2 and Spanning Tree (STP) has a lot of limitations that makes it less than optimal for a Datacenter:

• Local problems have a network-wide impact
• The tree topology provides limited bandwidth
• The tree topology also introduces suboptimal paths
• MAC address tables don’t scale

In the traditional network, because STP is running, a tree topology is built. This works better for for flows that are North to South, meaning that traffic passes from the Access layer, up to Distribution, to the Core and then down to Distribution and to the Access layer again. This puts a lot of strain on Core interconnects and is not well suited for East-West traffic which is the name for server to server traffic.

A traditional Datacenter design will look something like this:

If we want end-to-end L2, we could build a network like this:

What would be the implications of building such a network though?

• Large failure domain
• Unknown unicast and broadcast flooding through large parts of the network
• A large number of STP instances needed unless using MST
• Topology change will have a large impact on the network and may cause flooding
• Large MAC address tables
• Difficult to troubleshoot
• A very brittle network

So let’s agree that we don’t want to build a network like this. What other options do we have if we still need layer 2? One of the options is Cisco FabricPath.

FabricPath provides the following benefits:

• Reduction/elimination of STP
• Better stability and convergence characteristics
• Simplified configuration
• Leverage parallell paths
• Deterministic throughput and latency using typical designs
• “VLAN anywhere” flexibility

The FabricPath control plane consists of the following elements:

• Routing table – Uses ISIS to learn Switch IDS (SIDs) and build a routing table
• Multidestination trees – Elects roots and builds multidestination trees
• Mroute table – IGMP snooping learns group membership at the edge, Group Member LSPs (GM-LSPs) are flooded by ISIS into the fabric

Observe that LSPs has nothing to do with MPLS in this case and that this is not MAC based routing, routing is based on SIDs.

FabricPath ISIS learns the shortest path to each SID based on link metrics/path cost. Up to 16 equal (ECMP) routes can be installed. Choosing a path is based on a hashing function using Src IP/Dst IP/L4/VLAN which should be good for avoiding polarization.

FabricPath supports multidestination trees with the following capabilities:

• Multidestination traffic is contained to a tree topology, a network-wide identifier (Ftag) is assigned to each tree
• A root switch is elected for each multidestination tree
• Multipathing is supported through multiple trees

Note that root here has nothing to do with STP, think of it in terms of multicast routing.

Multidestination trees do not dictate forwarding for unicast, only for multidestination packets.

The FabricPath data plane behaves according to the following forwarding rules:

• MAC table – Hardware performs MAC lookup at CE/FabricPath edge only
• Switch table – Hardware performs destination SID lookups to forward unicast frames to other switches
• Multidestination table – A hashing function selects the tree, multidestination table identifies on which interfaces to flood based on selected tree

The Ftag used in FabricPath identifies which ISIS topology to use for unicast packets and for multidestination packets, which tree to use.

If a FabricPath switch belongs to a topology, all VLANs of that topology should be configured on that switch to avoid blackholing issues.

FabricPath supports 802.1p but can also match/set DSCP and match on other L2/L3/L4 information.

With FabricPath, edge switches only need to learn:

• Locally connected host MACs
• MACs with which those hosts are bidirectionally communicating

This reduces the MAC address table capacity requirements on Edge switches.

FabricPath Designs

There are different designs that can be used together with FabricPath. The first one is routing at the Aggregation layer.

The first design is the most classic one where STP has been replaced by FP in the Access layer and routing is used above the Aggregation layer.

This design has the following characteristics:

• Evolution of current design practices
• The Aggregation layer functions as FabricPath spine and L2/L3 boundary
  – FabricPath switching for East – West intra VLAN traffic
  – SVIs for East – West inter VLAN traffic
  – Routed uplinks for North – South routed flows

• Access layer provides pure L2 functions
  – FabricPath core ports facing Aggregation layer
  – CE edge ports facing hosts
  – Optionally vPC+ can be used for active/active host connections

This design is the simplest option and is an extension of regular Access/Aggregation designs. It provides the following benefits:

• Simplified configuration
• Removal of STP
• Traffic distribution over all uplinks without the use of vPC
• Active/active gateways
• “VLAN anywhere” at the Access layer
• Topological flexibility
  – Direct-path forwarding option
  – Easily provision additional AccessAggregation bandwidth
  – Easily deploy L4-L7 services
  – Can use vPC+ towards legacy Access switches

There is also the centralized routing design which looks like the following:

Centralized routing has the following characteristics:

• Traditional Aggregation layer becomes pure FabricPath spine
  – Provides uniform any-to-any connectivity between leaf switches
  – In simplest case, only FabricPath switching occurs in spine
  – Optionally, some CE edge ports exist to provide external router connections

• FabricPath leaf switches, connecting to spine, have specific “personality”
  – Most of the leaf switches will provide server connectivity, like traditional access switches in “Routing at Aggregation” designs
  – Two or more leaf switches provide L2/L3 boundary, inter-VLAN routing and North-South routing
  – Other or same leaf switches may provide L4-L7 services

• Decouples L2/L3 boundary and L4-L7 services provisioning from Spine
  – Simplifies Spine design

The different traffic flows in this design looks like the following:

Another design is the multi-pod design which can look like the following:

The multi-pod design has the following characteristics:

• Allows for more elegant DC-wide versus pod-local VLAN definition/isolation
  – No need for pod-local VLANs to exist in core
  – Can support VLAN id reuse in multiple pods

• Define FabricPath VLANs -> map VLANs to topology -> map topology to FabricPath core port(s)
• Default topology always includes all FabricPath core ports
  – Map DC-wide VLANs to default topology

• Pod-local core ports also mapped to pod-local topology
  – Map pod-local VLANs to pod-local topology

This post briefly describes Cisco FabricPath which is a technology for building scalable L2 topologies, allowing for more bisectional bandwidth to support East-West flows which are common in Datacenters. To dive deeper into FabricPath, visit the Cisco Live website.