Network Campus Design

Introduction

Modern networks need to be enabled for voice and video. These applications
do not tolerate a lot of loss before quality becomes unacceptable. This
requires us to build networks that are scalable, resilient and converge
quickly. This post will describe key points of building a network that
fullfills those requirements.

Hierarchical Network Design

It’s important to think of the network in terms of building blocks and
hierarchy. Define different building blocks like campus, small remote site,
medium remote site and large remote site so that not every new network
needs an unique design. Building a hierarchical network will:

• Make it easier to understand, grow and troubleshoot
• Create small fault domains, clear separation of layer 2 and 3.
• Allows for load sharing and redundancy
• Deterministic traffic patterns and convergence

The key point is to not end up with a network looking like this:

Like anyone working in the real world I know that budget constraints, lack of
available fibre or many other factors can limit our network designs. We can’t
always win but we should make it clear to the company/management that we can’t
support voice and video unless we design the network as it should be. Do you want
to be on call for a network that is poorly designed and maybe you are the only
one that knows how it works? In Optimal Routing Design, Russ White talks about
the 2 AM test. If someone calls you up at 2 AM do you know how your network works?
If you don’t it’s a sign that it is too complex.

The different building blocks of a hierarchical network

Traditionally networks have been built in a three tier model. This model
consists of access, distribution and core. In smaller networks it may be
acceptable to have a layer that acts as both distribution and core and
this is called the collapsed core model.

Access Layer

The role of the access layer is to:

• Provide connectivity into the network
• Enforce security to prevent ARP/IP spoofing
• Boundary for trust for QoS model
• Provide PoE for phones and access points

Distribution layer

The role of the distribution layer is to:

• Aggregate wiring closets(access layer) and uplinks to core
• Provide high availability, load sharing and QoS
• Protect the core from high density peering and problems in access layer
• Summarize routes towards core and fast convergence
• Provide first hop redundancy towards the access layer

Core Layer

The role of the core layer is to:

• Provide connectivity between all the building blocks
• Provide high performance and high availability
• Aggregate the distribution layer
• A separate core layer helps with scalability

When do I need a core layer?

There is no text book answer to this question but consider the following
topology:

There are currently two building blocks. Every distribution layer switch has
three IGP neighbors and 3 links. What happens if we add another building
block?

The network went from three IGP peers to five IGP peers. Also the total number
of links went to 15 from previously 6. You can see that this gets out of hand
pretty quickly.

Different Campus Designs

There are a few common designs that can be used to build the campus access and
distribution layer. Which design that fits best will depend on if there is a
need to span VLANs and how modern equipment you have in your network.

Layer 3 Distribution

This design has no VLANs spanning the switches. This is what we want to have
but usually there is some requirement that keeps us from doing this. It could be
some application, Vmotion or maybe one common wireless network across the entire
campus. There is no layer 2 loop in this design which means we are not relying
on STP for convergence. Here are some points to consider for this design:

• Tune CEF to avoid polarization leading to underusing links
• Summarize routes towards the core
• Don’t peer IGP across links unless you intend to use them
• Set the trunk mode to on/nonegotiate
• Ports toward users hardcoded to access and enable portfast
• Configure root guard or BPDU guard towards users
• Enable security features such as DHCP snooping and DAI

Layer 2 Distribution

It’s quite common that some VLANs need to span the campus. This means that there
must exist an L2 link between distribution switches. There is now a loop in the
topology so convergence is dependant on spanning tree. Some points to consider
for this design:

• Tune CEF to avoid polarization leading to underusing links
• Summarize routes towards the core
• Don’t peer IGP across links unless you intend to use them
• Set the trunk mode to on/nonegotiate
• Ports toward users hardcoded to access and enable portfast
• Configure root guard or BPDU guard towards users
• Enable security features such as DHCP snooping and DAI
• Align STP Root and HSRP primary on the same distribution switch
• Put Root Guard on downlinks (facing access switches)
• Put Loop Guard on uplinks (facing distribution switches)

Routed Access

The routed access design has no layer 2 links. It’s all routing which means
convergence is fast, no links are blocking and equal cost routing can be used.
The drawback is that no VLANs can span the topology. If MPLS is enabled in the core
some VLANs could still be able to span through the use of EoMPLS, VPLS etc. Key
points to consider for this design:

• How much more will routed access cost me?
• Do I need the performance/convergence gain?
• Do I have the need to span any VLANs?
• How many routes do my access layer devices support?
• Summarize routes towards the core
• Summarize routes towards the access
• Tune CEF to avoid polarization
• Don’t peer IGP across links unless you intend to use them
• Ports toward users hardcoded to access and enable portfast
• Configure root guard or BPDU guard towards users
• Enable security features such as DHCP snooping and DAI

Layer 2 Distribution with MLAG

Newer designs can utilize newer features like stacking, VSS and vPC. This means that
VLANs can span access switches but there is no physical loop because MLAG is used.
This gives us the advantage of a layer 2 distribution without the disadvantage of
relying on spanning tree for convergence. There is no need to run HSRP becaues the
distribution layer is acting as one device. The key points are similar to the layer 2
distribution:

• Tune CEF to avoid polarization leading to underusing links
• Summarize routes towards the core
• Set the trunk mode to on/nonegotiate
• Ports toward users hardcoded to access and enable portfast
• Configure root guard or BPDU guard towards users
• Enable security features such as DHCP snooping and DAI
• Put Root Guard on downlinks (facing access switches)
• Put Loop Guard on uplinks (facing distribution switches)

If doing a new design I would definitely go with some form of stacking or VSS or vPC
if deploying Nexus switches. This gives us the flexibility of using layer 2 in distribution
but still not needing to rely on STP and FHRP for convergence.

Recommendations for Fast Convergence

• Use only point to point interconnections
• Use fiber between all devices for fast convergence (debounce timer)
• Tune the carrier delay timer
• When possible use IP configuration on phsyical interface over SVI

I did a separate post on Detecting Network Failure which goes into more detail
on detecting failure.

Why should physical interfaces be used over SVI? The following steps take place when
converging on a physical interface:

1. Link Down
2. Interface Down
3. Routing Update

When using an SVI there are some additional steps however:

1. Link Down
2. Interface Down
3. Autostate
4. SVI Down
5. Routing Update

When using an SVI when the link goes down the switch must check if there are any
other ports up with that VLAN configured. If it is the SVI won’t be brought down.
Even if there isn’t it takes time to go through all the interfaces before declaring
the SVI down. This can worsen convergence by a good 200 ms. If you do use an SVI
then make sure that it is point to point so that it’s not allowed on any other
links than the link connecting the two switches.

Recommendations for Spanning Tree

• Don’t span VLANs across switches unless neccessary
• Use RSTP or MST for best convergence
• Even if you have no loops, STP is needed to protect against user side loops
• STP can protect against misconfiguration or hardware failures creating loops

Layer 2 Hardening

Cisco recommends the following features for hardening layer 2 in a campus design:

I agree with most of this but there are some caveats.

One issue is with Root Guard. Why do we run Root Guard? To protect against another
switch dictating the bridging topology. I would set the root to a priority of 0 and the
secondary root to a priority of 4096. That should provide protection and if you don’t
trust your employees to not mess up the network that is an education problem or
management problem. Restrict user accounts in Tacacs what they can’t do to remove
potentially dangerous commands such as switchport trunk allowed vlan, no router bgp,
no router ospf and so on.

So what is the issue with Root Guard? The STP root will also be the HSRP primary device.
Because the network is designed with Equal Cost Multi Paths (ECMP) some traffic may
arrive at the standby HSRP router. This is the network without blocking links:

No issues so far except that the crosslink is being used but it’s not a major deal.
But what happens if the links between the HSRP primary and standby fails?

The access switches are sending superior BPDUs but the secondary distribution switch will
block the link due to Root Guard being implemented. This means that any traffic arriving
at the secondary distribution switch destined for the access layer switches will be
black holed. That is why I would not implement Root Guard towards the access layer.

Recommendations for Layer 3

Here are some recommendations for Layer 3:

• Build triangles not squares for deterministic convergence
• Use passive-interface default and only peer on links used for transit
• Design the network with dual Layer 3 paths for resiliency
• Summarize from the distribution to the core to cut down on flooding and Active queries
• Tune CEF to avoid polarization of linkz

What happens if we design with squares?

If a device goes down the network has to rely on flooding of updates or LSAs before it
can converge. There is no secondary path that can be immediately installed.

But if the design is a triangle instead:

There are already dual paths so losing one won’t affect convergence and the other route
is already in the FIB so traffic can keep flowing.

Conclusion

There are many network designs out there. Learn the strengths and weaknesses of
different designs. Look at best practice designs from the vendors but don’t follow
them blindly. As I have shown sometimes recommendations will not work for all
scenarios.

Finding the right design depends on business needs, budget and what kind of applications
that are running. Read more on campus design in BRKCRS-2031 and also look at the Cisco
Validated Network Designs.