So I’ve just started to read the ARCH book and I want to do some posts to help with
my understanding but I’m also very interested in hearing from you readers what
kind of design you like and why. Post in the comments which design you would prefer
and why you prefer it.
So this post is about designing the access to distribution layer. I will show a
couple of different ways of designing this block and give my point of view on
the advantages and disadvantages of each.
First out we have the layer 2 loop free design.
The links between the access and distribution layer are layer 2 trunks but the
link between the distribution switches is layer 3.
Pros:
All links are forwarding.
No bridging loops possible.
Fast convergence, only dependent on FHRP.
Load balancing possible by tweaking STP and FHRP.
Cons:
Can’t stretch VLANs across multiple switches.
FHRP traffic must pass through the access layer switch.
This is a good design because we have no layer 2 loops. This also means that we
will have faster convergence in case of a failure since we are not relying on
spanning tree to reconverge. Often it is not possible to use this design because
we have the need to stretch VLANs across multiple switches. Maybe we always have one
VLAN for management, or the same VLAN is used for voice or some requirement like that.
That would keep us from using this design.
Then we have the layer2 loop design. This is the traditional design that is probably
most commonly deployed.
Pros:
Can use any VLANs we like and stretch them across multiple switches.
Cons:
Possibility for bridging loops.
Dependent on both STP and FHRP convergence.
Not all links forwarding.
So this is the one most of us uses I think. It’s easy and comfortable and you can
spread your VLANs. You pay by risking bridging loops and you aren’t utilizing
all of your links.
Then we have the layer 3 routed access design.
Pros:
No risk of bridging loops.
Spanning tree not necessary on ISLs.
Fast convergence.
No need for FHRP.
Cons:
More expensive solution to run layer 3 in access layer.
Can’t span VLANs across multiple switches.
This solution is really nice if you can fit it in your budget. You will only rely on IGP
for convergence and you get rid of spanning tree almost entirely. You also don’t need
to run a FHRP since now your default gateway is on the access layer device.
If we have Catalyst 6500 with VSS or something like stacked 3750s in the distribution
layer then we can run a Multichassis EtherChannel (MEC) which would make the two links
to the distribution appear as one logical and we would have more bandwidth and no
blocking links. It would look like this.
So now I’m interested in hearing from you readers which one you like best and which
one you use. Argument why you prefer the one that you use. Are you seeing much layer 3 in
future designs or are you still stuck with L2?
Of course MLAG is better than you have described other following situations , i’m always using stacking or mlag feature depends on device or vendor.
better is VSS but module is capable for this feuture, i pooling for Layer 3 arc, looping free and easy for handle traffic
I’ve done a lot of large scale (2000+ port) LAN designs and implementations over the years. A well designed L2 looped technology is fine with a stretched management, common service and AP VLANs, VLANs per closet (with option to expand across multiple switches in the future) load sharing via fhrp configs and rpvst+. These scale surprisingly well if you design from the edge into the core. (Don’t assume that’s common sense!)
I don’t like L2LoopFree – I’ve seen this used just for the sake of it with clumsy access loopbacks/distribution statics for management, and limitations on future expansion projects (can’t easily redesign a 6000 port network to add extra switches – sometimes they want to reuse non-stackables). Problematic with service modules as well. The considerations to avoid blackholing are more than those for intelligent ST design.
L3 access is a good solution, but the extra costs ramp up on large LANs. It’s becoming more popular on smaller networks and in server blocks, but there hasn’t really been a great reason to use it, especially as we have VSS on the 6500, and coming on the 4500/4500x platforms
At the end of the day, companies who want large LANs will be spending a lot of cash. Simplicity and economy are driving factors
Glad to hear from someone with experience from large deployments.
I agree with what you are saying. I guess it will depend a lot on the skill of your engineers if you get into any trouble with the layer2 loop design but it’s easy to work with when you need to add more switches/VLANs.
L2 Looped for me with L2 to L3 boundary at the distribution/collapsed core-distribution layer
You never know when your going to need the scalability or ease of new service migration.
I am a keen fan of VSS too.
We are using the last drawing on this page, L2 port-channel between switch stacks. Our access devices are sometimes stacks and sometimes solo, but, invariably we are interconnecting them redundantly into the upstream switch through a two member port-channel group, each connected to a different stack member. In fact, in our network, the aggregation of our access devices occurs in our routing core, a L3 3750 switch stack, soon to be replaced with 3850 switches.
Well, maybe not replaced. We’ll see. We are moving away from traditional MPLS layer 3 inter network services, interconnecting our sites, to a VPLS layer 2 service. In the interest of ease of deployment and switchover, we are contemplating a simple drop in replacement strategy with the 3850 stack being moved into the role of WAN router and the new aggregation point for all network stacks.
hello,
I have question for the first case, please help me to understand.
Suppose that there are 2 access switches with 2 local VLANs each and is needed to have a load-balance for traffic with a FHRP protocol, how to configure the L2 links and VALN mapping but not to loss the communication between Active and standby router ?
Hi,
First of all, to do FHRP you would need to have same VLAN(s) on both access switches. If you use HSRP you can create multiple groups, one where Dist1 is primary and one where Dist2 is primary. Then you need to assign some clients to GW = Dist1 and some to GW = Dist2. This can be a bit messy though especially if you run DHCP. For this to work you have to map VLANs so primary VLANs for Dist1 go out on the trunk to Dist1 and vice versa.
If you do use GLBP then you can use that to do the load balancing instead.
Very informative post, thank you. I setup the layer 2 looped design in packet tracer with one distribution switch as the stp and hsrp root for vlan 2 (backup for 3) and vice versa on the second distribution switch.
Two access switches with vlan 2 and 3 on both. Following the packet flows for ping, they do travel up and down the uplinks a lot, it doesn’t have a straight forward packet flow, is this normal?
Seems like a waste of bandwidth while adding extra latency, is there a way to solve this while keeping the layer 2 looped design?
Hi Ian,
I’m not sure what you mean. Between where were you pinging? Between different VLANs or the same?
Between different VLANs seems to be the worst. Ill get my packet tracer out and look at the flow again.
Ian
Now I got confused. I was reading about high availability in campus design, and I found this:
….For those campus designs requiring greater flexibility in subnet usage (for instance, situations in which VLANs must span multiple wiring closets), distribution block designs using Layer 2 switching in the access layer and Layer 3 switching at the distribution layer provides the best balance for the distribution block design…..
If I understood well, it says I can use L2 links between Access and Distribution switches and L3 between Distribution switches to span VLANs.
http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/routed-ex.html
Hi, 4 years later and I’m confused too 😉
I think it’s a bug. It should read “distribution block designs using Layer 2 switching in the access layer and Layer 2 switching at the distribution layer provide the best balance…”
Greetings. Wonderful blog from someone who id consider a thought leader in the field sharing their expertise. Any chance we can get updated broken images for the diagrams on this one?
Hi Mike! Thanks!
I think the broken images are because I migrated from a public WordPress install to a private one. It seems this link is showing the images, though -> https://reaper81.wordpress.com/2012/11/07/a-look-at-access-to-distribution-block-designs/