QoS – Quick Post on Low Latency Queuing

A friend was looking for some input on low latency queuing yesterday. I thought the exchange we had could be useful for others so I decided to write a quick post.

The query was where the rule about the priority queue being limited to 33% came from. The follow up question is how you handle dual priority queues.

This is one of those rules that are known as a best practice and doesn’t really get challenged. The rule is based on Cisco internal testing within technical marketing. Their testing showed that data applications suffered when the LLQ was assigned a to large portion of the available bandwidth. The background to this rule is that you have a converged network running voice, video and data. It is possibly to break this rule if you are delivering a pure voice or pure video transport where the other traffic in place is not business critical. Other applications are likely to suffer if the LLQ gets too big and if everything is priority then essentially nothing is priority. I have seen implementations using around 50-55% LLQ for VoIP circuits which is a reasonable amount.

How should dual LLQs be deployed? The rule still applies. Do not provision more than 33% to the LLQs in total if there is a mix of voice, video and data.

TCP based applications will likely suffer if they don’t have enough bandwidth. Don’t break this rule unless you have to and be aware of the consequences.


I had some feedback from Saku Ytti about the relevance of a LLQ. Many of these best practices revolving QoS came from when networks were much more fragile and we had poor quality VoIP solutions that were very sensitive to jitter and packet loss. Modern VoIP solutions aren’t as sensitive to these conditions and we can use codecs that predict audio when packets are lost. Modern network devices add very little jitter to packets so jitter should not be a large issue today. For these reasons the importance of the LLQ has definitely decreased.

I would be interested if any of my readers has any data on placing VoIP and/or video in a standard queue vs a LLQ and what the result was. Bonus points if you have graphs to show 🙂

General – The Future of Networking – Pete Lumbis

The next person I interviewed about the future of networking is my friend Pete Lumbis. Pete used to be the routing escalations TAC leader at Cisco and now he is working at Cumulus as a SE. Pete holds both a CCIE and a CCDE.

Daniel: The networking world is changing. What are the major changes coming up in the next few years that you think we will see?

Pete: Automation is the big thing these days. Either through APIs or abstraction tools like Ansible or Puppet. I think there will be more embracing of automation, but as a side effect I think we will have to start building networks that are more automation friendly by creating fewer exceptions and one-offs. This also touches on a larger point which is the need to build systems and networks that are less fragile. Automation is less scary when you have an architecture that can tolerate some level of failure.

Daniel: What are the major skills that people in networking need to learn to stay ahead of the curve?

Pete: Fundamentals don’t change. ARP is ARP. MAC addresses still have 48-bits. Understanding fundamentals will always be key. Beyond that it’s going to be about talking to other parts of the tech organization. How do we have meaningful conversations with applications and systems and storage stakeholders so we can build all routed networks? What can we do to stop creating poorly designed networks to solve problems on poorly designed applications? Failure to do this will just make the network team look bad and will just drive our customers (servers and apps owners) to public cloud. We don’t need to become Docker experts but just like we know how email and voice operates over our pipes we’ll need to have a better understanding of storage, containers and applications.

Daniel: Is this drive for automation and SDN really that much different than anything we have seen in the past?

Pete: No, but what’s different is that they aren’t being created in a vacuum. On the automation front there have been incredible advances in the server space driven by a need to rapidly spin up and down servers in the cloud. As a result Ansible, Puppet and Salt have really taken off. They have solved so many problems in the server space that it’s hard for networking to ignore. There have been solutions like this in the past: EEM, Cisco Security Manager, NMS Stations, but they all tried to solve the same problem without any kind of sharing. Now that we have open source tools to work from a lot of ground work is already laid for us to build on top of.

On the SDN front it’s had some great marketing names in the past like Application Aware Network or AVVID. We’ve tried for years to have the network have more dynamic control but we’ve always failed because, again, we are solving problems in a vacuum, without talking to the stakeholders. We are trying to have the network guess what application is on the wire instead of just providing value to the application owner and identifying it before it enters the network. I think products like VMWare’s NSX are smart because they are moving that edge closer to the application owner to get that “software defined” goodness. The SDN things that are the most successful will be easy for the server and apps people to consume and provide direct value to them.

Daniel: What does the network engineer of the future look like?

Pete: Network engineers will have to become more jack of all trades. If we look at SRE (Site Reliability Engineers) or DevOps the technical success there has been because the people in these roles can work across boundaries. They understand Linux and storage and databases and can span the architecture to solve issues. I see the network engineer of the future being a key member of those teams. The network engineer will be the expert at networking (just like the person next to them may be a Linux expert), but the network engineer will have a much broader skill set. The ability for this to happen is related to the fact the networking industry is starting to gather around a few solutions. We have fewer protocols, we have simpler hardware platforms and the hardware is all starting to look the same. It’s going to take the same energy as before, just applied differently.

Daniel: What does the network architect of the future look like?

Pete: The network architect of the future, like the network engineer of the future, will need to really cross domains even more than they do today. I think the difference is that today we gather requirements and then build this incredibly bespoke network to suit those bespoke application needs. In the future we will build a simple network (think layer 3 datacenter Clos) and then we will work with application and systems owners to make them understand the rules and how to be successful in the environment. Beyond this the network architect will have to take some of the learnings from the server space around selecting the best automation tool for the job, how to do modeling and automated change management and testing.

Daniel: How will the traditional networking skills such as knowledge of protocols be relevant in the future, if at all?

Pete: They will always be important, but just reduced. The fine grained differences between OSFP and IS-IS will be less critical, but I have a conversation almost once away about why VxLAN’s encapsulation using UDP solves a lot of problems with hashing and ECMP that GRE and IP in IP have had in the past. I don’t need to know every bit field but that little bit of knowledge at the protocol level will make a huge difference in the architecture.

Beyond this, as I mentioned earlier, there’s a protocol consolidation happening. BGP as a routing protocol. Segment routing for label switching. PIM is still around but everyone seems to be excited about BIER(Bit Indexed Explicit Replication). The alphabet soup of Frame Relay, X.25, LDP, RSVP, and RIP are (hopefully) behind us.

Daniel: Is it still valuable to learn traditional networking skills?

Pete: Absolutely. Even in a team where skills are mixing you still need to be able to build and troubleshoot the network. Understanding how routing works can allow you to build anycast networks for way less. Understanding MAC flooding can help explain duplicate packets or period slowness on the network. Vendors will always be there to help, but without that full picture of the environment that you have as the network engineer (or the team member with a specialization in networking) it will always take the vendor much longer to solve the problem.

Daniel: Do all network engineers have to become programmers?

Pete: Nope. I think learning some basic concept of programming are infinitely useful. Understand how to build a loop. Understand how an if/then statement works. Pick a language, even Bash scripting, and be able to write some terrible code in it to do your job. My poor programming skills have made my life WAY easier, but I told my current employer during the interview process that if they want me to program I shouldn’t work here. No one should pay me for my terrible code. I can use a little python but I don’t know Go, Ruby or Perl. If you talk to systems administrators they have varying levels of scripting skills but they are quick to point out they are not programmers. I believe it’s similar for us.

Daniel: How should network engineers find interesting projects to work on if everyone will build their networks in a similar fashion?

Pete: Move up, down or lateral to the stack. Move up and learn how to provide more value to the business and the stake holders. Move down the stack and get involved in Open Source or IETF with protocol and technology creation or move lateral and make other people build better products with your networking skills. I think Docker is a super cool technology but networking in their first release was 100% designed based on PAT. No self respecting network engineer would have allowed that. As a result the folks over at SocketPlane got acquired to help Docker fix their networking issue. Most wouldn’t have thought about Docker and needing network expertise.

Daniel: Do you think we will see a move towards more standardized solutions where the customer buys compute, storage and networking in fixed packages?

Pete: I expect to see the opposite. The success of servers over the last 20-30 years has been that you can buy a server and run any OS on it. That OS behaves mostly the same on every server. A few years ago VMWare came along and said you can run VMs on any server and they’ll mostly look the same. Now Docker is saying you can run containers and they’ll mostly look the same. That ability to create small components that have clean abstraction layers (Hardware to OS, OS to application) has led to incredibly innovation.

I expect to only see more of this. Multiple switch platforms that all look mostly the same (which we see with whitebox/britebox switches). Operating Systems that look similar (Cumulus Networks, Pica 8, OpenSwitch) and then applications that can run on top (Quagga, GoBGP, Bird). (Disclaimer: I work for Cumulus Networks, so I have a bias towards this future)

When you have these clean lines then it doesn’t matter what gets plugged in. If it’s an EMC storage array or a Ceph storage node or Nutanix, it’s just a place to park bits. There will always be a market for the customers that want the turn key approach where I plug in a rack and it just works, but I expect a lot of those customers to move to the cloud in the next few years.

Daniel: What is the next big thing that people should prepare for?

Pete: Giving up control. If I knew the next big tech I’d be rich 🙂
But seriously, network engineers are control freaks. We love our boxes. We love our protocols. We love our SNMP stats (okay, no one loves SNMP), but my point is that we’ve always had very precise control of the network. In order to build more cost effective networks that do more, we have to give up some of that control. Again, using VMWare’s NSX as an example, it’s a security/policy engine running in the hypervisor. In some shops the network team control that, in others that’s driven by the server team. At Cumulus we are working with customers to run BGP on their servers to get rid of layer 2 and mLAG entirely. Again, only possible when we are willing to give up a little bit of control.

Daniel: How do you stop sipping from the firehose? It’s tough to keep up with automation, SDN, cloud and everything going on in the industry.

Pete: This is hard and I struggle with this every day. I think the first part is build simpler things. Try to limit the number of choices your stakeholders (servers and apps) get in order to provide them lower costs (financially and emotionally) and greater reliability. After that find specific problems with clear goals and implement them. Learn Ansible by using it to build configuration templates. Learn AWS to build a VPN between your public and private cloud. It’s always hard to pick up a technology where you don’t know where you’re going with it. I’ve struggled with containers because outside of understanding basic operations I don’t have a broader problem that they solve for me. Be willing to walk away from something that’s “cool” or “hip” if it makes your life more difficult or doesn’t solve your problems.

Thanks to Pete for some very insightful answers!

General – The Future of Networking – Russ White

Hello my friends,

Lately I have been thinking a lot about the future of networking and the career paths in this domain. As you probably know I like to guide and mentor people and with everything going on in the industry it can be confusing to find your way and to know what skills to work on to stay ahead of the curve.

I decided to reach out to some of my friends to ask them of their vision of the role of the future networking engineer and how to prepare for the changes that we are now seeing. First out is my friend Russ White who is also the co-author of the book Unintended Features that we wrote together.

Daniel: What are the major skills that people in networking need to learn to stay ahead of the curve?

Russ: Some of these have never changed — for instance, communication and abstraction. Some skills have been more important forever, such as people skills and project manage, but they never seem to really rise to the top in terms of actual demand. I don’t think this is going to change much; companies say they want people skills, and then recruit based on point technical skills, no matter how much they say they want to do otherwise. In fact, this tends to be what warps the entire technology market — we don’t go out looking for a good engineer, we go out looking for a person who knows a specific product. The engineering market isn’t going to change much until we separate product skills from engineering skills in an intentional way.

In the strictly technical realms, there is a stronger focus than ever on learning one vendor, because the market is fragmenting as we see people move away from open standards. There is a growing undercurrent towards more openness in other ways, particularly open source and hyperscale operators opening their work to widespread use, as well as the possibility of separating software from hardware, movements that drive skills towards understanding technologies, rather than vendor implementations. In the future, I would expect to see the engineering world bifurcate even more strongly between those who know a particular vendor product, and those who know the technology. Which will be better paid, or more in demand, is hard to judge, but it’s likely to be those who know the technology.

The one skill we really need, and never seems to be in the mix, is the ability to see how technology decisions today modify the direction of the business.

Daniel: Is this drive for automation and SDN really that much different than anything we have seen in the past?

Russ: Nope. We used to want to simplify in order to auto-configure. SDN sometimes just seems like an admission that we’re going to do unnecessarily complex things, so let’s automate them to cover up the complexity.

Daniel: What does the network engineer of the future look like?

Russ: I see the market splitting into different pieces, and all the different pieces will called “network engineer,” although they really deserve their own names. The first will be the person who buys vendor product, and moves towards “full stack,” which may end up meaning “buys and integrates hyperconverged products from vendors.” The second will be the person who focuses on the technology, and ends up disaggregating, mixing open source with white box and point vendor products to create solutions. These will both have engineering components, but they are different “kinds of things.” This is likely to be the same sort of divide we see with enterprise/service provider today.

Daniel: What does the network architect of the future look like?

Russ: As much as I might hope otherwise, probably much the same as today… What I would hope for is someone who’s moved from the second sort of engineer above — a focus on the technology, rather than product — into a mix of business and technology. What’s likely to happen is the word “architect” is going to be used for anyone who rises to a certain level or experience, rather than a specific mix of skills.

Daniel: How will the traditional networking skills such as knowledge of protocols be relevant in the future, if at all?

Russ: It depends on which side of the “great engineering divide” you fall on. If you fall on the hyperconvergence/move it to the cloud side, then protocols are just a curiosity that don’t really impact your day-to-day life. If you fall on the technology side, then protocols will continue to be the lifeblood of your day-to-day experience and engineering effort.

Daniel: Is it still valuable to learn traditional networking skills?

Russ: If you fall on the side of networking that’s building solutions, rather than buying product, then yes.

Daniel: Do all network engineers have to become programmers?

Russ: This is a fairly broad question, actually. Programming isn’t just knowing a language, it’s also a set of background skills mixed with a certain set of attitudes and thought processes. For any of these things, there isn’t one “right” answer in the world of programming, either. For instance, do you prefer waterfall or agile? There are algorithms, test frameworks, repositories, coding styles, languages, and a host of other things out there in the programming world that have developed over the decades. Which specific kind of “programmer” are we talking about?

I think learning a language is great — but I think learning the mindset of a coder is better than learning a single language. Many of the tools and ideas the coding world has developed can be applied to the world of network engineering. For instance, treating hyperconverged solutions as black boxes or components to be integrated can, perhaps be drawn in parallel with using libraries of existing code, or commercial frameworks. On the other side, the disaggregated world is going to need to rely on many of the skills of a coder to build repositories, test regimes, and the other components to make it all work.

Daniel: How should network engineers find interesting projects to work on if everyone will build their networks in a similar fashion?

Russ: It can be an interesting project to make the network as simple as possible. We focus too much on “deployed the latest whiz bang gizmo from vendor A across 5000 sites, finding and crushing bugs,” rather than on “I reduced our opex by 50% by simply removing a lot of stuff we didn’t really need, and worked with business folks to figure out what we really need.” We’re far too impressed with an engineer’s ability to deploy complex things than in their ability to make things simple.

Beyond this, though, there is always, in every business, something that makes that business unique in some way. If there is no snowflake in the business, there is no business, because there’s nothing to sell. Find that snowflake and figure out how to make it work with the network. Don’t integrate it wholesale into the network; you don’t want the entire network to be a snowflake. Rather, find a way to separate the snowflake out so the network can be simplified while that little snowflake becomes the point of connection between the business and the technology.

Daniel: Do you think we will see a move towards more standardized solutions where the customer buys compute, storage and networking in fixed packages?

Russ: In some realms, yes. In others, no.

Daniel: What is the next big thing that people should prepare for?

Russ: The networking field is splitting. Do you know where you want to be, and why? Do you know how to get there?

Daniel: How do you stop sipping from the firehose? It’s tough to keep up with automation, SDN, cloud and everything going on in the industry.

Russ: Rule 11 is your friend, particularly if you’re going to be on the side of networking that focuses on building solutions from open source and other parts, rather than the vendor/hyperconvergence focused side of the industry. Learn the ideas, not the implementations.

Thanks to Russ for writing such extensive answers. I think these posts will be very helpful for all the people in the networking industry.

General – Network Engineering vs Coding


There has been a lot of talking about the future of the network engineer for the last couple of years. Many articles have declared that we MUST learn to program or we will be banished from the world by the programming overlords! I definitely do not agree with this bold statement but lately I have started to learn Python. Why?

Why Learn Programming?

As a network architect I probably won’t ever write a line of code or at least very rarely so. So why bother learning?

I didn’t learn a lot of programming back in my days of school. I fiddled around a bit with Basic, some Pascal and then at the university I tried some C# and C++. I never felt connected with programming. I never felt that I was good at it. This surprised me a bit because I’ve always been good at learning things. I’m good at analyzing things, troubleshooting things and I have a strong background in maths and science in general. I had all the skills that good programmers normally have so why couldn’t I learn programming? Because I struggled I didn’t enjoy doing it so I never pushed through until it “clicked”.

Later I have learned that things don’t always “click”. Sometimes it takes time, reading something multiple times, labbing with it, reading about the same thing from another resource. I never felt connected with those programming languages.

I started learning Python a couple of months ago. I immediately felt that this language was more suited for me. I could read the code and have some form of feeling of what it did even if I didn’t yet understand the code. While some people won’t like the indenting of the code to me it was more readable, it was easier to see what code blocks belonged to which loop or function.

So why did I start to learn Python?

There are several reasons to learn programming.

Automate simple tasks – You don’t have to learn a lot of programming to be able to do useful things. Especially with all the plugins and modules available you can use the code of better programmers to simplify your own scripts. One example is being able to convert a MAC address from the format of 00:01:02:03:04:05 to 0001.0203.0405 with just a few lines of code. Even simple tasks like this can get very tedious when you have to do it manually at scale. Another example would be to create an access-list with the same rule where only the subnet changes and you have 100 subnets to go through. Tedious by hand, easy by code. These simple tasks are not what your job is about so if you can automate and get them done faster you will have more time to do learning or more important work.

Learn the tools and language of Dev – Traditionally different job roles have been very differentiated where each discipline of engineer have had their own silo that they tend to. This is not efficient and the modern engineer needs to have a broader skill set. If we learn to use tools like Git and Travis CI we can both become more efficient in our own work as well as being more able to understand the language of the Dev people. We as network engineers can learn a lot from the coding side about how to scrub for bugs and how to test the configuration that we deploy onto devices.

Mental fitness – Programming can be seen as a form of mental training. You have to think about how things fit together, how to calculate different things and how to make sure your code is working. When your code is not working you have to troubleshoot it. It promotes a scientific approach for solving problems. It also helps in learning you how to attack a complex problem by dividing it into smaller parts and solving each part.

Understand applications – If you know how to code you will have a better understanding of how applications work. Why does this application require a L2 adjacency? Is it because it does not have its own proper HA so it must rely on things being part of the same subnet? Once you start talking the language of Dev you’ll be more fit to understand their requirements and come up with arguments of your own why they are building an app in the wrong way if it requires too much from the network.

Will We All Become Coders?

I don’t believe that we all will become coders or that it is a desirable outcome. If we simply replace CLI commands with scripts we won’t have gained that much. Configuring our devices shouldn’t require advanced programming knowledge. It’s the job of the vendor to develop proper tools to interact with the devices but also to promote open source solutions so that we have options for what tool to use to configure our devices.

Knowing how to code vill be a valuable skill but we also need to know networking, virtualization, storage, applications and we can’t be experts in them all. We should leave the advanced programming to the people that actually are developers. We can commit code to a project but someone with more experience should check the code for errors.

Under all the overlays there are still classic network protocols running such as ISIS and BGP. We will still have to know these tools in the future but it will be more for troubleshooting than for configuring, at least in a data center fabric. In a service provider network it’s still required to configure these things by hand.


The sky is not falling. You don’t have to learn coding if you don’t want to but doing so will evolve your skillset and possibly accelerate your career. Don’t be afraid to try it out and don’t be afraid if things aren’t “clicking” immediately. Give it time and you will soon notice that day by day you are becoming a better coder.

Book – Unintended Features

Hi everyone,

I have some exciting news to share with you. I’ve been working on a book lately together with Russ White. It’s called Unintended Features – Thoughts on thinking and life as a network engineer. The book is partly based on blog post we have written in the past but also some unique content for the book. The outline of the book is as follows:

So you’ve decided you want to be a network engineer—or you’re already you a network engineer, and you want to be a better engineer, to rise to the top, to be among the best, to… Well, you get the idea. The question is, how do you get from where you are now to where you want to be? This short volume is designed to answer just that question.

This book tries to teach concepts not found in other writings such as thinking more about architecture and seeing patterns in technology and how to stay current in the networking industry. With the rapid pace of the networking industry it seems like we are sipping from the fire hose. How can we prevent this? Isn’t every new technology pretty much an old one with some new parts? When you start thinking at that level and start to see the patterns you won’t feel as stressed about the evolvement of the industry.

If you want to buy the book it’s priced at 9.99$ currently + VAT Unintended Features.

We’ve just published it and it’s published at Leanpub which means we can push corrections and updates to it. There will probably be some editing errors and spelling errors etc so please report those and we will work them out.

CCDE – My Journey To Becoming Swedens 2nd CCDE

On May the 17th I passed the CCDE practical in Madrid and became Swedens 2nd CCDE, CCDE #20160011. This post describes my journey to passing the CCDE practical in my 1st attempt and the materials that I used to do so.

Let me start by saying that this is a tough exam, a very tough exam. You need to be an expert in RS and SP technologies and there is no instant feedback in the exam, like you would get in the CCIE lab. In the CCIE lab you will see you are missing routes or if your output does not match the output the lab guidelines told you to match. In the CCDE practical there will be very few questions that you are 100% sure that you got the optimal answer. Design is a more subjective skill than implementation. I had several moments where I felt that I could just as well leave because there was no chance I was going to pass the lab. You need to be mentally strong to put those thoughts aside and just keep performing your best throughout the whole exam. You might be doing a lot better than you think.

The first section will focus on mandatory books to read for the CCDE practical. Only reading these books or even reading all of the material I am referencing here will not make you a CCDE. You need to have the depth of knowledge within these technologies but this list will help you get started with the most essential resources.


CCDE Study Guide

When I started studying for the practical, there was no book that summarized the knowledge needed to attempt the practical. Marwan Al-shawi wrote this excellent book which is an essential read for the CCDE practical. This book teaches business requirements, technical constraints, network design principles and the most important technology that is included in the CCDE program. Don’t rush through this book, you must understand the concepts and you will probably end up reading it multiple times.

Optimal Routing Design

This book written by Russ White, Alvaro Retana and Don Slice is the bible of routing design. This book is over 10 years old but the principles still apply. This book will teach you about fault domains, modularization, aggregation of topology information, summarization of prefix information. When we aggregate routes we have a more optimized forwarding table but what are we giving up? There’s always a tradeoff! If we summarize we may have suboptimal routing, sometimes also called stretch. There is also the risk of summarization black holes. This book is a must read to understand how to design networks using different routing protocols such as EIGRP, OSPF and ISIS.

Definitive MPLS Network Designs

This book in my opinion is the book that has the most resemblance to the CCDE practical. This book takes you through different fictious scenarios which are based on the experiences of the authors. As the scenario develops they will explain why a technology was chosen and what the impact is to the design. It goes through a lot of technologies such as IGP, BGP, MPLS, MPLS-TE, Inter-AS and so on. It’s quite a heavy book but an essential read for the CCDE practical. It is useful to use this book and discuss the scenarios with other people preparing for the CCDE practical.

The Art of Network Architecture: Business-Driven Design

This is another book by Russ White and the co author Denise Donohue. This book focuses on the business side of network architecture but also explains a lot of important concepts such as mean time to repair (MTTR), redundancy vs resilience and the OODA loop. This book explains different topologies such as fully meshed, rings, CLOS and so on.

The next session focuses on Cisco Live presentations. This is actually one of the most valuable resources and almost all of the content is 100% free!

Cisco Live Sessions

BRKRST-2337 – Intermediate – OSPF Deployment in Modern Networks

This session is a good complement to Optimal Routing Design. It has some of the more modern concepts such as prefix suppression, LFA, rLFA, BFD and goes through routing design for OSPF in different topologies such as fully meshed and hub and spoke. It also has a lot of information on using OSPF as PE-to-CE protocol in MPLS VPN networks. You must be very knowledgable in what kind of topology changes trigger SPF runs, the different area types and how the number of areas affect the scalability of an ABR. When we use stub areas we get less routes to the routers in the stub area but what do we give up? Once again, optimal routing. Does our business require optimal routing though? That’s where you have to map the requirements of the business to the technical design that you will use.

BRKRST-2338 – Intermediate – ISIS Deployment in Modern Networks

This session on ISIS is also a must read to complement the Optimal Routing Design book. It starts out by comparing ISIS to OSPF and then demonstrates some best practices for ISIS. The session goes through different designs and shows how a L1 router may use suboptimal routing because it will only have a default route to the L1L2 router unless routes are leaked. It also show the concept of multi topology and single topology. The session shows important concepts in achieving fast convergence and important concepts such as LDP IGP sync and LDP session protection.

BRKRST-3321 – Advanced – Scaling BGP

This session on scaling BGP starts out by comparing confederations to route reflectors. It then explains the concepts of hierarcy within route reflection and route reflector clusters. It has a useful chart for comparing confederations to route reflection. It shows best path selection when RR is used and how more paths can be sent by using technologies such as Add Path, Shadow RR and shadow session. It also shows how hot potato routing can be done when using route reflectors. It also shows different scaling options such as carving RT’s between route reflectors and using route target constraint (RTC). It also demonstrates the concept of running Internet in a VRF and different MPLS label assignment modes such as per CE or per VRF as opposed to the default of per prefix.

BRKRST-3363 – Routed Fast Convergence

This session is all about fast convergence and shows the four steps that are involved in converging, detecting, notifying, calculating and installing new routes. The session shows that failure detection via interrupt based mechanisms is generally much faster than doing polling. It shows how interrupt based signalling may not always work if there are other devices between two routers as an example. It compares fast hello’s to BFD and shows how different IGP’s can be tuned to achieve fast convergence.

LTRCCDE-3006 – Advanced – CCDE Lab

This is actually a debrief session for a paid CCDE lab available at Cisco Live. I highly recommend that you take the CCDE techtorial and labtorial if you are serious about this cert and are going to Cisco Live. This session is still useful even if you didn’t take the lab though. It demonstrates different type of questions that you will face during the lab and it also does a debrief of the scenario Best Buddy. You can still learn from this even if you didn’t take the lab. I reviewed this session a few days before taking the practical and I started finding things I weren’t fully agreeing with in the slides. This is a good sign that you are getting prepared for the practical. This session shows the concept of branching questions which are very important for the practical.

BRKCRT-8001 – CCDE: The Cisco Certified Design Expert (Session 1)

This session explains what the CCDE is and why you should get involved in network design, it’s just not about plumbing! This session also shows different technologies that are expected to be on the practical. Then the session goes through a fictious scenario called LISP and shows how you will receive documents and e-mails and examples of different type of questions. This session is very useful to get a feel for what the CCDE practical is like. If you watch the video, there is a part 2 to this session that is called BRKCRT-8002 – CCDE: The Cisco Certified Design Expert (Session 2).

These are the sessions that MUST watch/read but I have probably gone through 50-100 sessions in total on different topics such as DMVPN, GETVPN, FW design, WAN design, DC design and so on.

CCDE Training

There are mainly two prominent CCDE trainers out there, Jeremy Filliben and Orhan Ergun. They are both very talented and strong instructors and I would recommend that you get material from at least one of them if not both.

Jeremy Filliben

I used CCDE scenarios from Jeremy in my preparation for the practical. As far as I know, Jeremy has the most scenarios and each scenario is a different platform that revolves around different concepts such as a merger or divestiture, adding technology etc. The scenarios which are delivered in PDF format simulates the exam experience by giving you initial information and then communicating new information through e-mails. It has different type of questions such as multiple choice, single answer, charts, diagrams and so on.

Jeremy also delivers bootcamps, I attended such a bootcamp roughly a month before the practical. This was a great experience to learn from someone who is already a CCDE. I learned a lot about how to approach the exam and how to think when you are answering questions for the practical. I was already fully prepared from a technology standpoint when I took the bootcamp. You should not take this bootcamp expecting Jeremy to teach you all the technical content. No bootcamp can do that in only one week of training.

If you are interested in Jeremy’s training, visis his web site here.

Orhan Ergun

Orhan is also producing content for the CCDE. His offering is called the Designworld where he offers different materials such as CCDE scenarios, CCDE videos and comparison charts between different technologies. I think that Orhan’s charts are very good and a key to getting prepared for the CCDE practical. The technology videos are good to refresh your knowledge on different technologies and to see how an experienced designer approaches different technologies such as first hop routing protocols (FHRP’s) or IGP’s, BGP etc.

Orhan also offers bootcamps and personal coaching sessions. He delivers bootcamps both online and onsite. The next bootcamp is delivered in August. One of my friends, Martin Duggan, just passed the CCDE practical in London and he had been receiving training from Orhan.

If you are interested in joining Designworld, go to Orhan’s site here.

Blog Posts

Blogs can be another important resource in getting prepared for the CCDE practical. I recommend the following excellent blog posts by Diptanshu Singh which were posted to the Packet Pushers blog.

MPLS TE Design -Part 1
MPLS TE Design -Part 2
MPLS TE Design -Part 3

These posts are very good to understand what MPLS-TE is, what a tactical deployment is and what a strategic deployment is. It discusses how you can scale MPLS-TE and different FRR methods. This is the best writing I have seen on MPLS-TE outside of the books. Make sure you understand the content from these blogs.


This post describes the concept of accumulated IGP (AIGP) metric which can be used to carry an IGP metric across BGP domains. This is used to optimize routing between different ASNs since normally MED is used and that is not representative of the total end to end path cost. For this reason AIGP is better than MED in achieving optimal routing.

IP FRR And Micro-Loops Part 1
IP FRR And Micro-Loops Part 2

These two posts introduces the concept of LFA and how micro loops can be formed. It’s a bit heavy on the math side which is not very important for the CCDE but the concepts are important. You should understand what IP FRR is, when to use it and why micro loops are formed and what can be done to prevent it.

BGP RR Design – Part 1
BGP RR Design – Part 2

These posts are very important in understanding BGP RR design. They will show the challenges of BGP RR such as increased convergence time, suboptimal routing and reduced path diversity. Routing loops can also exist in a RR design if the physical topology is not congruent with the logical topology. It also shows how BGP RR can be combined with fully mesh to achieve a reasonable scale and more optimal routing than a full RR design.

I have also done a lot of post for the CCDE. In my opinion writing is one of the most efficient ways to learn something at a deeper level than simply reading. I recommend you read the following posts that I have put a considerable effort into to summarize information from books, Cisco Live sessions and real life.


This blog is about CSC which is a concept where a backbone carrier can efficiently carry routes of a customer carrier.


Inter-AS VPNs is an important concept for connecting VPNs between two different ASNs of the same organization or to connect two different organizations together. This technology can be important in mergers where there needs to be a temporay setup between the ASNs until one side can get integrated into the other.


BGP confederation is an alternative to using BGP RR although the two technologies can be combined as well. BGP confederations can be use when there are different groups of people responsible for different parts of the network but they still belong to the same organization.


This post describes the important considerations for BGP convergence. The main concept is to have a fast converging IGP. It goes through bgp next hop tracking (NHT) and different timers such as the minimum route advertisement interval (MRAI).


DMVPN is a Cisco proprietary technology but it is still something you need to study for the practical. This post talks about some ways to scale the DMVPN by using a dual tier topology where the mGRE control plane is handled by one router and the crypto control plane by another router.


GETVPN is another Cisco proprietary technology which is a tunnel less VPN built over private WAN. This post is a good summary of Cisco Live presentations and the GETVPN design guide.


This post explains different WAN rates such as T1, E1, DS3, OC-192 etc. In the practical you might get the BW in clear writing but I still recommend you learn these basic rates by heart.


MPLS primary one-hop tunnels is a way of scaling MPLS-TE networks and achieving FRR for both MPLS-TE LSPs and LSPs signalled by LDP as well as plain IP traffic. One-hop tunnels are tunnels that are one hop and built between adjacent routers.


Security is not as big a part of the practical as RS and SP technologies but you still need some understanding of it. You need to understand where to place security devices along choke points and concepts such as a routed firewall vs a transparent firewall. Where is it most optimal to place an IPS? What is the difference between an IPS and an IDS?


This post on load balancing describes different load balancing designs such as one-armed and direct server return (DSR). Load balancing is not a big topic for the CCDE but I recommend that you learn the basics.


These two posts describe PIM BiDir which is normally used in many to many multicast deployments such as in the financial vertical. What is the role of the RP in PIM BiDir? How is RP redundancy achieved? These posts teach these concepts.


This is an interview I did with the CCDE program manager, Elaine Lopes. It explains why you should go for the CCDE and what study resources are available for the CCDE.


IPv6 multicast is probably not a key technology for the CCDE but you should be familiar of the concepts which I have summarized in this blog post.


In the road to deploying IPv6, what transition technologies can we use in the mean time? What is 6RD? What is 6PE? What can we deploy when we have moved to IPv6 but need to maintain IPv4 connectivity? Those concepts are explained in this post.


What is bisectional bandwidth? Why does STP waste bandwidth? How can anycast HSRP achieve more efficient forwarding and utilization of links in a leaf and spine topology? Learn about this in this blog post.


QoS is a very important topic in the CCDE. This blog post is basically a summary of the End to End QoS Design book. What are the characteristics of different type of applications such as voice and video? How should we mark traffic? How much bandwidth should be set aside for the LLQ? This post is important to understand QoS for the CCDE practical.

There are some more posts available but I recommend that you start with these. To reach all of the posts I’ve done for the CCDE, you can follow this link.

CCDE Slack Study Group

I and my friend Kim Pedersen started a CCDE study group in Slack. In my opinion it was one of the key factors that made me pass this exam in my first attempt. We discussed different technologies and designs and the group consists of a lot of different subject matter experts. We have experts in a wide range of technologies. The CCDE is very difficult to achieve on your own which is possible when studying for the CCIE.

Cisco Validated Designs

Cisco has a program called Cisco validated designs (CVD) where different architectures such as Campus design are explained and what the best practices are in such environments. I recommend that you read through at least the following CVD’s.

Campus LAN and Wireless LAN Design Summary – October 2015

Internet Edge Design Summary – October 2015

WAN Design Summary – October 2015

Intelligent WAN Technology Design Guide – February 2016

Closing Words

I studied for this exam for about two years. There are no shortcuts. You need to be an expert in RS and SP technologies and you need a basic understanding of DC and Security design as well. Reading all the material in this blog post will not make you a CCDE but it provides you with the foundation of the knowledge that you need for the exam. By putting this in a blog post I hope to help the CCDE candidates out there to a more efficient study path.

CCDE – I passed the CCDE Practical in Madrid!

Hi everyone.

I’ve not been posting lately because I have been studying very hard for the CCDE practical.

Passed the lab in Madrid? Isn’t this guy from the North? I was supposed to take this exam in Frankfurt on Tuesday the 17th of May. Wise from my trips to the CCIE lab in Brussels I took a flight that landed around noon on Monday. I have a routine I like to use the day before a big exam. I had just scouted the Pearson Professional Centre (PPC) location and got back to my room. At 14.05 I receive an e-mail from Pearson Vue saying they can’t deliver my exam. Can you imagine the panic I felt? I had been preparing for months of furious studying for this day. The CCDE practical is only delivered every three months so I would have to wait for three more months to take it if I could even get a seat then. I had prepared for this day and my plan was to try to pass it and if I didn’t, come back in three months and pass it then.

There was no time to waste. I found an open seat in Madrid and booked it but now I also had to find a flight, a hotel and so on. Not the way I wanted to get into the right mental state for the exam for sure! I got on a plane and arrived at the NH Madrid Lagasca around 01.00 and then tried to get some sleep for the next day. Fortunately the exam starts at 10.30 in Madrid. Apparently those Madristas don’t like early mornings 🙂 I still woke up at 6 AM though. Go figure…

The PPC is literally around the corner from the NH hotel so if you go to Madrid, this is a good place to stay in. It was a very small location. The woman in the desk was surprised I got a seat there. There were literally like 4 or 5 seats in total and I was the only one taking the CCDE practical there.

When I arrived I had to leave all personal belongings. I wasn’t allowed to bring in anything to eat. The only thing I could bring was my passport and the key to the locker. When you sign in they will ask for two forms of identification so make sure to both bring your passport and another form of identification, I used my drivers license. They will also ask you to write your signature and make sure that it matches to the signature of your identification. They will in addition to this also photograph you.

After leaving my things I was handed some laminated paper, pens and ear plugs. I used the ear plugs to gain some extra focus. The laminated paper was enough for my needs. They are double sided so I think there were like at least 6 pages available.

I had arrived a bit early and was able to start the exam at around 10. When you start out you will get some information on the exam format and then you will arrive at the real scenario. You will land on a page that has the first question and you will have some initial documents in your inbox. You should be notified everytime something new arrives in the inbox but do check there as well as I’ve heard stories of people that have not been notified when new documents arrive. If you feel you are missing some information make an extra check to see that you haven’t received a new document.

The documents you receive will be background information, technical information, e-mails, diagrams etc. There can be a lot of information to go through and digest. Highlight what you think seems most important such as business goals, business constraints, technical constraints and so on.

I was pleasantly surprised by the lab delivery and GUI. I thought the diagrams were well made and it was easy to interact with the exam.

You should spend around 10-15 minutes reading through all the information in the beginning. I’m a fast reader so I used a little less than that.

You need to try to connect with the exam. Imagine that you are really working in the role they are telling you that you are. I struggled a bit to connect with my first scenario but felt better about the others. I didn’t think I would end up passing this thing though.

I went through the first scenario, struggling, as I felt it. It was a real slap in the face. I had studied very very hard and I go in there feeling like I know nothing. It’s mind games, people. You need to shake that off and do your best and just keep going. I felt like I could just leave righ then because there was no way I was going to pass this thing. You will receive different type of questions.

Multiple Choice Single Answer – There is only one correct answer. Try to pick it!

Multiple Choice Multiple Answer – There are several correct answers. They will tell you how many to pick!

Charts – Tick the boxes that apply for row/column pair. There may be several boxes that need to be ticked in a row or none! Don’t feel that you have to tick boxes if it doesn’t make sense to you!

Implementation Questions – These questions ask you to perform implementation steps in the correct order that produces the end result with the least impact on the existing network.

Diagrams – You will interact with a diagram. In some of them you click boxes to check which role a device has and what technologies it needs to run. On some diagrams you may have to place devices and add/remove links etc.

I’m sure I’ve missed something but my brain is pretty mush right now.

I finished the first scenario with roughly 30 min to go. I’ve heard varied reports about how stressed for time you are in this exam, fortunately it was not a factor for me.

The second scenario felt better but it was still very difficult. I know I had some extra time left over from the first scenario so I wasn’t rushed for time. I went through all the questions and ended with around 30 minutes to spare. You have to take a mandatory 1h lunch plus the time I had left over so that meant I had to go away for 1.5h. You can’t add the time from the first half of the day to the second half.

It’s important to remember that your questions will not only relate to the most recent documents you have received. You need to remember the goals and constraints that were handed to you in the beginning. You might be tempted to pick a technology that makes the most sense from a technical standpoint but maybe the organization told you in the beginning that they are not allowed to deploy this technology. There may also be situations where you have to compare technologies in some chart and then they may ask you which one to pick. So try to remember which one you think made sense!

After returning from a light lunch and went back into the PPC. I had to leave my things again and show my passport to get back to my seat.
Once again I finished the scenario with roughly 30 minutes to go. I knew I had extra time for the final scenario so on some questions I spent a lot of time to make sure I thought through everything. I was able to finish with around 30 minutes to go. I was 100% sure I had failed this thing so I was happily surprised when it said PASS on the score report. If you have ever taken one of these you will know you have to check it like 10 times to make sure your eyes aren’t deceiving you.

That’s why it so important to perform at your best level through the entire exam. After the first scenario I was crushed and although the others felt better I was sure I wouldn’t pass this. This was after all my first attempt at this thing.

I really want to make a point that if you fail this exam this does not make you a bad Network Designer or Network Architect. I have friends that I highly respect and that are in leading positions in their companies that have yet to pass this exam. It is a test and we all perform differently in tests and one test may suit a person perfectly and not the other. I think the exam does a good job of simulating the work of a Network Designer or Network Architect but it’s still very different from what I do in my day job. To all of you that didn’t get it yesterday. I’m sure you will get it next time!

I’ll write a post within a week or so on how I prepared for the exam and what study material I recommend. That is all for now!
Adios amigos!


In the previous posts I talked about why it’s important to build a network and how you can do it but there is still one component missing. Any guesses?

How do we maintain our network once we have built it?

Stay In Touch

You spent all this time and put effort into building a network. Are you going to let this effort go to waste? I hope not. It’s important to stay in touch every now and then and check in how your friends are doing. This could be by sending an e-mail, a text message, just giving them a call or going for a lunch. Don’t contact them only when you need their assistance. Don’t be a leech. Show that you appreciate them and the help you have received from them in the past.

Return The Favor

One of your contacts helped you with a technology or troubleshooting an issue which helped you move forward in a project. The next time they may require assistance from you. When this time comes, maybe you are very busy at work. Do you simply turn them down? I hope not and if you do don’t expect any help the next time you ask them. Maintaining a network is about taking and giving. Even if you are busy, make some time or tell them that you will get back to them in the evening or the next day or whatever.

Expand The Network

Don’t be afraid to expand your network even if the person that wants to join your network is not as experienced as you. Don’t understimate the power of paying it forward. This person may also have reach into other networks and organizations that would be beneficial to you. Worst case you make a new friend. Is that so bad? 🙂 I know from experience that it’s difficult helping out everyone that contacts you if you are a somewhat known person in the industry. I always try to answer in a polite manner though.

I hope these series of non technical posts has been helpful in inspiring you to go out and network! There may be some more posts coming up on how to build a mindset for certifications and topics in that range. Good luck in your building of networks!

Cisco Live – News About the Customer Appreciation Event (CAE)

Cisco Live takes place in Las Vegas between the 10th and 14th of July this year. Every Live event, Cisco holds a customer appreciation event (CAE) in an arena close by the conference center. Last year we saw an amazing performance from Aerosmith hosted in San Diego. The year before that, Imagine Dragons put on a show in San Francisco.

This years event will be hosted at the T-Mobile Arena on the Las Vegas strip. This is a very new arena that opened on April, 6, just days ago. The pictures below show renderings of the arena.

T-Mobile Arena® will be the destination in Las Vegas for live events – from amazing music acts to thrilling sporting events – it will set a new standard for what entertainment means in the city that does it best. The 20,000-seat T-Mobile Arena ® will host exciting, world-class events with something for everyone – from UFC, boxing, hockey, basketball and professional bull riding to high-profile awards shows and top-name concerts.

Cisco is not only holding their CAE there. The arena also uses Cisco technology called Cisco StadiumVision which is an innovative digital content distribution system. The system is used to centrally manage and deliver all of the video and digital content in the arena, except the scoreboard. This means we can expect high quality video delivered to the screens in the arena. More information about StadiumVision can be found in the following link.

Now for the interesting part and this is information that is exclusive until April, 12. The CAE will take place at July 13 in the T-Mobile arena and the opening act is Elle King. Elle is probably most known for her hit “Ex’s and Oh’s” from the album “Love Stuff” that earned her two Grammy nominations. Elle is an american singer, songwriter and actress. Her music style encompasses country, soul, rock and blues.

The main act is going to be Maroon 5. Maroon 5 is one of the biggest pop rock bands and has sold more than 20 million albums and 70 million singles worldwide.They have also been the recipients of several awards including three Grammys. They have produced hits like “One More Night”, “Moves Like Jagger”, “She Will Be Loved” and many more.

More information about the CAE can be found at the following link.

I look forward to meeting you at the CAE event.

CCIE – Cisco Learning Network Sale on CCIE Training for the CCIE RS Lab

Are you preparing for the CCIE RS lab? Cisco 360 is the official training program for the CCIE. There are other training vendors out there which are also high quality, like INE and Narbik, Cisco 360 has an advantage in that they can leverage the real platform of the lab though. If you want to assess how ready you are you can take an assessment lab at Cisco 360. You will also have the opportunity to get more comfortable with the lab platform that is used in the lab. You will also have the opportunity to practice the TS and DIAG section to make sure you are comfortable with those sections of the lab when the big day comes.

CLN will have a sale during April and May which means that you can save between 10-20% on these products to help you prepare for the CCIE RS lab. For the CCIE there are currently three products on sale.

The first product is a bundle and it’s a starter and advanced mini bundle for 1599$ and contains the following.

  • Core and Advanced Workbooks with 25 Expert-level labs for hands-on practice. Labs 01–20 have troubleshooting and configuration sections each, labs 21–25 include large-scale configuration labs.
  • 250 hours of virtual rack rental of Cisco IOS on Linux (IOL) platform.
  • Reference library with more than 2,000 pages of technical material.
  • Pre-assessment lab that measures baseline technical skills.
  • Four performance assessment labs of student’s choice with troubleshooting and configuration sections each.
  • Two diagnostic assessment labs of student’s choice.
  • Detailed answer keys and interactive mentor guide for every lab.

If you are interested in this bundle, click here to go to the CLN store.

There is also a starter mini bundle for 999$ and is currently 10% off. This product contains the following.

  • Workbook with 10 Expert-level labs for hands-on practice. Each lab has a troubleshooting and configuration section.
  • 100 hours of virtual rack rental of Cisco IOS on Linux (IOL) platform.
  • Preassessment Lab that measures baseline technical skills.
  • Two standalone performance assessment labs of the student’s choice. Each lab has a troubleshooting and configuration section.
  • Reference library with more than 2000 pages of technical material.

If you are interested in this product, click here to go to the CLN store.

The final product is an advanced mini bundle, also listed at 999$ and 10% off. It contains the following.

  • 15 Expert-Level Labs with troubleshooting and configuration sections. Five of the configuration sections include large-scale topologies with about 30 devices.
  • 150 hours of virtual rack rental of IOS on Linux (IOL) platform
  • 2 Advanced Performance Assessment labs with troubleshooting and configuration sections in each lab
  • 2 Diagnostic Assessment labs
  • Detailed answer keys
  • Interactive mentor guide

If you are interested in this product, click here to go to the CLN store.

If you are preparing for the CCIE RS lab, this is a good opportunity to get comfortable with the lab environment and assess how ready you are for the lab. If you buy these products now you’ll save between 10-20% off the ordinary price. Good luck in your studies!

Disclaimer: I will make a small amount on the purchase if you go through my blog.

Networking articles by CCIE #37149/ CCDE #20160011