As we all know Cisco recently released the CSR1000v. This post will describe how
to create a home lab consisting of Dynamips and CSR1000v running on ESXi.
You should already have deployed the CSR and have a Dynamips box ready. I will
use Ubuntu for my Dynamips machine but you can choose whatever OS you like.
So to start you should have ESXi 5.0. If you have an enterprise version of ESX
that is great but I don’t so I’m using ESXi. I am managing it via the vSphere client.
I have installed Ubuntu desktop 12.10 64-bit version. You will need some tools to
have a good setup. I recommend you install the following:
sudo apt-get install dynamips sudo apt-get install dynagen sudo apt-get install xrdp sudo apt-get install wireshark sudo apt-get install open-vm-tools sudo apt-get install screen sudo apt-get install gnome-session-fallback cd ~ touch .xsession echo gnome-session --session=gnome-fallback > .xsession
You can then use RDP to connect to the Ubuntu machine. If you don’t need the graphics you
can use use SSH as usual.
I will put together a topology that looks like this:
As you can see I will be using 3 VLANs. One VLAN is for managing the devices.
I can login to the CSRs and the Dynamips routers from this network. The CSRs
uses GigabitEthernet0 by default as a management interface that are placed
in the VRF Mgmt-intf.
You can use a dedicated vSwitch or create the VLANs on the standard vSwitch. I have
just created VLANs in the regular vSwitch. You configure this under Inventory -> Configuration
-> Networking -> Add networking
After clicking “Add Networking” choose connection type “Virtual machine”
Choose an existing vSwitch or create a new one if you wish.
Then choose the name for your network and assign a VLAN ID to it. You can use the same
numbers I did or choose something else.
Finish the guide and the new network will be present. We want to edit a setting
for the networks that will connect to Dynamips. We want to set the port group to
promiscous mode so that CDP frames and other traffic not destined to the VM can
arrive to the VMs. This will create some overhead but shouldn’t be an issue in
a lab network. Click “Properties…” for the vSwitch.
Select the network and choose “Edit…” then under the Security tab set
“Promiscous Mode:” to Accept.
After creating all the networks they need to be assigned to the virtual machines.
For the CSRs the GigabitEthernet0 will be assigned to the MGMT network and Gi1 to
CSR to Dynamips 1 and Gi2 to CSR to Dynamips2.
Right click the VM and choose “Edit Settings…”. The NICs should be assigned like this:
Do the same also for the Dynamips VM. In theory there should now be connectivity.
We will use a topology that looks like this:
We need to create a .net file that can be used to create this topology.
5 routers will be running in Dynamips so 1 or 2 hypervisors should be enough.
As usual you need to find suitable Idle-PC value for your topology. My .net
looks like this.
autostart = False [127.0.0.1:7200] workingdir = /home/daniel/dynamips/working/CSR udp = 10000 [] image = /home/daniel/IOS/c7200-adventerprisek9-mz.150-1.M1.bin-unpacked ram = 256 idlepc = 0x628cc49c ghostios = True [[ROUTER R1]] model = 7200 console = 20061 f1/0 = R2 f1/0 f1/1 = R3 f1/0 f2/0 = nio_gen_eth:eth1 [[ROUTER R2]] model = 7200 console = 2002 f1/0 = R1 f1/0 f1/1 = R4 f1/0 [[ROUTER R3]] model = 7200 console = 2003 f1/0 = R1 f1/1 f1/1 = R4 f1/0 [[ROUTER R4]] model = 7200 console = 2004 f1/0 = R2 f1/1 f1/1 = R3 f1/1 [[ROUTER R5]] model = 7200 console = 2005 f1/0 = nio_gen_eth:eth2
The only thing special here is that R1 and R5 are connecting to the outside
world. By using the generic NIO descriptor we are connecting to the Ethernet
interfaces leading to the VM networks.
It’s time to start the Dynamips process. I will use screen because I want to
keep the process running even if I disconnect my session.
daniel@Dynamips:~/.gns3$ sudo screen -mS dynamips dynamips -H 7200 & daniel@Dynamips:~/.gns3$ dynagen CSR3.net
I have started all devices so I should be able to reach them and configure them now.
I will configure routers R1-R4 to run OSPF. R4 will announce its loopback 22.214.171.124
and this should be reachable from R5 on the other side of the network.
R1 will run BGP to both CSR1 and 2. This is the configuration applied to R1.
interface FastEthernet1/0 ip address 126.96.36.199 255.255.255.0 ip ospf 1 area 0 duplex auto speed auto ! ! interface FastEthernet1/1 ip address 188.8.131.52 255.255.255.0 ip ospf 1 area 0 duplex auto speed auto ! ! interface FastEthernet2/0 ip address 10.10.10.1 255.255.255.0 duplex auto speed auto ! ! interface FastEthernet2/1 no ip address shutdown duplex auto speed auto ! ! router ospf 1 router-id 184.108.40.206 log-adjacency-changes redistribute bgp 1 subnets ! router bgp 1 no synchronization bgp router-id 220.127.116.11 bgp log-neighbor-changes redistribute ospf 1 neighbor 10.10.10.11 remote-as 100 neighbor 10.10.10.12 remote-as 100 no auto-summary
Configuration for CSRs is very simple.
interface GigabitEthernet1 ip address 10.10.10.11 255.255.255.0 negotiation auto ! interface GigabitEthernet2 ip address 18.104.22.168 255.255.255.0 ip ospf 1 area 0 negotiation auto ! interface GigabitEthernet0 vrf forwarding Mgmt-intf ip address dhcp negotiation auto ! router ospf 1 redistribute bgp 100 subnets ! router bgp 100 bgp log-neighbor-changes redistribute ospf 1 neighbor 10.10.10.1 remote-as 1
CSR2 only has different addressing. Now do we see any routes?
CSR1#sh bgp ipv4 uni BGP table version is 7, local router ID is 22.214.171.124 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *> 126.96.36.199/32 10.10.10.1 3 0 1 ? *> 188.8.131.52/24 10.10.10.1 0 0 1 ? *> 184.108.40.206/24 10.10.10.1 0 0 1 ? *> 220.127.116.11/24 0.0.0.0 0 32768 ? *> 18.104.22.168/24 10.10.10.1 2 0 1 ? *> 22.214.171.124/24 10.10.10.1 2 0 1 ?
Looks good. Now let’s just verify that R5 sees them as well.
R5#sh ip route ospf | be Gate Gateway of last resort is not set 126.96.36.199/32 is subnetted, 1 subnets O E2 188.8.131.52 [110/1] via 184.108.40.206, 00:03:33, FastEthernet1/0 [110/1] via 220.127.116.11, 00:05:18, FastEthernet1/0 18.104.22.168/24 is subnetted, 1 subnets O E2 22.214.171.124 [110/1] via 126.96.36.199, 00:03:33, FastEthernet1/0 [110/1] via 188.8.131.52, 00:05:18, FastEthernet1/0 184.108.40.206/24 is subnetted, 1 subnets O E2 220.127.116.11 [110/1] via 18.104.22.168, 00:03:33, FastEthernet1/0 [110/1] via 22.214.171.124, 00:05:18, FastEthernet1/0 126.96.36.199/24 is subnetted, 1 subnets O E2 188.8.131.52 [110/1] via 184.108.40.206, 00:03:33, FastEthernet1/0 [110/1] via 220.127.116.11, 00:05:18, FastEthernet1/0 18.104.22.168/24 is subnetted, 1 subnets O E2 22.214.171.124 [110/1] via 126.96.36.199, 00:03:33, FastEthernet1/0 [110/1] via 188.8.131.52, 00:05:18, FastEthernet1/0
ECMP is implemented since the cost is the same to ASBRs.
Final test is to ping 184.108.40.206.
R5#ping 220.127.116.11 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 18.104.22.168, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/24 ms
And it works! Pretty cool stuff. So now we have a lab with both IOS and IOS-XE.
If we had real switches we could throw this into the topology as well.
If you have enterprise ESX you could even practice Nexus with N1kv image.
So you see that we can build some great topologies here.
This does take some computing power, mostly RAM. If I check ESXi I see that
the Ubuntu box is using around 2GHz CPU and about 2GB RAM. The CSRs are using
each 750MHz CPU and 3GB RAM. So in total you are looking at around 4GHz CPU
and 8GB RAM. It is doable on a well equipped laptop.
Running Dynamips takes a lot of CPU and memory and running a full CCIE topology on
a Windows machine can be tough. I do a lot of studying on my commute to my job
and I run some smaller labs but I have not been able to run a full topology on
my laptop until now.
I came across a post on IEOC (Internetwork Experts forum) on how to
dramatically decrease CPU usage. Original credit goes to Journeyofanetworkengineer.
There is a value called idlemax which is related to the famous idle-PC value.
There is not much information on what this value does. According to Greg
at Hacki forum idlemax specifies how many times the address that the idle-PC
value references is used before going to sleep. The default value is 1500.
I’m not sure about the magic behind this, maybe someone with more expertise
in Dynamips can explain this but lowering this value dramatically decreases
the CPU usage.
I was able to run the full INE topology at 20-40% CPU load on my Core2 duo
@ 2.13 GHz and 4 GB RAM. Without idlemax applied my CPU runs at close to
This is before idlemax.
This is after idlemax.
This is what a hypervisor entry looks like in the .net file.
We will need four entries like this with an unique port for localhost and unique port for UDP.
You can download my complete .net file here if you need it for reference.
If you use this tip please post in comments how much your CPU was decreased and if you
have any stability issues when running it at 100.
Going to do a rack rental tonight since I’m going to do multicast labs. From what I’ve heard Dynamips has some issues with multicast, not sure if it still has issues but I’ll do a rack rental to not spend time on troubleshooting Dynamips when I could spend the time labbing and troubleshooting multicast. What is your experience with Dynamips and multicast? If you’ve tried it please leave a comment.
If you as I are studying for the CCIE R&S you should be using Putty? Why? Because for the
R&S track Putty is used in the lab. Yes, I an many others use SecureCRT and love the tabs
but you don’t want to go to the lab and do anything for the first time, except get your number!
Therefore we should use Putty to practice.
I use rack rentals and Dynamips for my studies, in either case I need to connect to a least 10
devices, the 6 routers and the 4 switches. This can be a bit tedious to do everytime you
want to study. Here is my solution for it. Start by installing Putty, I usually just use the
executable but you want to install it to be able to use it from the command line.
Download it here.
Install it to a directory of your preference.
Open a command prompt and type putty, what happens? Nothing, because Windows
doesn’t know where Putty is located. Either you can type C:\Program files\Putty\Putty.exe
or where you installed it everytime you want to use it but there is a better way.
Start by clicking the Start button and then right clicking on “Computer” and then
Then we will have a new screen where we whant to choose “Advanced system settings”.
After that choose “Environment Variables…”.
Scroll down the “System variables” until you see “Path”. Select this and then choose edit.
After every entry there needs to be a semicolon. Scroll to the right and after your
final entry put “;C:\Program files\PuTTY” or the directory where you installed it.
You do not need the “”.
Windows will now know where Putty is located. If you type Putty from the
command prompt Putty will load. The advantage now is that we can use Putty from the
command line, you can used putty -telnet or putty -ssh to open sessions from the
command line but we are interested in putty -load. This is how I have things setup.
We need to create sessions in the GUI that we can load from the command prompt.
If you are using rack rentals like graded labs you telnet to a terminal server and
can create sessions by using different usernames. You can see that I connect
to racks.ine.com on port 23.
Some things that you might like to change, in the “Windows” section change
the scrollback. I use 5000 lines of scrollback. You want to be able to run debugs
and then scroll back and go through the output.
I use a different font than standard, you can change this in “Appearance”.
The one I use is called Lucida Console.
I also like to change the selection behaviour. The default is to paste on right click.
I change it to bring up a menu on right click. If I want to paste something directly
I can use shift+insert.
This is enough to get us started but first lets make some more changes.
Under “Behaviour” we can set “Window title:”.
Set this to the device name like R1, R2, SW1 etc.
This will help us to know which device we are logged in to.
When you have edited your settings you need to save them to a session.
Choose “Session” and then “Saved Sessions” and enter a name for the session
and then click “Save”. To save more sessions leave the settings as they are, enter
a new name under “Saved sessions”, change the window title under behaviour, when
this is done click save and the session will be saved with the new name.
Do this for all the devices.
To load the sessions from the commandline we use putty -load.
I made a quick .bat file to start all my sessions. The code looks like this:
I have implemented a 2 second wait between each session because the sessions
time out quite fast and we need time to type username and password.
The timeout may not be availabe in systems older than Vista but you may
solve this by using a ping command or so. This bat-file can also be used to open
remote sessions to a Dynamips-server. The only thing that changes in the Putty
sessions is that we connect via ssh and then ports 2000 and upward.
If you connect to Dynamips you don’t need to implement the timeout command.
Hopefully this post has showed you new ways of using Putty and has speeded up your labstudies.
Some more updates to script:
- Had missed to remap some serial interfaces on r4-r6
- Port Fa0/6 on SW2 wasn’t remapped which led to connectivity issues between R1 and R6
Found some more bugs:
- Wasn’t matching specific enough in the sed statements for bb1
- Added 100/full to backbone routers ethernet interfaces
- Redid how I delete interfaces for backbone routers
Should be pretty close to finished now but I find something almost every day.
Found a lot of errors in my script and added some features. Things that got fixed:
- Searching for interfaces for r1-r6 that didn’t need to be changed
- Setting all interfaces to 100/full since Dynamips doesn’t like auto
- Some patterns weren’t specific enough which led to multiple entries
- Switchport mode dynamic desirable not available, set to hardcoded trunk
- Switchport host not available, using switchport mode access instead
- Delete some of interfaces on BB1 because we have separate frameswitch in topology
That should take care of most of the errors, if you find anything else please report it. Script is still avaible here.
So I have made some more findings about Dynamips
- Running more hypervisors is not more effecient than running one, will load balance over cores anyway
- GNS3 is just a hassle, stick with Dynagen if you have a set topology
This means that in my previous post it is only necessary to run two hypervisors, one local and one remote. I decided to run R1-R4 on my laptop and everything else on my other computer. This keeps the load at around 35% on my laptop and 75% on the other computer. It’s good that my laptop doesn’t get so high CPU load because I use it to run all the telnet sessions. I am using Putty to login to the console since this is what is used at the R&S lab.
As most CCIE candidates know Dynamips runs much better on Linux. Many of us run Windows at work or as our primary OS and might be afraid to mess around with partitioning and dual booting. There is a safer alternative that is called WUBI (Windows UBuntu Installer). WUBI installs Ubuntu like a regular program and it can be removed via add/remove programs. WUBI adds an option at boot to boot either Ubuntu or Windows. If you don’t choose anything Windows will boot. Get WUBI here.
I use my laptop to run labs but it’s not that powerful. It’s a Core 2 duo @ 2.13 GHz with 4GB RAM and runs Windows 7 Professional as primary OS. I can run the full INE topology at 75% CPU that is with only the base configs. I have another computer that is used as a HTPC, its a Core 2 duo @ 2.33 GHz with 2GB RAM and Windows XP as primary OS. I got the idea to try to load balance with the two computers. I started out booting Ubuntu on one computer and Windows on the other but I ran in to all sorts of issues. I decided to run Ubuntu on both computers and with WUBI its easy and safe to do so and still keep your Windows (take backups just in case!). Getting the load balancing to work turned out to be more difficult than expected, that is why I hope people can benefit from this blogpost.
Start by installing Ubuntu on both computers via WUBI and boot the computers. You need to install Dynamips and Dynagen, if you want to install GNS3 also. Either use the Synaptic package manager and search for the programs or from a terminal:
sudo apt-get install dynamips
Repeat this step for Dynagen and GNS3 if wanted. The great thing with Synaptic and apt-get is that it will check dependencies and download everything needed. If you want to control the second computer from the first one configure remote desktop settings.
When using both local and remote hypervisors you cannot use localhost in your .net file, use the real IP instead. Put entries like this in your .net file:
udp = 10000
workingdir = /Applications/Dynagen/sample_labs/internetworkexpert/working
image = /Applications/Dynagen/images/c3725-adventerprisek9-mz.124-15.T10.extracted.bin
ram = 128
ghostios = True
idlepc = 0x614ac21c
I use two hypervisors per computer so the next entry will be 192.168.1.65:7201 and udp = 10100. Then after that I have 192.168.1.67:7202 and udp = 10200 which is the other computer.
You need to create directories to put the files, this is done with mkdir:
The reason I created these directories is that is what INE uses in their .net file (I use their workbooks). You can download the .net files and base configs here.
Make sure that the dir ‘working’ that is created from the zip file has the right permissions (write) otherwise Dynagen will crash when it tries to write log files and other files to the dir.
INE and I use an uncompressed IOS to boot the devices faster. To uncompress the image you can use unzip:
warning [c3725-adventerprisek9-mz.124-15.T10.bin]: 28328 extra bytes at beginning or within zipfile
(attempting to process anyway)
To change the filename use mv
mv C3725-AD.bin c3725-adventerprisek9-mz.124-15.T10.extracted.bin
Now everything should be in place. Start GNS3 and choose ‘open network file’ and import the .net file. I will attach mine to show how it is done. If paths to directories or to the IOS image is wrong the import will fail so typing the right names is critical.
You should now have load balancing between the computers. I was able to go down from 75% CPU to around 45-50 with this setup. If you have any questions post them in comments. Here is the .net file.