I’m in final preparation for my second attempt and I have been doing a lot of troubleshooting scenarios lately. I created a MPLS topology in GNS3 and sent it to my friend Darren for testing. He is taking his lab very soon and he performed well on this lab. The lab contains multiple faults but I won’t say how many since that would spoil some of the surprise.
The assignment is to make sure CE1 can ping CE2 loopback 18.104.22.168.
Post in comments what you did to make it work or if you need a hint to get you going in the right direction. You need to edit the .net file to use your own working dir and IOS image. You need IOS images for 3725 and 7200. Start with the configurations provided by importing the configs or simply pasting them in whatever you prefer but you should not look at the startup config before starting.
Download the .net and config files here.
This is what the topology looks like.
Did some of these TS labs this week. They are very challenging but they are designed by Petr Lapukhov so they should be… I found these to be more difficult than the ASET TS labs. From what I’ve heard from Petr these labs are not specifically designed to help you do the TS at the lab but more of a learning tool to become very proficient in TS. If you can solve these tasks my guess is that the TS at the lab should not be that difficult except that the lab topology is much larger.
While doing one of the tasks I learned a new cool command from the solution guide, debug ip packet detail dump. The dump at the end is a hidden command and will show the contents of the packet. This can be useful when troubleshooting authentication if the key is in plain text. Newer versions of IOS seem to show the key in log messages but it could still be handy. This is how we use it.
As you can see, I do all of my labs with PuTTY to be comfortable with it on the exam.
Did another TS lab yesterday. This one was a bit more challenging than the first. If I grade myself (no auto grading) I would have 6 or 7 out of 10 correct which is not too far away from the 80% passing score. I did not expect to be an expert at TS yet but this shows that I am on the right path. One thing that is annoying is that you don’t know the initial configurations and there is no solutions guide. You only get the final configurations that includes the correct configuration.
The next time I do one of these I think I will do a show run on all routers and download the config and then download final configs and do a diff to see what is different.
During these labs I have noticed that the wording is very important. Look at the following example
.6 TROUBLE TICKET 6
R10 and R11 are not seeing routes for the R21-R28 network or for the VPN “Foo” host 22.214.171.124. Determine the cause and correct the issue.
When I did the lab I interpreted this as that networks behind routers R21 to R28 are not reachable. However I think that what they really mean is that the link connecting R21 to R28 (running RIP) is not reachable in the domain. Depending on how you read the task you will get a very different result and do a lot of unnecessary steps.
I like that the topology is large since that is what we can expect at the lab. The user experience with topology diagram and connecting to routers seems to be similar to the real thing if we compare to the lab exam demo.
I might try some of the configuration labs later but for now I am mainly focusing on INE material.
In unicast routing we are interested in how to forward packets to their destination. In multicast routing we are interested where the source came from, multicast packets need to pass a RPF (Reverse Path Forwarding) check. Packets that are received on an interface are checked that the route back to the source is through the same interface, otherwise the RPF check will fail. RPF check failing is one of the most common errors in multicast networks. Lets look at the topology.
The goal of the scenario is that R6 should be able to ping SW4 which has joined multicast group 126.96.36.199. PIM dense mode has been enabled on R6 -> R4, between R4 and R5 PIM is only enabled on the frame-relay connection. PIM is enabled between R5 -> SW2 and SW2 -> SW4. Dense mode is being used. This is configuration from SW4.
Ping from R6 to SW4.
Not successful, lets look at what multicast packets are being sent. We need to disable fast switching on the interface to see any packets.
Packets are not coming in on the RPF interface. Lets look at the multicast routing table.
We are interested in the (188.8.131.52, 184.108.40.206) which is a dense mode group. Our RPF neighbor is 220.127.116.11 which is the address of R4 on the S0/1/0 interface which is not enabled for PIM. How do we reach 18.104.22.168?
Traffic to R6 is sent over the S0/1/0 interface which is not enabled for PIM, this is a problem…How can we pass the RPF check? By adding a static mroute we can enable the frame-relay interface to be a valid RPF interface.
The ping should now be successful.
Traffic is flowing, one final look at the mroute table.
The RPF neighbor is now 22.214.171.124 which is the next-hop over frame-relay. When doing multicast we need to think more about traffic patterns and ensuring that interfaces that are in the multicast transit path should be PIM enabled or not be used for multicast traffic.
Sometimes prefixes in BGP do not get installed into the routing table, if the route is also in an IGP that might be a reason but then a RIB-failure would be indicated. This scenario shows another possible source of problems. Once again, the topology is this.
All internal routers are running iBGP in a full mesh. Routers R4 and R6 have eBGP peerings to the backbone routers which are injecting external prefixes into the AS. All internal routers are announcing their loopbacks into BGP. SW3 is trying to reach 126.96.36.199 in the prefix 188.8.131.52/8 but is unable to do so, lets look at some output.
SW3 can’t reach 184.108.40.206, why?
We have no route there, what routes can we see from BGP?
We can see all the loopbacks just fine but we have no route to the external prefixes. What is R6 announcing to us?
R6 is announcing the external prefixes to us but what do we have in our BGP table? Output has been abbreviated.
So we do have 220.127.116.11/8 via 18.104.22.168 and 22.214.171.124 but how do we get to the next-hops, remember that route recursion will occur and that the first rule of the BGP best path is that we must have a valid next-hop. We can see that the route is valid but not best.
We have an invalid next-hop, so that is why the route is not being installed, lets fix this.
That should take care of the next-hops, lets check the routing table.
We now have a route for the next-hop. Lets look at the BGP table again.
So the path is now have a best path, is it in the routing table?
Route is installed, we should be good to go.
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 126.96.36.199, timeout is 2 seconds:
Packet sent with a source address of 188.8.131.52
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/37/84 ms
Success. Always remember to have a valid next-hop in BGP. Next-hops are modified over eBGP peerings but not over iBGP. To resolve this kind of problem either redistribute connected interface to the external peer into IGP or use next-hop-self on iBGP peerings. A route-map can also be used to achieve the same thing. I hope this post has showed you how to do BGP troubleshooting step by step.
Yesterday I did some Internetwork Expert vol1 labs on BGP. I was having trouble getting some of the peers to come up and had to troubleshoot. This post will describe how to troubleshoot when peers won’t form. First, lets look at the topology. Thanks to DennisD on IEOC forums for the image.
R2 and R5 should peer with each other in AS 100. R2 is setup to peer with R5′s IP 184.108.40.206 and R5 is setup to peer with R2′s IP 220.127.116.11. It would have been better to peer over the 18.104.22.168/24 subnet directly but this is to show the steps of troubleshooting. So the session will not form, why? Lets look at some output from debug ip tcp transactions.
We can see that R5 is initiating the connection, it is sending a TCP SYN to R2 on port 179 but R2 responds with a TCP RST which resets the connection. This could indicate that either R2 is not running BGP or that their is a problem with the neighbor statements.
So we want to know what IP R5 is using when sending TCP packets to R2. Lets debug IP packets.
R5 is using its IP of 22.214.171.124 to communicate with 126.96.36.199 but R2 expects R5 to setup the BGP session from the IP of 188.8.131.52. Let’s verify why R5 is using 184.108.40.206 to get to 220.127.116.11. This is a look at the routing table.
We can see that R5 has two equal cost paths to reach the IP of R2. The next hop is either 18.104.22.168 or 22.214.171.124 and these are reachable via the connected subnet of Serial0/0. That is why R5 is using the IP of 126.96.36.199 to source packets. How can we solve this? Either we can setup the neighbor statement to point at 188.8.131.52 or we can change the update-source.
A debug IP packet confirms that the right interface is now being used.
Show ip bgp confirms that they are now peers.
Show tcp brief is a good command to see TCP sessions to/from the router.
And this is how to do basic BGP troubleshooting.