Sometimes we have a need for debugging and Cisco has a lot of options for debugging, this is one of the things why I definately prefer Cisco to HP or other network vendors.
I did some vol2 labbing yesterday and had a need to debug OSPF packets to verify that an adjacency was being form over unicast and not multicast. Sounds easy but when running labs with many neighbors there can be a lot of packets flowing and we might miss some important information when debugging. If we are debugging IP packets we can specify an ACL to narrow down the selection but I am not aware of such an option when debugging routing protocols, at least not for the IGP’s. What I found out is that there is an option for settings conditions on what to debug. This is how we do it.
So we have some different options we can used, the two most obvious are ip and interface. I will show how to debug OSPF packets coming in or out a specific interface, this will narrow down traffic a lot if we have multiple neighbors.
Only traffic coming in or out Serial 0/1 will be debugged. Use show debug condition to see what conditions has been set.
Lets debug OSPF packets.
So this is a very handy command to use when the need for debugging arises. As every CCIE candidate we should avoid using Google to find this information and you can find the document describing this feature by going to Cisco.com -> Configure -> Products -> Cisco IOS and NX-OS Software -> Cisco IOS -> Cisco IOS Software Release 12.4 Family -> Cisco IOS Software Releases 12.4T -> Reference Guides -> Command References -> Debug
The direct URL is here.
Did my first round of vol2 tonight and man it was brutal. It’s a total different thing trying to get all protocols running together nicely than running them each separately. Also when doing entire scenarios you have to think if I solve requirement X in this way will it break requirement Z later on? These things make the labs really challenging. I now know how much I still have to learn but that is the first step I guess to achieving something great. I’m still really motivated to keep this thing going though. Hopefully after a few sessions I will get into the groove of it
I’ve added some MPLS and MPLS VPN’s to the flash cards. The flash cards are covering a pretty wide base by now and I have finished the vol1 labs from INE (not all of them) and as I go through vol2 I will add more content to the flash cards. If you have feedback please post a comment. You can find the file here.
I’ve finished the Vol1 labs following the schedule from INE. That is about 25% of all the Vol1 labs. Will start doing the vol2 soon. I’ll be adding content to the flash cards with MPLS and MPLS VPN questions today or tomorrow when I’ve finished written them.
Added some multicast content to the flash deck. There is now a total of 151 questions in the deck. The next section that will be added when I have finished the labs is MPLS VPN. As usual, the deck is here.
In unicast routing we are interested in how to forward packets to their destination. In multicast routing we are interested where the source came from, multicast packets need to pass a RPF (Reverse Path Forwarding) check. Packets that are received on an interface are checked that the route back to the source is through the same interface, otherwise the RPF check will fail. RPF check failing is one of the most common errors in multicast networks. Lets look at the topology.
The goal of the scenario is that R6 should be able to ping SW4 which has joined multicast group 188.8.131.52. PIM dense mode has been enabled on R6 -> R4, between R4 and R5 PIM is only enabled on the frame-relay connection. PIM is enabled between R5 -> SW2 and SW2 -> SW4. Dense mode is being used. This is configuration from SW4.
Ping from R6 to SW4.
Not successful, lets look at what multicast packets are being sent. We need to disable fast switching on the interface to see any packets.
Packets are not coming in on the RPF interface. Lets look at the multicast routing table.
We are interested in the (184.108.40.206, 220.127.116.11) which is a dense mode group. Our RPF neighbor is 18.104.22.168 which is the address of R4 on the S0/1/0 interface which is not enabled for PIM. How do we reach 22.214.171.124?
Traffic to R6 is sent over the S0/1/0 interface which is not enabled for PIM, this is a problem…How can we pass the RPF check? By adding a static mroute we can enable the frame-relay interface to be a valid RPF interface.
The ping should now be successful.
Traffic is flowing, one final look at the mroute table.
The RPF neighbor is now 126.96.36.199 which is the next-hop over frame-relay. When doing multicast we need to think more about traffic patterns and ensuring that interfaces that are in the multicast transit path should be PIM enabled or not be used for multicast traffic.
Going to do a rack rental tonight since I’m going to do multicast labs. From what I’ve heard Dynamips has some issues with multicast, not sure if it still has issues but I’ll do a rack rental to not spend time on troubleshooting Dynamips when I could spend the time labbing and troubleshooting multicast. What is your experience with Dynamips and multicast? If you’ve tried it please leave a comment.
I have added some IPv6 content to the flash cards. Next up is multicast. Will add more advanced content and more content as I start with the vol2 labs. I now only have multicast left from the vol1 labs (following the INE program) and then I’m moving on to vol2. You can find the file here.
Sometimes prefixes in BGP do not get installed into the routing table, if the route is also in an IGP that might be a reason but then a RIB-failure would be indicated. This scenario shows another possible source of problems. Once again, the topology is this.
All internal routers are running iBGP in a full mesh. Routers R4 and R6 have eBGP peerings to the backbone routers which are injecting external prefixes into the AS. All internal routers are announcing their loopbacks into BGP. SW3 is trying to reach 188.8.131.52 in the prefix 184.108.40.206/8 but is unable to do so, lets look at some output.
SW3 can’t reach 220.127.116.11, why?
We have no route there, what routes can we see from BGP?
We can see all the loopbacks just fine but we have no route to the external prefixes. What is R6 announcing to us?
R6 is announcing the external prefixes to us but what do we have in our BGP table? Output has been abbreviated.
So we do have 18.104.22.168/8 via 22.214.171.124 and 126.96.36.199 but how do we get to the next-hops, remember that route recursion will occur and that the first rule of the BGP best path is that we must have a valid next-hop. We can see that the route is valid but not best.
We have an invalid next-hop, so that is why the route is not being installed, lets fix this.
That should take care of the next-hops, lets check the routing table.
We now have a route for the next-hop. Lets look at the BGP table again.
So the path is now have a best path, is it in the routing table?
Route is installed, we should be good to go.
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 188.8.131.52, timeout is 2 seconds:
Packet sent with a source address of 184.108.40.206
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/37/84 ms
Success. Always remember to have a valid next-hop in BGP. Next-hops are modified over eBGP peerings but not over iBGP. To resolve this kind of problem either redistribute connected interface to the external peer into IGP or use next-hop-self on iBGP peerings. A route-map can also be used to achieve the same thing. I hope this post has showed you how to do BGP troubleshooting step by step.
Greg at Etherealmind has a call for educational IOS up on his site. It had to be redone because the signatures were lost from the first round. Dynamips is great but it has it’s issues and takes a lot of horsepower. Using IOU is illegal, it would be for the best if Cisco made an educational IOS available, an IOS that has all features but cripples the throughput which doesn’t matter in a lab anyway. Cisco is now also using IOU in the CCIE lab for the troubleshooting, maybe they could release this in some form? Please sign the petition here.