Archive
Debug condition – conditional debugging
Sometimes we have a need for debugging and Cisco has a lot of options for debugging, this is one of the things why I definately prefer Cisco to HP or other network vendors.
I did some vol2 labbing yesterday and had a need to debug OSPF packets to verify that an adjacency was being form over unicast and not multicast. Sounds easy but when running labs with many neighbors there can be a lot of packets flowing and we might miss some important information when debugging. If we are debugging IP packets we can specify an ACL to narrow down the selection but I am not aware of such an option when debugging routing protocols, at least not for the IGP’s. What I found out is that there is an option for settings conditions on what to debug. This is how we do it.
Rack8R2#debug condition ?
application Application
called called number
calling calling
card card
glbp interface group
interface interface
ip IP address
mac-address MAC address
match-list apply the match-list
standby interface group
username username
vcid VC ID
vlan vlan
voice-port voice-port number
xconnect Xconnect conditional debugging on segment pair
So we have some different options we can used, the two most obvious are ip and interface. I will show how to debug OSPF packets coming in or out a specific interface, this will narrow down traffic a lot if we have multiple neighbors.
Rack8R2#debug condition interface serial 0/1
Condition 1 set
Only traffic coming in or out Serial 0/1 will be debugged. Use show debug condition to see what conditions has been set.
Rack8R2#show debug condition
Condition 1: interface Se0/1 (1 flags triggered)
Flags: Se0/1
Lets debug OSPF packets.
Rack8R2#debug ip ospf packet
OSPF packet debugging is on
Rack8R2#
*Mar 1 00:14:44.523: OSPF: rcv. v:2 t:1 l:48 rid:150.8.3.3
aid:0.0.0.0 chk:BB84 aut:0 auk: from Serial0/1
Rack8R2#
*Mar 1 00:14:53.987: OSPF: rcv. v:2 t:1 l:48 rid:150.8.3.3
aid:0.0.0.0 chk:BB84 aut:0 auk: from Serial0/1
Rack8R2#
*Mar 1 00:15:03.859: OSPF: rcv. v:2 t:1 l:48 rid:150.8.3.3
aid:0.0.0.0 chk:BB84 aut:0 auk: from Serial0/1
So this is a very handy command to use when the need for debugging arises. As every CCIE candidate we should avoid using Google to find this information and you can find the document describing this feature by going to Cisco.com -> Configure -> Products -> Cisco IOS and NX-OS Software -> Cisco IOS -> Cisco IOS Software Release 12.4 Family -> Cisco IOS Software Releases 12.4T -> Reference Guides -> Command References -> Debug
The direct URL is here.
First round of vol2 started
Did my first round of vol2 tonight and man it was brutal. It’s a total different thing trying to get all protocols running together nicely than running them each separately. Also when doing entire scenarios you have to think if I solve requirement X in this way will it break requirement Z later on? These things make the labs really challenging. I now know how much I still have to learn but that is the first step I guess to achieving something great. I’m still really motivated to keep this thing going though. Hopefully after a few sessions I will get into the groove of it 
MPLS added to flash cards
I’ve added some MPLS and MPLS VPN’s to the flash cards. The flash cards are covering a pretty wide base by now and I have finished the vol1 labs from INE (not all of them) and as I go through vol2 I will add more content to the flash cards. If you have feedback please post a comment. You can find the file here.
Vol1 finished
I’ve finished the Vol1 labs following the schedule from INE. That is about 25% of all the Vol1 labs. Will start doing the vol2 soon. I’ll be adding content to the flash cards with MPLS and MPLS VPN questions today or tomorrow when I’ve finished written them.
Flash cards – multicast added
Added some multicast content to the flash deck. There is now a total of 151 questions in the deck. The next section that will be added when I have finished the labs is MPLS VPN. As usual, the deck is here.
Troubleshooting multicast – RPF failure
In unicast routing we are interested in how to forward packets to their destination. In multicast routing we are interested where the source came from, multicast packets need to pass a RPF (Reverse Path Forwarding) check. Packets that are received on an interface are checked that the route back to the source is through the same interface, otherwise the RPF check will fail. RPF check failing is one of the most common errors in multicast networks. Lets look at the topology.
The goal of the scenario is that R6 should be able to ping SW4 which has joined multicast group 224.10.10.10. PIM dense mode has been enabled on R6 -> R4, between R4 and R5 PIM is only enabled on the frame-relay connection. PIM is enabled between R5 -> SW2 and SW2 -> SW4. Dense mode is being used. This is configuration from SW4.
Rack11SW4#sh run int vlan 10
Building configuration…
Current configuration : 96 bytes
!
interface Vlan10
ip address 155.11.10.10 255.255.255.0
ip igmp join-group 224.10.10.10
end
Ping from R6 to SW4.
Rack11R6#ping 224.10.10.10 re 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 224.10.10.10, timeout is 2 seconds:
…
Not successful, lets look at what multicast packets are being sent. We need to disable fast switching on the interface to see any packets.
Rack11R5(config)#int s0/0/0
Rack11R5(config-if)#no ip mroute-cache
Rack11R5(config-if)#^Z
Rack11R5#debug ip mpacket
IP multicast packets debugging is on
Rack11R5#
Feb 15 10:30:53.727: IP(0): s=155.11.146.6 (Serial0/0/0) d=224.10.10.10 id=93, ttl=253, prot=1, len=104
(100), RPF lookup failed for source
Feb 15 10:30:53.727: IP(0): s=155.11.146.6 (Serial0/0/0) d=224.10.10.10 id=93, ttl=253, prot=1, len=104
(100), not RPF interface
Rack11R5#
Feb 15 10:30:55.723: IP(0): s=155.11.146.6 (Serial0/0/0) d=224.10.10.10 id=94, ttl=253, prot=1, len=104
(100), not RPF interface
Rack11R5#
Feb 15 10:30:57.723: IP(0): s=155.11.146.6 (Serial0/0/0) d=224.10.10.10 id=95, ttl=253, prot=1, len=104
(100), not RPF interface
Packets are not coming in on the RPF interface. Lets look at the multicast routing table.
Rack11R5#sh ip mroute
IP Multicast Routing Table
Flags: D – Dense, S – Sparse, B – Bidir Group, s – SSM Group, C – Connected,
L – Local, P – Pruned, R – RP-bit set, F – Register flag,
T – SPT-bit set, J – Join SPT, M – MSDP created entry,
X – Proxy Join Timer Running, A – Candidate for MSDP Advertisement,
U – URD, I – Received Source Specific Host Report,
Z – Multicast Tunnel, z – MDT-data group sender,
Y – Joined MDT-data group, y – Sending to MDT-data group,
V – RD & Vector, v – Vector
Outgoing interface flags: H – Hardware switched, A – Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 224.10.10.10), 00:03:41/stopped, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:03:41/00:00:00
Serial0/0/0, Forward/Dense, 00:03:41/00:00:00
(155.11.146.6, 224.10.10.10), 00:00:02/00:02:57, flags:
Incoming interface: Null, RPF nbr 155.11.45.4
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:00:02/00:00:00
Serial0/0/0, Forward/Dense, 00:00:02/00:00:00
(*, 224.0.1.40), 00:39:35/00:02:54, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:38:30/00:00:00
Serial0/0/0, Forward/Dense, 00:14:20/00:00:00
We are interested in the (155.11.146.6, 224.10.10.10) which is a dense mode group. Our RPF neighbor is 155.11.45.4 which is the address of R4 on the S0/1/0 interface which is not enabled for PIM. How do we reach 155.11.146.6?
Rack11R5#sh ip mroute
IP Multicast Routing Table
Flags: D – Dense, S – Sparse, B – Bidir Group, s – SSM Group, C – Connected,
L – Local, P – Pruned, R – RP-bit set, F – Register flag,
T – SPT-bit set, J – Join SPT, M – MSDP created entry,
X – Proxy Join Timer Running, A – Candidate for MSDP Advertisement,
U – URD, I – Received Source Specific Host Report,
Z – Multicast Tunnel, z – MDT-data group sender,
Y – Joined MDT-data group, y – Sending to MDT-data group,
V – RD & Vector, v – Vector
Outgoing interface flags: H – Hardware switched, A – Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 224.10.10.10), 00:03:41/stopped, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:03:41/00:00:00
Serial0/0/0, Forward/Dense, 00:03:41/00:00:00
(155.11.146.6, 224.10.10.10), 00:00:02/00:02:57, flags:
Incoming interface: Null, RPF nbr 155.11.45.4
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:00:02/00:00:00
Serial0/0/0, Forward/Dense, 00:00:02/00:00:00
(*, 224.0.1.40), 00:39:35/00:02:54, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:38:30/00:00:00
Serial0/0/0, Forward/Dense, 00:14:20/00:00:00
Rack11R5#sh ip route 155.11.146.6
Routing entry for 155.11.146.0/24
Known via “eigrp 100″, distance 90, metric 2172416, type internal
Redistributing via eigrp 100, ospf 1
Advertised by ospf 1 subnets
Last update from 155.11.45.4 on Serial0/1/0, 00:55:04 ago
Routing Descriptor Blocks:
* 155.11.45.4, from 155.11.45.4, 00:55:04 ago, via Serial0/1/0
Route metric is 2172416, traffic share count is 1
Total delay is 20100 microseconds, minimum bandwidth is 1544 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1
Traffic to R6 is sent over the S0/1/0 interface which is not enabled for PIM, this is a problem…How can we pass the RPF check? By adding a static mroute we can enable the frame-relay interface to be a valid RPF interface.
Rack11R5(config)#ip mroute 155.11.146.6 255.255.255.255 155.11.0.4
The ping should now be successful.
Rack11R6#ping 224.10.10.10 re 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 224.10.10.10, timeout is 2 seconds:
Reply to request 0 from 155.11.108.10, 52 ms
Reply to request 1 from 155.11.108.10, 44 ms
Reply to request 2 from 155.11.108.10, 44 ms
Traffic is flowing, one final look at the mroute table.
Rack11R5#sh ip mroute
IP Multicast Routing Table
Flags: D – Dense, S – Sparse, B – Bidir Group, s – SSM Group, C – Connected,
L – Local, P – Pruned, R – RP-bit set, F – Register flag,
T – SPT-bit set, J – Join SPT, M – MSDP created entry,
X – Proxy Join Timer Running, A – Candidate for MSDP Advertisement,
U – URD, I – Received Source Specific Host Report,
Z – Multicast Tunnel, z – MDT-data group sender,
Y – Joined MDT-data group, y – Sending to MDT-data group,
V – RD & Vector, v – Vector
Outgoing interface flags: H – Hardware switched, A – Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 224.10.10.10), 00:06:53/stopped, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:06:53/00:00:00
Serial0/0/0, Forward/Dense, 00:06:53/00:00:00
(155.11.146.6, 224.10.10.10), 00:03:14/00:02:35, flags: T
Incoming interface: Serial0/0/0, RPF nbr 155.11.0.4, Mroute
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:03:14/00:00:00
(*, 224.0.1.40), 00:42:47/00:02:47, RP 0.0.0.0, flags: DCL
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
FastEthernet0/0, Forward/Dense, 00:41:42/00:00:00
Serial0/0/0, Forward/Dense, 00:17:32/00:00:00
The RPF neighbor is now 155.11.0.4 which is the next-hop over frame-relay. When doing multicast we need to think more about traffic patterns and ensuring that interfaces that are in the multicast transit path should be PIM enabled or not be used for multicast traffic.
Multicast session booked for tonight
Going to do a rack rental tonight since I’m going to do multicast labs. From what I’ve heard Dynamips has some issues with multicast, not sure if it still has issues but I’ll do a rack rental to not spend time on troubleshooting Dynamips when I could spend the time labbing and troubleshooting multicast. What is your experience with Dynamips and multicast? If you’ve tried it please leave a comment.
IPv6 content added to flash cards
I have added some IPv6 content to the flash cards. Next up is multicast. Will add more advanced content and more content as I start with the vol2 labs. I now only have multicast left from the vol1 labs (following the INE program) and then I’m moving on to vol2. You can find the file here.
BGP troubleshooting – route not installed
Sometimes prefixes in BGP do not get installed into the routing table, if the route is also in an IGP that might be a reason but then a RIB-failure would be indicated. This scenario shows another possible source of problems. Once again, the topology is this.
All internal routers are running iBGP in a full mesh. Routers R4 and R6 have eBGP peerings to the backbone routers which are injecting external prefixes into the AS. All internal routers are announcing their loopbacks into BGP. SW3 is trying to reach 119.0.0.1 in the prefix 119.0.0.0/8 but is unable to do so, lets look at some output.
Rack1SW3#ping 119.0.0.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 119.0.0.1, timeout is 2 seconds:
Packet sent with a source address of 150.1.9.9
…..
Success rate is 0 percent (0/5)
SW3 can’t reach 119.0.0.1, why?
Rack1SW3#sh ip route 119.0.0.0
% Network not in table
We have no route there, what routes can we see from BGP?
Rack1SW3#sh ip route bgp
150.1.0.0/24 is subnetted, 10 subnets
B 150.1.7.0 [200/0] via 155.1.79.7, 01:26:58
B 150.1.6.0 [200/0] via 155.1.67.6, 01:26:58
B 150.1.5.0 [200/0] via 155.1.45.5, 01:26:58
B 150.1.4.0 [200/0] via 155.1.146.4, 01:26:58
B 150.1.3.0 [200/0] via 155.1.37.3, 01:26:58
B 150.1.2.0 [200/0] via 155.1.23.2, 01:26:58
B 150.1.1.0 [200/0] via 155.1.146.1, 01:26:58
B 150.1.10.0 [200/0] via 155.1.108.10, 01:26:45
B 150.1.8.0 [200/0] via 155.1.58.8, 01:26:45
We can see all the loopbacks just fine but we have no route to the external prefixes. What is R6 announcing to us?
Rack1SW3# sh ip bgp nei 155.1.67.6 routes
BGP table version is 11, local router ID is 150.1.9.9
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
* i28.119.16.0/24 54.1.1.254 0 100 0 54 i
* i28.119.17.0/24 54.1.1.254 0 100 0 54 i
* i112.0.0.0 54.1.1.254 0 100 0 54 50 60 i
* i113.0.0.0 54.1.1.254 0 100 0 54 50 60 i
* i114.0.0.0 54.1.1.254 0 100 0 54 i
* i115.0.0.0 54.1.1.254 0 100 0 54 i
* i116.0.0.0 54.1.1.254 0 100 0 54 i
* i117.0.0.0 54.1.1.254 0 100 0 54 i
* i118.0.0.0 54.1.1.254 0 100 0 54 i
* i119.0.0.0 54.1.1.254 0 100 0 54 i
*150.1.6.0/24 155.1.67.6 0 100 0 i
Total number of prefixes 11
R6 is announcing the external prefixes to us but what do we have in our BGP table? Output has been abbreviated.
Rack1SW3#sh ip bgp
BGP table version is 11, local router ID is 150.1.9.9
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
* i119.0.0.0 204.12.1.254 0 100 0 54 i
* i 54.1.1.254 0 100 0 54 i
So we do have 119.0.0.0/8 via 204.12.1.254 and 54.1.1.254 but how do we get to the next-hops, remember that route recursion will occur and that the first rule of the BGP best path is that we must have a valid next-hop. We can see that the route is valid but not best.
Rack1SW3#sh ip route 54.1.1.254
% Network not in table
We have an invalid next-hop, so that is why the route is not being installed, lets fix this.
Rack1R6(config)#router eigrp 100
Rack1R6(config-router)#network 54.1.1.0 0.0.0.255
Rack1R4(config)#router eigrp 100
Rack1R4(config-router)#network 204.12.1.0 0.0.0.255
That should take care of the next-hops, lets check the routing table.
Rack1SW3#sh ip route 54.1.1.254
Routing entry for 54.1.1.0/24
Known via “eigrp 100″, distance 90, metric 2174976, type internal
Redistributing via eigrp 100
Last update from 155.1.79.7 on Vlan79, 00:02:04 ago
Routing Descriptor Blocks:
* 155.1.79.7, from 155.1.79.7, 00:02:04 ago, via Vlan79
Route metric is 2174976, traffic share count is 1
Total delay is 20200 microseconds, minimum bandwidth is 1544 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 2
We now have a route for the next-hop. Lets look at the BGP table again.
Rack1SW3#sh ip bgp
BGP table version is 31, local router ID is 150.1.9.9
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete
Network Next Hop Metric LocPrf Weight Path
*i119.0.0.0 204.12.1.254 0 100 0 54 i
* i 54.1.1.254 0 100 0 54 i
So the path is now have a best path, is it in the routing table?
Rack1SW3#sh ip route bgp
B 119.0.0.0/8 [200/0] via 204.12.1.254, 00:02:27
Route is installed, we should be good to go.
Rack1SW3#ping 119.0.0.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 119.0.0.1, timeout is 2 seconds:
Packet sent with a source address of 150.1.9.9
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/37/84 ms
Success. Always remember to have a valid next-hop in BGP. Next-hops are modified over eBGP peerings but not over iBGP. To resolve this kind of problem either redistribute connected interface to the external peer into IGP or use next-hop-self on iBGP peerings. A route-map can also be used to achieve the same thing. I hope this post has showed you how to do BGP troubleshooting step by step.
Educational IOS
Greg at Etherealmind has a call for educational IOS up on his site. It had to be redone because the signatures were lost from the first round. Dynamips is great but it has it’s issues and takes a lot of horsepower. Using IOU is illegal, it would be for the best if Cisco made an educational IOS available, an IOS that has all features but cripples the throughput which doesn’t matter in a lab anyway. Cisco is now also using IOU in the CCIE lab for the troubleshooting, maybe they could release this in some form? Please sign the petition here.
