This post will describe something interesting that happens to traceroute
when you are doing NAT. The inspiration for this post came from a
thread at IEOC started by Nick O’neill. He was doing ip nat outside translation but I
have found that the same behavior is true also for inside translations.
We will use this topology:
R1 has a loopback 188.8.131.52/32 on the inside. On the outside this will be known as
100.100.100.100. You can see the networks connecting the routers are 184.108.40.206/24,
220.127.116.11/24 and 18.104.22.168/24. R3 will be the router doing NAT and this is the
configuration of it:
interface FastEthernet0/0 ip address 22.214.171.124 255.255.255.0 ip nat inside ! interface FastEthernet0/1 ip address 126.96.36.199 255.255.255.0 ip nat outside ! ip route 188.8.131.52 255.255.255.255 184.108.40.206 ! ip nat inside source static 220.127.116.11 100.100.100.100
This is not a post describing the basics but make sure that you have routing setup.
I am using static routing here. When doing static NAT like this there is bidirectional
communication setup. Remember that when the outside to inside translation is done NAT
is performed before the routing lookup so we must have a route to 18.104.22.168 on R3.
The command above translates the inside local address to an inside global address.
R4 will traceroute to 100.100.100.100 and we will see what happens.
R4#traceroute 100.100.100.100 num prob 1 Type escape sequence to abort. Tracing the route to 100.100.100.100 1 22.214.171.124 28 msec 2 100.100.100.100 124 msec 3 100.100.100.100 84 msec
Interesting. Hops 2 and 3 are the same and they have the inside global address as source.
What happened here? The first hop is correct. The ICMP TTL exceeded should come back from
the outgoing interface used to reach 126.96.36.199 which was the source of the UDP traceroute
packet sent. Hop 2 should have been 188.8.131.52.
If we compare it to MPLS this almost looks like some core hiding feature as it does not
reveal the internal addressing of our network. If we look at debug on R3 we will see
what is happening.
R3# debug ip nat det IP NAT detailed debugging is on
NAT: i: icmp (184.108.40.206, 33435) -> (220.127.116.11, 49164)  NAT: s=18.104.22.168->100.100.100.100, d=22.214.171.124  NAT: i: icmp (126.96.36.199, 33436) -> (188.8.131.52, 49165)  NAT: s=184.108.40.206->100.100.100.100, d=220.127.116.11 
We clearly see that the inside addresses are being translated to 100.100.100.100
even though there is no matching NAT statement. The number 33435 is the source
port number and 49164 is the destination port number.
This is the IOS version used for this lab:
R3#sh ver | i IOS Cisco IOS Software, 3700 Software (C3725-ADVENTERPRISEK9-M), Version 12.4(15)T10, RELEASE SOFTWARE (fc3)
Now if we try it on mainline IOS instead…
R3#sh ver | i IOS Cisco IOS Software, 3700 Software (C3725-ADVIPSERVICESK9-M), Version 12.4(25d), RELEASE SOFTWARE (fc1)
R4#trace 100.100.100.100 num prob 1 Type escape sequence to abort. Tracing the route to 100.100.100.100 1 18.104.22.168 68 msec 2 22.214.171.124 164 msec 3 100.100.100.100 196 msec
As you can see with the mainline IOS we can see the real addresses as they are not
Knowing me you know that I like to dig deep to find out what is going on. I reached
out to a contact at Cisco to see if he could find any reference to what is happening
here. It turns out that this is related to bug CSCsu37097. Basically the customer
didn’t want the internal addressing revealed and so this was incorporated into IOS.
The symptom was described as: “Symptom:
Traceroute/ICMP unreachable doesn’t translate properly and can cause security problem.”
The following quote is also available from Cisco: “In Cisco IOS Release 15.1(3)T and
later releases, when you configure the traceroute command, NAT returns the same
inside global IP address for all inside local IP addresses.” The source is this
So basically this was implemented as a security feature for NAT. I hope you
learned something interesting.
I’m going through the blueprint and now I checked off IP accounting. The feature is very simple, it lets us see which source destination pairs that are sending traffic to each other. We can also configure to look what precedence values that are in the packets. There is also an option to look at the MAC-addresses of the packets passing through and also packets that are being denied by an access-list. The topology is dead simple, see below.
Configure your routing protocol of choice to get reachability. I’m using OSPF, it does not matter at all as long as you have connectivity. Now lets say that we are interested in which source and destination pairs that are sending traffic THROUGH the router (transit). Packets destined TO the router will not be seen in the accounting. I’ll configure accounting on R2′s interface to R1 and then initiate a ping from R1 to R3. I’ll send traffic both to the loopback and R3′s FastEthernet interface to see two different source/destination pairs.
OK, lets ping.
Now we will check the accounting database with the show ip accounting command.
So that shows us what sources/destinations are sending traffic to each other, interesting! We can also see the number of packets and number of bytes. If we want to check statistics for only certain hosts we can use the global ip accounting-list command to define what hosts we are interested in. We define hosts/networks as in ACL with network/wilcard combination.
Storing entries in the IP accounting database requires some memory, there could be a risk of exhaustion if we have too many entries but the default is set to max 512 entries. We can define this with the global ip accounting-threshold command.
So now we want to check what IP precedence values pass through our interfaces and also what MAC addresses that are sending/receiving traffic. Lets configure this.
Then we send some pings from R1, I will send with a ToS of 128, what IP precedence/DSCP is that? Think quick.
Lets verify at R2 if we see anything, the command to use is show interface precedence.
So a ToS of 128 was a IP prec of 4 but you already figured that, right? What is that traffic with IP prec 6? Mysterious…We are running routing so that is OSPF which is marked with an IP precedence of 6 automatically by the router itself. We can also check what MAC addresses have been learned.
Here we also see OSPF represented by the MAC address 01-00-5E-00-00-05. We can also see when the last packet was sent which is quite handy. Now we will turn on accounting for access-lists as well, first we will define an ACL denying ICMP to 126.96.36.199 which is the loopback of R3. Note that we need the log keyword in the ACL.
Now we send traffic from R1 to 188.8.131.52.
For some reason I don’t see anything with the show ip accounting access-violations. Maybe this is a software issue? I tried turning off CEF as well. If any of my readers get this working I would be interested.
Lastly lets have a brief look at how traceroute works in IOS. Cisco devices uses UDP traceroute compared to ICMP used by Windows. The router sends packets with TTL of 1 and then N+1 the further away the probe goes. Traceroute sends three packets for every hop. The first hop will have a destination port of 33435, the second one will have 33436 and so on. If we want a router to not respond to traceroute we can turn off IP unreachables. Note that this will not hinder traceroute for which this router is not the final destination. Only the final device will send an ICMP unreachable (port unreachable) which is ICMP code 3. The other routers will send time exceeded which is ICMP code 11.
If we did want to block traceroute going through the router we could block this with an ACL denying packets that have ttl-exceeded or all packets lower than a certain TTL. If we need to find ICMP codes we can reference the ASA library. This should be available at the lab. You can find the reference by following this path.
Products > Security > Firewalls > Firewall Appliances > Cisco ASA 5500 Series Adaptive Security Appliances > Configure > Configuration Guides > Cisco ASA 5500 Series Configuration Guide using the CLI, 8.4 > Reference > Addresses, Protocols, and Ports > ICMP types
So this is just another feature that is handy to have.
I found a very useful tool when practicing the INE labs. How to generate
traffic with traceroute. I’ve used telnet lots of times to generate TCP
traffic on different ports but what if we want to generate UDP traffic instead?
We can used traceroute to our advantage.
The topology is the one I’ve been using for my last posts with two routers
connected by a FastEthernet link.
First we create an access-list on R1 that will deny UDP on ports 9 and 19
but allow everything else.
We will confirm connectivity by doing a ping and then a telnet.
The traffic is passing successfully. Lets check the access-list on R1.
We have matches in the ACL, now lets generate traffic with traceroute.
We will type traceroute and then enter the options.
The important thing here is of course to change the port to something else
than the default port 33434. You can see by the !A in the answer that the
traffic was prohibited. Lets confirm this with looking at the ACL on R1.
And that is how you generate traffic with traceroute. Combined with the telnet
tool we can pretty much simulate most of TCP or UDP traffic. This gives us an
advantage in the lab so that we may test our ACLs to see that they are working
As I make progress through the INE workbooks I’m writing posts about features that I find interesting and that might not be that known to the general public. I wasn’t aware that you could filter packets based on TTL in IOS. I’m not sure where this would be used in the real world but one use could be to filter BGP packets coming in from external peers and checking the TTL of the packets. BGP does this by itself, but one scenario could be someone trying to flood BGP packets towards a router and then it might be better to filter them in an ACL then to let the CPU handle the packets. One important note when doing TTL filtering, look at this picture.
On ingress the ACL is checked before the TTL is decremented. On egress the ACL is checked after the TTL has been decremented.
Lets take a look at the configuration.
So we have a few options here, we can match on an exact TTL or a range or a TTL less than or greater than a value. We have a lot of options. In this example we will filter packets with TTL less than 3.
Packets with TTL less than 3 are denied and the rest are allowed. We need to apply the ACL to an interface, we are filtering packets outbound in this example.
This is how the ACL looks so far.
Let’s try a traceroute. The traceroute command can set a min and max TTL. If we set it to min 4 the packet will pass and we will see hop 4 and onward in the trace. If we set it to 1 the packet will be filtered.
This is the log output.
And finally, we have matches in the ACL.
So this post has showed how we can filter packets based on TTL in IOS. Post feedback in comments if you like these posts.