If you've ever done a traceroute from one IOS box to another, you've undoubtedly seen output like this:

R8# traceroute 192.168.100.7
Tracing the route to 192.168.100.7
VRF info: (vrf in name/id, vrf out name/id)
  1 192.168.0.1 4 msec 3 msec 4 msec
  2 192.168.100.7 4 msec *  0 msec

That msec * msec output. Why is the middle packet always lost?? And why only on the last hop??

This was always something curious to me but not something I ever bothered to learn about. Well it turns out that IOS has a rate limiter that meters the generation of ICMP Unreachable messages. The default setting for the rate limiter is 1 ICMP Unreach every 500ms. Since IOS's traceroute doesn't put a delay between its probe packets, the delay between when 192.168.100.7 receives the first and second probe packets is much less than 500ms. The second packet violates the rate limiter and so 192.168.100.7 drops it.

Why isn't the third packet also dropped? Because the traceroute command waits for 3 seconds (by default) before deciding that a probe packet was lost and generating the next packet. This makes the delay between the second and third packet 3 seconds during which time the rate limiter resets allowing 192.168.100.7 to respond to that third packet.

Why doesn't 192.168.0.1 also drop the second packet? Because the probe packets aren't actually destined to 192.168.0.1, they're only transiting that router. Before IOS forwards a packet onwards, it decrements the Time To Live in the IP header. If the TTL is 0 after it's decremented, IOS tosses the packet away and issues an ICMP Time Exceeded message. The Time Exceeded message is not subject to the rate limiter. The rate limiter is only applicable to ICMP Unreachable packets and ICMP Unreachables will only be generated by the host that the traceroute is directed to.

The command to modify the rate limiter's behavior is:

(config)# ip icmp rate-limit unreachable <ms>

Where msis the time interval in milliseconds over which 1 ICMP Unreach will be sent. Eg, if you want to raise the ICMP Unreach rate to 5 per second, you would use an ms value of 200.

This rate limiter is also responsible for the "U.U.U" output you see when you ping a router that has an ACL applied on the interface that denies ICMP. The "U" responses are the ICMP Unreachables coming back and the "." responses are actually indicating the remote router did not reply at all. By adjusting the rate limiter, the remote router can be made to reply to every ping packet.

R10#ping 192.168.7.12
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.7.12, timeout is 2 seconds:
UUUUU
Success rate is 0 percent (0/5)

More information:


Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Cisco Systems.