In our work environment we are always surrounded by people who are either smarter than us or act smarter than us. If we are surrounded by people who are smarter than us, the advantage of working with such people is that you will learn a lot from them, this obviously if they are the good ones who happily share knowledge. If they are unwilling to share knowledge, their presence or absence doesn’t make a difference. If we are surrounded by people who just act smart rather than are smart, we must be very careful as the knowledge which they would be sharing might not always be right as they just want to show that they know while they know not. This can be very bad because at the learning stage if you learn all the wrong things, your foundation will be built on wrong information. The best thing anyone who does not know certain thing can do is just say that he doesn’t know so that we can search for the right reason. The best thing about knowledge is that it is no ones property. Whoever desires to acquire knowledge can easily do it.

What is Traceroute?

Traceroute is a utility to check the path a packet takes across a network and it is a very important tool for troubleshooting. If you are having connectivity issue to a destination, trace the destination and on the hop the packets start dropping, there is a very good chance that the hop is having congestion.

The reason for writing this blog is because there are times when a tool as important as traceroute is rendered useless because it drops packets on the links where there are no congestion. One of the reason why trace drops packets is when large service providers rate limit the icmp packets to a certain bandwidth. Whenever the bandwidth meant for icmp exceeds the limit, the trace packets will be dropped. The other reason why trace drops packets is because of the way Trace works. In the next section, we will discuss the different implementations of traceroute.

Different Implementations of Traceroute

I was working in one of the largest ISPs in India and we used to sit adjacent to the Network Security team. There was a case where icmp was allowed for a customer server and I was expecting that the server should be traceable. My colleague told me that he will have to create a policy for allowing trace and I was like “What?”. This is because there are 2 major implementations of Traceroute.

Windows Tracert

Windows tracert primarily relies on icmp packets for tracing to destination. In order to illustrate it, I will use an example. I am tracing from my Laptop to one of the routers on the Internet. My PC will generate an icmp request packet with source ip address of my laptop and destination ip address of the router with ttl of 1. When this packet reaches my gateway, the ttl will become 0 and the gateway will reply with time to live exceeded icmp message. From this packet, the tracert function will get detail about the first hop ip address and the round trip times. Similarly, the second icmp packet will be built with ttl of 2. The first router will get the packet and forward it towards the destination. When this packet reaches the second hop, the ttl will become 0 and the router will  reply with time to live exceeded icmp message. From this packet, the tracert function will get detail about the second hop and the round trip times. The process will be continued until the packet reaches the last hop and instead of giving time to live exceeded icmp message, the router replies with an icmp reply packet and the trace is complete. You can find my trace to a router 46.31.71.81

tracert -d 46.31.71.81

Tracing route to 46.31.71.81 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 188.231.0.252
2 <1 ms <1 ms <1 ms 46.31.71.57
3 <1 ms <1 ms <1 ms 46.31.70.1
4 1 ms 1 ms 1 ms 46.31.71.1
5 137 ms 137 ms 136 ms 46.31.71.81

Trace complete.

Also, find below the corresponding packet capture.

*NIX Traceroute

The way in which Unix, Linux, Cisco, etc. traceroute works is by using udp datagrams. The outgoing packets will be udp packets with source ip address of your Laptop and destination ip address of the Internet router with destination port of 33434. Generally the port number keeps on increasing from 33434 as the ttl increases. To illustrate the traceroute functionality, we will use the same example we used for windows tracert.  I am tracing from my Laptop to one of the routers on the Internet. My *nix Laptop will generate a udp packet with source ip address of my laptop and destination ip address of the router with ttl of 1 and destination port of 33434. When this packet reaches my gateway, the ttl will become 0 and the gateway will reply with time to live exceeded icmp message. From this packet, the tracert function will get detail about the first hop ip address and the round trip times. Similarly, the second udp packet will be built with ttl of 2 and destination port of 33434. The first router will get the packet and forward it towards the destination. When this packet reaches the second hop, the ttl will become 0 and the router will  reply with time to live exceeded icmp message. From this packet, the tracert function will get detail about the second hop and the round trip times. The process will be continued until the packet reaches the last hop. At the last hop, the udp packet will have a ttl of 0 but since the destination address will match the destination in the udp packet, no time exceeded message will be sent. Instead, the traffic will be forwarded to udp port 33434. There is no service running on port 33434 so the router will send an icmp port unreachable message to the Laptop. From this message, trace will come to know that the destination has been reached and all the hops and associated times will be seen.

So why does trace to a router drops some packets on final hop?

In the above section we have seen how *nix (including) traceroute works. In this section we will see why trace to a router drops some packets on the final hop. By default, a Cisco router rate limits icmp unreachable messages. This is a security feature which has been put into place to reduce the unnecessary unreachable replies during a DOS or DDOS attack. Due to this feature, the router will only be able to reply with some unreachable messages as the few other messages are dropped due to rate limit. Due to this behavior you will see that the trace to a router will show a time in the first entry, drop in the next and time in the third.

This is an example of a trace from R1 to R3 via R2.

R1#traceroute 1.1.23.3 nu

Type escape sequence to abort.
Tracing the route to 1.1.23.3

1 1.1.12.2 8 msec 36 msec 28 msec
2 1.1.23.3 8 msec * 24 msec

As can be seen, there is a drop in the last hop. By default, a router allows only 1 icmp unreachable message per 500ms. As the time comes for R3 to reply to the second trace packet, it says that one unreachable packet has been already allowed in the first 500ms and so the second will not be allowed. By the time the third packet comes, we are into the next 500ms slot and thus the unreachable is allowed.

The way we can allow the Router R3 to reply with unreachables for all trace packets is by decreasing the time for which 1 unreachable packet is allowed. As I said, by default, 1 unreachable is allowed per 500ms. We will configure this time as a low value so that by the time the second packet comes, the first rate-limit slot will be over. We will configure the time as 10 ms and you can experiment with this value.

R3(config)#ip icmp rate-limit unreachable 10

After configuring the above command, we check our trace once again.

R1#traceroute 1.1.23.3 nu

Type escape sequence to abort.
Tracing the route to 1.1.23.3

1 1.1.12.2 16 msec 20 msec 12 msec
2 1.1.23.3 8 msec 36 msec 36 msec

As can be seen above, the trace packet is not heavily rate limited and we get the unreachable message to R1 which generates the proper last hop in the trace without the *.

I hope my post has been helpful in your life but the only guide which can help you in the hereafter is the Qur’an. You can download the English translation of the Qur’an here.