Those who have read my article on Non Technical tips on CCIE Troubleshooting and who are following my Technical tips on CCIE Troubleshooting would be wondering why I have chosen to write an article on NTP after writing a post on Troubleshooting MPLS VPN. NTP is trivial as compared to MPLS VPN, so why not the new post on BGP or OSPF? The message I want to send home is that no topic is simple in CCIE. I thought NTP is easy and did not practice it and during Lab, this simple topic caused me to go blank because without practice, there is no strategy and without strategy you don’t know what to do. In this article we will see few scenarios with regards to troubleshooting NTP in a Cisco network.

The Setup

The setup is pretty straight forward with 6 routers having IGP running between each other. The ip address is as given in the diagram where X is the router number. Also, the loopback ip address is as given.

There are 5 independent requirements that have to be met.

Requirement 1

R5 should get authenticated time from R1.

Troubleshooting

Although I am not in favor of using show runs to troubleshoot an issue, I will still use it to highlight the disadvantage of using it (yet some times it is the only thing which helps you troubleshoot). Lets see a show run of R1 with regards to NTP.

R1#sh run | i ntp
ntp authentication-key 1 md5 045802150C2E 7
ntp master 5

Let us see the same for R5

ntp authentication-key 1 md5 14141B180F0B 7
ntp authenticate
ntp trusted-key 1
ntp clock-period 17180092
ntp source Loopback100
ntp server 1.1.1.1 key 1

The first things which we see is the hashed passwords and that both the hashed passwords are different. Now how many of us know that one password can be hashed in many ways by the router. A password of cisco can be represented as 045802150C2E, 14141B180F0B, 13061E010803, 05080F1C2243, 00071A150754, etc and many more ways. So we jump to conclusion that the password is wrong and change the password but is the password really wrong? Let us see what the output of show ntp association detail has to say

R5#sh ntp ass de
1.1.1.1 configured, insane, invalid, unsynced, stratum 16
ref ID 0.0.0.0, time 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
our mode client, peer mode unspec, our poll intvl 64, peer poll intvl 64
—output truncated—-

From this output we come to know that NTP is configured but it did not sync with R1 for time. There are 2 possible reasons for this. One is that NTP is not able to communicate with R1 1.1.1.1 and the other is authentication failure. Let us see whether we can get anything from debugs. We have started debugs for NTP authentication, validity and packets. You will have to be patient here because everything in NTP takes time so will the debug.

R5#
Dec 4 12:08:16.727: NTP: xmit packet to 1.1.1.1:
Dec 4 12:08:16.731: leap 3, mode 3, version 3, stratum 0, ppoll 64
Dec 4 12:08:16.731: rtdel 0000 (0.000), rtdsp 10001 (1000.015), refid 00000000 (0.0.0.0)
Dec 4 12:08:16.731: ref 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
Dec 4 12:08:16.731: org 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
Dec 4 12:08:16.731: rec 00000000.00000000 (00:00:00.000 UTC Mon Jan 1 1900)
Dec 4 12:08:16.731: xmt D64999B0.BA450743 (12:08:16.727 UTC Wed Dec 4 2013)
Dec 4 12:08:16.735: Authentication key 1

From this debug we see that NTP is transmitting packet but it is not receiving any. So there has to be some place where NTP is being blocked. After checking the routing table, the first upstream device towards 1.1.1.1 is R3. Let us see the ACL on R3.

R3#sh ip access-lists
Extended IP access list 100
10 permit ip 1.1.0.0 0.0.255.255 host 1.1.1.1 (10 matches)
20 permit ip host 1.1.1.1 1.1.0.0 0.0.255.255 (6 matches)
Extended IP access list 123
10 deny udp any eq ntp any eq ntp (234 matches)
20 permit ip any any (564 matches)

NTP uses udp port 123 and it is blocked on ACL 123, so we must modify the ACL to accomodate ip address of R5. There are 2 options here. First is to remove the statement 10 but I am personally against removing statements in ACL unless it is absolutely necessary. Second is to make a specific entry for our requirement above statement 10. (Note: There can be cases where an ACL entry that seems harmful might be just a distraction. So there could be an ACL entry appearing to block NTP but the ACL is not applied anywhere OR the ACL is applied to an interface irrelevant to us. In our case we can be certain that ACL entry is blocking because we see hits on it)

R3(config)#ip access-list extended 123
R3(config-ext-nacl)#5 permit udp any host 1.1.1.1 eq 123

Now the debug on R5 is showing a received packet

Dec 4 12:39:12.807: NTP: rcv packet from 1.1.1.1 to 1.1.5.1 on Loopback100:
Dec 4 12:39:12.807: leap 0, mode 4, version 3, stratum 5, ppoll 64
Dec 4 12:39:12.811: rtdel 0000 (0.000), rtdsp 0002 (0.031), refid 7F7F0701 (127.127.7.1)
Dec 4 12:39:12.811: ref D649A0C2.79C091C4 (12:38:26.475 UTC Wed Dec 4 2013)
Dec 4 12:39:12.815: org D649A0F0.C07117B5 (12:39:12.751 UTC Wed Dec 4 2013)
Dec 4 12:39:12.815: rec D649A0F0.B21B4E27 (12:39:12.695 UTC Wed Dec 4 2013)
Dec 4 12:39:12.819: xmt D649A0F0.B31E7C1A (12:39:12.699 UTC Wed Dec 4 2013)
Dec 4 12:39:12.819: inp D649A0F0.CDB67FCF (12:39:12.803 UTC Wed Dec 4 2013)
Dec 4 12:39:12.819: Authentication key 1

Let us see the status now

R5#sh ntp ass de
1.1.1.1 configured, authenticated, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time D649A102.79BF1870 (12:39:30.475 UTC Wed Dec 4 2013)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
—output truncated—–

So our time is synced and it was NOT a case of wrong password.

Requirement 2

R6 should get authenticated time from R1.

Troubleshooting

Let us see the show runs to get an idea of what is present in R1 and R6 regarding NTP.

R1#sh run | i ntp
ntp authentication-key 1 md5 045802150C2E 7
ntp master 5

R6#sh run | i ntp
ntp authentication-key 1 md5 070B285F4D06 7
ntp authenticate
ntp trusted-key 1
ntp server 1.1.1.1 key 1

From the previous requirement we got an indication that the hash value is not a reliable way of isolating passwords mismatch. Also, the configuration looks much similar to the previous requirement. So let us start how we started the troubleshooting for the previous requirement. Lets run the 3 ntp debugs

The output of the Debug ntp validity gives us a very important clue

*Dec 4 13:24:22.783: NTP: packet from 1.1.1.1 failed validity tests 10
*Dec 4 13:24:22.783: Authentication failed

So we must rectify the password. In this post, we are syncing 5 routers with an NTP server so it is not sensible to change the server password. We will change the client password but how do we know what is the password? We can do this by decrypting the hash using the router itself. Lets do this on R1

R1(config)#key chain DECRYPT
R1(config-keychain)#key 1
R1(config-keychain-key)#key-string 7 045802150C2E        (This hash value has to be taken from ntp authentication command on R1)

To view the password, use the following command

R1#sh key chain DECRYPT
Key-chain DECRYPT:
key 1 — text “cisco”
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]

So now we know that the password is cisco, let us change R6 accordingly

R6(config)#ntp authentication-key 1 md5 cisco

R6#sh ntp ass de
1.1.1.1 configured, authenticated, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time D649AEC2.79B15525 (13:38:10.475 UTC Wed Dec 4 2013)
our mode client, peer mode server, our poll intvl 64, peer poll intvl 64
—output truncated—

So our time is synced and it was a case of wrong password.

(Note1: We have unlimited time now due to which we decrypted the hash back to plain text password but this is not the appropriate way to resolve authentication issue where time is limited. If you find such an issue and you don’t have time, you must remove the keys from both the server and client and put your own common password on both the routers.

ntp authentication-key 1 md5 whateverpassword

Note2: There was a distraction in R4 (if you are in the habit of seeing show runs)

R4#sh ip access-lists
Extended IP access list 123
10 deny udp host 1.1.6.1 host 1.1.1.1 eq ntp
20 permit ip any any (1411 matches)

Seeing this you could have easily mistaken the issue to be of ACL rather than of password mismatch. If you are observant enough, you will know that normally ntp packets have source ip address of outgoing interface, so the ntp packet will have a source ip address of 1.1.46.6 and not 1.1.6.1 which is present in the ACL

Note3: From our experience with Type 7 passwords we know that when we copy and paste user accounts from one router to the other, the hash of passwords have no problem with copy paste. The same is not observed to be true with NTP. If we copy the hash of R1 and paste it in R6 like ntp authentication-key 1 md5 045802150C2E 7, you will still get authentication failure.)

Requirement 3

R3 should get authenticated time from R1.

Troubleshooting

Let us see the show runs to get an idea of what is present in R1 and R3 regarding NTP.

R1#sh run | i ntp
ntp authentication-key 1 md5 045802150C2E 7
ntp master 5

R3#sh run | i ntp
ntp authentication-key 1 md5 00071A150754 7
ntp authenticate
ntp server 1.1.1.1 key 1

From the show run it is obvious that there are few commands less on R3 as compared to other routers but let us not rely on it. We will start with our usual debugs

*Dec 4 14:43:40.075: NTP: packet from 1.1.1.1 failed validity tests 10
*Dec 4 14:43:40.075: Authentication failed

From the excerpt of the debug, we see that there is authentication failure. Let us put the new authentication command and see

R3(config)#ntp authentication-key 1 md5 cisco

*Dec 4 14:56:28.082: NTP: packet from 1.1.1.1 failed validity tests 10
*Dec 4 14:56:28.082: Authentication failed

Authentication failure again!!!! It must be something else. Let us use the show commands

R3#sh ntp ass de
1.1.1.1 configured, authenticated, insane, invalid, unsynced, stratum 16
—Output Truncated—–

According to the debug “Authentication failed” but according to sh ntp association details command authentication passed, this case only happens when the ntp trusted-key 1 command is missing. Let us put the command and see whether ntp syncs time or not.

R3(config)#ntp trusted-key 1

R3#sh ntp ass de
1.1.1.1 configured, authenticated, our_master, sane, valid, stratum 5
—Output Truncated—–

So our time is synced and it was a case of missing commands.

Requirement 4

R4 should get authenticated time from R1.

Troubleshooting

Let us see the show runs to get an idea of what is present in R1 and R4 regarding NTP.

R1#sh run | i ntp
ntp authentication-key 1 md5 045802150C2E 7
ntp master 5

R4#sh run | i ntp
ntp authentication-key 1 md5 05080F1C2243 7
ntp authenticate
ntp trusted-key 1
ntp server 1.1.1.1 key 2

Like always, we will begin with the ntp debugs

*Dec 4 15:19:43.418: NTP: packet from 1.1.1.1 failed validity tests 10
*Dec 4 15:19:43.418: Authentication failed

Again a case of authentication failure!!!!!!!!! Let us set new ntp password command and see

R4(config)#ntp authentication-key 1 md5 cisco

Dec 4 16:11:01.693: NTP: packet from 1.1.1.1 failed validity tests 10
Dec 4 16:11:01.693: Authentication failed

Again password failure!!!!!!! Now even the ntp trusted-key 1 is present. The only way now to troubleshoot this issue without show run is via packet capture. Since we cannot run packet capture every where, let us observe the show runs. In the R4 config, ntp is using cisco as key 1 and we have trusted the key 1 to be used by ntp but we have set key 2 in ntp server command while key 2 is non existent. So if you were to run a packet capture, you would see that ntp requests are going to R1 without any key from R4 and that is the reason authentication is failing. Let us rectify it and see whether the time syncs

ntp server 1.1.1.1 key 1

R4#sh ntp ass de
1.1.1.1 configured, authenticated, our_master, sane, valid, stratum 5                                   —-output truncated—

Finally the time on R4 has synced with R1

Requirement 5

R2 should get authenticated time from R1.

Troubleshooting

Let us see the show runs to get an idea of what is present in R1 and R2 regarding NTP.

R1#sh run | i ntp
ntp authentication-key 1 md5 045802150C2E 7
ntp master 5

R2#sh run | i ntp
ntp authentication-key 1 md5 13061E010803 7
ntp authenticate
ntp trusted-key 1
ntp clock-period 17179883
ntp server 1.1.1.1

Let us see the current status of the time

R2#sh ntp ass de
1.1.1.1 configured, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time D649D542.79829398 (16:22:26.474 UTC Wed Dec 4 2013)          —-output truncated—-

The time seems to be synced with R1 but our requirement is not completely met. We want authenticated time from R1. We must observe the show run here as to why R2 synced time without authentication. The key command here is ntp server 1.1.1.1. R2 is not asking for authenticated time so it will not be given that. Let us change the config so that R2 asks for authenticated time.

R2(config)#ntp server 1.1.1.1 key 1

Now let us check whether we are getting authenticated time or not

R2#sh ntp ass de
1.1.1.1 configured, authenticated, our_master, sane, valid, stratum 5
ref ID 127.127.7.1, time D649D642.798169CE (16:26:42.474 UTC Wed Dec 4 2013)         —-output truncated—-

So our requirement is met.

I hope my post has been helpful in your life but the only guide which can help you in the hereafter is the Qur’an. You can download the English translation of the Qur’an here.