Jump to content

NTP

From Wikitech
This page may be outdated or contain incorrect details. Please update it if you can.

All servers are time-synchronised using the NTP protocol. In pmtpa, this is managed by Puppet, with the ntp::client class in ntp.pp. The NTP servers are dobson and linne.

Testing

Testing is very important, as the November 2005 event demonstrated. It's easy to test whether ntpd is working.

/usr/sbin/ntpq -p

Here is the output from a happy server:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*vl-2-0.csw1-pmt albert.pmtpa.wm  3 u   43   64  177    0.486    1.155   1.135

Note the asterisk in the first column, that tells you it's happy. It's synchronised to csw1, which is on stratum (st) 3, and the refid gives the stratum 2 server. The other important columns are:

  • when: this tells you how long ago it received a response from the server, in this case 43 seconds
  • offset: this tells you how far off the clock is, in milliseconds.

Here is the output from a server which is on its way to synchronisation:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 vl-2-0.csw1-pmt albert.pmtpa.wm  3 u   81 1024    7    0.573  -203.77   1.635

There's no asterisk, which means it hasn't synchronised yet. The offset is substantial, so it will take a while to get into sync. The fact that the remote, refid, st and when columns are reasonable tells you that it is actually working. Hopefully we check back later, offset should be smaller.

Here is the output from a completely broken server:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 vl-2-0.csw1-pmt .B▒▒.          16 u   16   64    0    0.000    0.000 4000.00

It seems to know what server it's meant to be reading from, but the other columns are just silly. There's no such thing as stratum 16, and I'm quite sure the network delay is meant to be more than zero. If you see something like this, you need to fix it.

ntpq can be run remotely. The output of ntpq -c peers csw1-pmtpa currently shows:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 207.142.131.255 0.0.0.0         16 u    -   64    0    0.000    0.000 16000.0
 10.0.255.255    0.0.0.0         16 u    -   64    0    0.000    0.000 16000.0
 clock2.redhat.c .CDMA.           1 -  18d   64    0   73.200    1.322 16000.0
 ntp-s1.cise.ufl 85.83.78.79     16 -  18d 1024    0   18.140    0.673 16000.0
 raptor.tera-byt 0.0.0.0         16 -    - 1024    0    0.000    0.000 16000.0
*albert.pmtpa.wm ntp-s1.cise.ufl  2 u   45   64  377    0.790   -3.597   0.400

Three broken external servers, two broadcast domains and albert, which is a working stratum 2 server. Finally, albert gives:

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*ntp-s1.cise.ufl .USNO.           1 u   86  128  377   22.090   12.095   5.867
+ip-207-145-113- .GPS.            1 u   89  128  377   76.639   20.227   5.468
+solarnet.ru     hora.cs.tu-berl  2 u   92  128  377  180.877   11.987   3.481
-blah.jabber.dk  ntp2.sth.netnod  2 u   97  128  377  135.076   22.336   5.524
 LOCAL(0)        LOCAL(0)        10 l   12   64  377    0.000    0.000   0.001

A nearby stratum 1 server at the University of Florida is selected as the reference, but I've configured three other servers from pool.ntp.org in case that one goes down. Two of them are contributing to the averaging process, the third is ignored because its clock doesn't agree with the others. If all 4 are unreachable, the local clock will be used. It's currently undesirable because it's been declared stratum 10.

Monitoring

Service Restart Check

A failure of this check indicates that the configuration file for ntp (ntp.conf) was changed but the service itself was not restarted. A CRITICAL alert is raised if the time delta between the configuration file change and service restart exceeds two (2) hours.

This check is meant to be a warning alert and does not signify an error in the service but should be corrected immediately by restarting ntpd.service. If you are unsure about the changes, please contact the Traffic team.

You can do so by using the cookbook: sre.dns.ntp-roll-restart. Simply run it by passing the relevant query/alias (such as --alias dnsbox) and it will automatically restart the service with sane defaults.