Failures with the new NTP checking

[mutemule]

The NTP checking introduced 5.3.0 itself works fairly well, but get_ntp_info in checks/check_status.py is broken in some configurations.

The NTP checking pulls its configuration, including peers/servers to check against, from a configuration file. But get_ntp_info checks NTP offset against a hard-coded pool.ntp.org. It's not uncommon for datacenters to only allow a small number of servers to poll Internet-based NTP sources, and then to use those servers to re-distribute time internal to the local networks. In such a configuration, the ntplib.NTPClient().request call will always fail.

[LeoCavaille]

Thanks @mutemule for reporting this, we probably need to have better error handling and:

• to allow overriding this default pool.ntp.org (which will be changed soon by a datadog subdomain of the pool cc @remh)
• look at the NTP check settings if there are some?

[mutemule]

Thanks, @LeoCavaille!

I took a quick look at fixing this before I submitted the issue, but I'm not familiar enough with the DD internals. It seemed reasonable to me to use the check settings if they exist, so there's consistency in the upstream queries, and just set defaults in there.