How and why the leap second affected Cloudflare DNS
The root cause of the bug that affected our DNS service was the belief that time cannot go backwards. In our case, some code assumed that the difference between two times would always be, at worst, zero. RRDNS is written in Go and uses Go's time.Now() function to get the time. Unfortunately, this function does not guarantee monotonicity. Go currently doesn't offer a monotonic time source.
So the clock went "backwards", s1 - s2 returned < 0, and the code couldn't handle it (because it's a little known and infrequent failure case). Part of the root cause here is cultural -- Google has solved the leap-second problem internally through leap smearing, and Go seems to be fundamentally a Google product at heart. The easiest fix in general in the "outside world" is to use "ntpd -x" to do a form of smearing. It looks like AWS are leap smearing internally (https://aws.amazon.com/blogs/aws/look-before-you-leap-the-coming-leap-second-and-aws/), but it is a shame they aren't making this a standard part of services running on top of AWS and a feature of the AWS NTP fleet.ntp time leap-seconds fail cloudflare rrdns go golang dns leap-smearing ntpd aws
-
via twitter: "interesting conversation between author of a parenting book and the guy who introduced the concept of "flow"" -- summary, family life is interrupt-driven (via nagging) and fundamentally hard to align with "flow"
-
wow
dick-cheney pacemakers iot internetofshit wireless security via:jzdziarski
#
We've seen a bunch of issues with ntp servers around the world (and many gps devices get it horribly wrong). It's probably best to use ntp servers from one of the leap smearers (e.g. time.google.com) as next tier for your own ntp servers. Partly that's on the assumption that if they're leap smearing they have at least thought about the problem.
Wait, Google offers public NTP service with leap smear support? That's excellent!