A Brief History of Time

Time is one of those things we take for granted. You glance at your phone, it says 14:32, and 14:32 is the time. Everyone agrees. Job done.

Except none of that is actually true. The time on your phone is a convenient fiction - an agreement we've all quietly signed up to because the alternatives are worse. And that fiction is held together by an increasingly delicate stack of astronomy, international treaties, and a handful of engineers who would really rather you didn't think about leap seconds (yes, they are a thing).

This is the story of how we got here, why it matters to anyone running infrastructure, and what's changing over the next decade.

When every town had its own time

For most of human history, time was a local affair. Noon was when the sun was directly overhead - and because the Earth is a sphere spinning through space, that moment happens at a different instant depending on where you're standing.

The Earth rotates 360 degrees in roughly 24 hours, which works out to 15 degrees per hour. In the UK, Cardiff sits about 3.2 degrees west of Greenwich [1]. So when it's solar noon in Greenwich, the sun hasn't quite reached its highest point over Cardiff yet - it's still about 13 minutes away. Bristol, slightly closer in, runs about 10 minutes behind Greenwich on solar time.

Reading a sundial in a particular location gives you "apparent solar time", which is where the sun is in the sky at that moment; however, the sun isn't a steady timekeeper because the Earth's orbit is elliptical and its axis is tilted, the sun runs ahead of a uniform clock at some points of the year and lags behind at others - by as much as 14-16 minutes.

What a well-regulated mechanical clock shows is "mean solar time", a smoothed-out fiction based on a hypothetical sun that moves at a constant rate. Reconciling the two requires a correction called the equation of time, which astronomers tabulated as adjustments to apparent solar time [2].

Until the 19th century, that's how towns worked. Cardiff Mean Time was 13 minutes behind Greenwich Mean Time. Every town or village had its own reckoning, set by sundial (corrected via the equation of time). If you lived your whole life in one place, it didn't matter. Noon was noon, and everyone got on with their day.

Enter the railways

As railways connected communities, this local time discrepancy began to cause scheduling and safety issues. Collisions became a real risk on single-track lines where two trains were supposed to pass at a scheduled point and a scheduled time [3].

In 1840 the Great Western Railway adopted Greenwich Mean Time across its entire network regardless of where the station was geographically. Other railways followed suit, and by 1847 the Railway Clearing House had formally standardised on Greenwich Mean Time across the whole railway network. It was called "railway time."

American railways adopted a system of four standard time zones in 1883, and the International Meridian Conference in 1884 extended the idea globally: the Earth was divided into 24 zones, each an hour wide, all measured relative to Greenwich [4]. Time had gone from a local thing to a global coordinate system, known as Time Zones.

Daylight saving

The next wrinkle was daylight saving time. Daylight Saving or DST is the practice of advancing clocks to (in theory) make better use of the longer daylight available during summer by having darkness fall at a later clock time [5].

Germany was the first country to introduce it nationally, in 1916, during the First World War as a way to conserve coal during wartime. The UK followed weeks later with the Summer Time Act 1916, and the US adopted it in 1918.

The twice-yearly clock changes are associated with spikes in heart attacks, road accidents, and workplace injuries in the days following the transition [6].

The second

Before 1967, the second was defined by the Earth's rotation. However, the Earth's rotational speed isn't entirely constant - it can vary due to the Moon's gravitational pull and shifts in the planet's outer layer. These fluctuations create problems when trying to keep time precisely.

In 1967 the international definition changed to being based on the oscillation of a caesium atom. A second became 9,192,631,770 periods of the radiation corresponding to a specific transition in caesium-133 (also known as SI Seconds) [7].

The leap second

Over the decades, atomic time (TAI, International Atomic Time) and solar time (UT1, based on Earth's actual rotation) drifted apart. The compromise, introduced in 1972, was Coordinated Universal Time (UTC). UTC ticks along at the atomic second, but every so often - when atomic time and solar time have drifted about 0.9 seconds apart - we insert a leap second. The last minute of that day has 61 seconds instead of 60. The clocks go 23:59:58, 23:59:59, 23:59:60, 00:00:00. Twenty-seven leap seconds have been added since 1972. The last one was on 31 December 2016 [8].

When the leap second broke the internet

For software, the assumption that a minute has 60 seconds is baked in very deep. Database timestamps, scheduler queues, replication logs, log file rotation, TLS certificate validation, high-frequency trading systems - all of them carry this assumption. Leap seconds are announced only about six months in advance, because they depend on measurements of the Earth's rotation that can't be predicted further out than that.

The most notorious incident was on 30 June 2012. When the leap second was inserted, a latent bug in the Linux kernel fired. The timekeeping subsystem failed to notify the high-resolution timer subsystem that a leap second had occurred, causing CLOCK_REALTIME hrtimers to fire 1 second early and sub-second timers to fire immediately in a loop. Any application doing a lot of sleep-based work - MySQL and Java applications were the worst hit - saw CPU usage spike to 100% across all cores [9].

Reddit, LinkedIn, Mozilla, Qantas's reservation system, and a long list of others went down or were badly degraded. Operators spent the next few hours restarting services and rebooting machines. I remember getting paged to some very unhealthy MySQL instances!

Leap Smearing

Since 2008, Google, instead of applying leap seconds to servers in clock steps, have "smeared" the extra second across the hours before and after each step [10].

During the smear, clocks run slightly slower than usual. Each second of time in the smeared timescale is about 11.6 μs longer than an SI second.

From the perspective of any Network Time Protocol (NTP) client reading from smear-capable time servers, nothing unusual happens. Time just moves a fraction of a percent slower for a day. Amazon, Meta, and Microsoft now run similar smearing schemes on their internal clock infrastructure. AWS uses smeared time via its Time Sync Service (NTP only), which is what most EC2 instances receive by default. However, newer PTP hardware clocks do not support smearing [11].

The leap second is being retired

At the General Conference on Weights and Measures, the world's metrology bodies voted to phase out leap seconds by 2035. After 2035, UTC will be allowed to drift further from solar time - the new tolerance hasn't been agreed yet, but the proposals on the table let the difference grow to something like a minute before any correction is applied, which gives roughly a century of headroom.

Hopefully, this simplifies the time for future engineers!

What this means for you

If you're running anything on AWS, GCP, or Azure, you're almost certainly already on smeared time, and the cloud providers do a very good job ensuring this crucial service works without an issue, but it's worth knowing that your clock is not the same as UTC in the day around a leap event.

If you have software that cares about sub-second precision across systems - trading, telemetry, distributed databases with tight clock-skew assumptions - know where your time comes from and know whether any of your dependencies see leap seconds as real.

Who thought time could be so complex?