In May 2003 a single time server at the University of Wisconsin–Madison groaned under a load it should never have seen: hundreds of thousands of Netgear routers around the world were asking it for the time every second. The cause was not an attack but a simple programming mistake – a hardcoded IP address. It is a textbook example of how a harmless feature can turn into an accidental distributed overload.
The case was documented by network engineer Dave Plonka of UW–Madison. His 21 August 2003 write-up remains a lesson in what happens when huge numbers of cheap devices share the same default.
First: what does NTP do?
So that computers, routers and smart-home gadgets keep the correct time, they sync their clock against internet time servers using the Network Time Protocol (NTP) over UDP port 123. Simple devices use the stripped-down SNTP (Simple NTP) variant. That is perfectly normal – as long as a device behaves politely: it queries a server reachable by hostname (ideally a whole pool) only every few minutes to hours, and after an error it waits patiently instead of immediately knocking again.
The flaw: an IP address baked into the code
The affected Netgear routers (including the RP614 and RP614v2, MR814, DG814, HR314 and RO318) got three things wrong at once. Their built-in SNTP client:
- had a fixed server IP in the code: every device stubbornly queried
128.105.39.11– the exact serverntp1.cs.wisc.eduat UW–Madison, with no hostname and no fallback; - used a fixed source port: UDP port
23457– which later made the devices easy to identify but did nothing to shrink the problem; - polled far too aggressively: once every second until a reply finally arrived, only then switching to longer intervals (from one minute to 24 hours depending on model).
As long as the server answered, the devices calmed down. When it did not – for instance because it was overloaded – hundreds of thousands of routers kept hammering every second in unison. A vicious circle that fed itself.
The flood
On 14 May 2003, around 8 a.m., traffic exploded. Instead of the usual roughly 40,000 packets per second for the whole campus, at times more than 250,000 packets per second and over 150 megabit/s of SNTP requests alone poured in. Netgear later put the number of affected units shipped at 707,147; on some days the university counted over 500,000 unique clients – likely an undercount because of NAT. Had every device queried at the same moment, up to 700,000 packets/s (426 Mbit/s) would have been possible in theory.
Detective work with strings
At first it was unclear where the 76-byte UDP packets came from. The constant source port 23457 stood out. Via peer universities whose networks hosted suspect addresses, the senders were finally identified as Netgear MR814 routers. The firmware itself supplied the proof: with the simple command-line tool strings, Plonka pulled readable text out of the firmware files – and there, in black and white, was the hardcoded address 128.105.39.11.
How the case was resolved
Reaching Netgear was slow: the first email in mid-June 2003 drew only an automated reply, 23 days later. The breakthrough came by phone – a support director confirmed the fault – after which a 15-member review team of Netgear staff, university people and independent experts formed. The fixed firmware changed the behaviour fundamentally:
- queries only once a DNS server is configured – no more fixed IP;
- resolving
time-a.netgear.comandtime-b.netgear.comevery 10 minutes, i.e. via Netgear’s own infrastructure; - a time server is contacted only after name resolution succeeds;
- retries are 10 minutes apart, and after 5 attempts the device gives up.
In parallel the university first blocked the requests at the network edge and evaluated bigger fixes (such as a BGP-based anycast time service). Netgear and UW–Madison also negotiated a settlement, kept confidential.
Why it was not a one-off
Plonka’s report names a parallel case: in Australia about 85,000 SMC routers flooded the time server of the research body CSIRO, polling twice a minute. The pattern repeated: in 2006, hardcoded addresses in D-Link firmware knocked out, among others, a Danish operator’s time server – again followed by a settlement.
The internet community responded on two levels. As a standard: the RFC 4085 co-authored by Plonka (2005) carries the telling title „Embedding Globally-Routable Internet Addresses Considered Harmful“ – baking fixed public IP addresses into devices is officially considered harmful ever since. And in practice: the NTP Pool project offers manufacturers their own „vendor zones“ – a rotating set of many servers a device reaches by hostname, instead of besieging a single machine.
What vendors – and you – should take away
- Never burn a single third-party IP address into a device. Use hostnames and a server pool.
- Poll politely: long intervals, randomised delay (jitter) and real backoff after errors – never keep hammering every second.
- Anyone shipping many devices should run their own time infrastructure or use an NTP Pool vendor zone – with the operator’s permission.
- For users: keep your router firmware up to date. Exactly these kinds of flaws get fixed via updates.
The case is over 20 years old – and more relevant than ever. In an era of billions of cheap IoT devices, a single careless default in a firmware image can be enough to unintentionally bring a complete stranger’s server to its knees. Netgear’s routers were never malicious – they simply did, precisely and a million times over, exactly what someone had programmed them to do.
Source: Dave Plonka, „Flawed Routers Flood University of Wisconsin Internet Time Server“ (UW–Madison, 2003) · RFC 4085 · NTP Pool – Information for Vendors.