Emil Pearce published Novu’s DNS postmortem on 24 March 2025. The cutover was the day before. From 22:00 to 22:55 UTC, name resolution failed on and off for every user.
They were moving authoritative DNS from Amazon Route 53 to Cloudflare. The post does not say how many records moved.
The Route 53 → Cloudflare DNS playbook is the zone copy, the delegation, and the records Cloudflare will not store the same way. Novu’s lesson is what still breaks after that copy looks done.
What they did before the window
A week out, they lowered TTLs on the Route 53 records, including the NS record itself. They copied the records so the Cloudflare zone matched. They pointed the registrar at Cloudflare’s nameservers, finished the checks, and flushed caches through Google Public DNS and 1.1.1.1.
They also picked the hour with the least traffic. During the 55 minutes they posted updates on novustatus.com.
What the resolvers did anyway
Pearce’s root cause is cache timing. Resolvers do not refresh NS records on one clock.
Two things showed up. Some providers kept serving the old delegation. Others briefly served a negative cache. Some ISPs held the old NS records past the TTL Novu had set. Some cached a DNS_PROBE_FINISHED_DOMAIN error, which is the string the post prints.
A negative cache is normal DNS. A resolver that gets a temporary miss stores the miss, so it does not hammer the authoritative server. During this move, that stored miss kept Novu down for those users until the negative TTL expired. Flushing two public resolvers did not clear the ones Novu does not operate.
Why they moved
The reason on the page is performance. Cloudflare’s edge, faster DNS answers, faster SSL termination, and Argo Smart Routing. Pearce frames it as one piece of a longer pass on speed, security, and resilience. No before-and-after latency number is in the postmortem.
The prep list is the one I would steal, and then I would add a check the post shows they still needed. From a network you do not control, resolve the NS set after the registrar change and keep doing it until the old nameservers are gone. A flush of 8.8.8.8 and 1.1.1.1 is not that check.
Schedule the window when a 55-minute partial outage is survivable. They did, and they still had one. The playbook gets the records across. It does not make every recursive resolver drop a cached miss.