The culprit was a software bug in DynamoDB, AWS' DNS management system. DNS or "Domain Name System" is often analogized as the phone book of the internet, translating domain names preferred by human users into the IP addresses required by automated systems. On the modern internet, many services, such as cloud computing or streaming, need to be able to assign the same domain to multiple IP addresses in a network, allowing efficient use of geographically distributed servers.
DNS Enactor, a component of DynamoDB responsible for updating these tables, "experienced unusually high delays needing to retry its update on several of the DNS endpoints." While Enactor played catch-up, DynamoDB continued generating new plans for it to follow, which another, on-time DNS Enactor helpfully tried to implement.
The fresh DNS configuration that resulted was overwritten by a significantly outdated one when the delayed Enactor finally caught up, bypassing a safeguard meant to prevent just such an error which was also experiencing delays. The second, on-time Enactor then deleted the outdated plan altogether, detecting that it was stale. This state of affairs resulted in knock on effects across AWS, requiring engineers to manually diagnose the problem and resolve it.