On March 16, 2022, for approximately five hours, 0.11% of requests for the Spreedly iFrame Payment Form or one of its components failed with the HTTP error code 404 (File Not Found). Affected cardholders would have been unable to provide their card information unless they reloaded the page.

What Happened

Some web servers for the Spreedly iFrame Payment Form and related components were automatically redeployed as part of normal cloud instance lifecycle events. Automated systems created replacement servers from a base image and then updated the servers to include the latest version of iFrame. Then, automated tests of the replacement servers indicated there was an issue with this version, and the automated deploy system rolled back to the previously known-good version. However, this known-good version was the one deployed as part of the base image, and therefore out of date. Therefore, these web servers no longer contained iFrame 1.73. The remaining majority of the web servers, which did have the correct content, continued to successfully serve iFrame 1.73. In addition, the vast majority of requests were served from the caches of Spreedly’s global content distribution network.

Next Steps

As part of an already-planned architectural change, Spreedly has completed the migration of the Spreedly iFrame Payment Form and its constituent components to a simpler, static, web host, which no longer uses this deployment mechanism.

For systems which still use the automated deployment mechanism, Spreedly will accelerate and automate promotion of base images, to prevent recurrences of this issue from affecting other systems.