Spreedly v1.73 iFrame Errors
Incident Report for Spreedly
Postmortem

On March 16, 2022, for approximately five hours, 0.11% of requests for the Spreedly iFrame Payment Form or one of its components failed with the HTTP error code 404 (File Not Found). Affected cardholders would have been unable to provide their card information unless they reloaded the page.

What Happened

Some web servers for the Spreedly iFrame Payment Form and related components were automatically redeployed as part of normal cloud instance lifecycle events. Automated systems created replacement servers from a base image and then updated the servers to include the latest version of iFrame. Then, automated tests of the replacement servers indicated there was an issue with this version, and the automated deploy system rolled back to the previously known-good version. However, this known-good version was the one deployed as part of the base image, and therefore out of date. Therefore, these web servers no longer contained iFrame 1.73. The remaining majority of the web servers, which did have the correct content, continued to successfully serve iFrame 1.73. In addition, the vast majority of requests were served from the caches of Spreedly’s global content distribution network.

Next Steps

As part of an already-planned architectural change, Spreedly has completed the migration of the Spreedly iFrame Payment Form and its constituent components to a simpler, static, web host, which no longer uses this deployment mechanism.

For systems which still use the automated deployment mechanism, Spreedly will accelerate and automate promotion of base images, to prevent recurrences of this issue from affecting other systems.

Posted Mar 23, 2022 - 15:53 EDT

Resolved
The underlying issue resulting in 404 error responses when serving iFrame v1.73 has been determined, a fixed deployed, and is working as expected. A post incident review will be published. We apologize for any inconvenience and disruption to service.
Posted Mar 16, 2022 - 07:33 EDT
Monitoring
A fix has been implemented and we've tested and validate that the service is responding as expected. We will continue to monitor the situation.
Posted Mar 16, 2022 - 06:53 EDT
Identified
The issue has been identified and our Engineers are actively working on a resolution; however, we do not currently have a clear time to resolution that we can communicate. As soon as we do, or we have a fix in place, we will provide an immediate update. Additional details on root cause analysis will be share in a future post-mortem update once the issue is resolved and stable.
Posted Mar 16, 2022 - 06:40 EDT
Investigating
We are currently investigating an issue with the Spreedly iFrame v1.73 returning 404 errors. Updates will be provided as they become available.
Posted Mar 16, 2022 - 05:39 EDT
This incident affected: Core Transactional API.