November 11th, 2021 — iFrame Errors

On November 11th at 23:05 UTC a new version of Spreedly’s iFrame was deployed to production with a misconfiguration which prevented transactions from completing successfully. When the misconfiguration was identified, the deployment was rolled back on November 12th at 01:51 UTC and iFrame service was fully restored for all customers by 02:04 UTC.

What Happened

Spreedly iFrame deployment processes involve both manual and automated portions, and the manual process was not clearly documented. This confusion led to non-production values being promoted to the production environment. As customer cache validity values expired at Spreedly’s CDN, customers began to receive the problematic code. Over the course of two hours, a number of customers received the problematic code and experienced failures in processing based on the iFrame process. Once Spreedly Engineers were engaged, the issue was identified, the version rolled back, and all customer integrations were restored over a 30 minute period.

Next Steps

Internal documentation was clarified and expanded, automation was expanded to reduce reliance on manual steps, and additional tests were added to the CI/CD pipeline in order to ensure that only valid production values were configured for production environments.