iFrame Errors
Incident Report for Spreedly
Postmortem

November 11th, 2021 — iFrame Errors

On November 11th at 23:05 UTC a new version of Spreedly’s iFrame was deployed to production with a misconfiguration which prevented transactions from completing successfully. When the misconfiguration was identified, the deployment was rolled back on November 12th at 01:51 UTC and iFrame service was fully restored for all customers by 02:04 UTC.

What Happened

Spreedly iFrame deployment processes involve both manual and automated portions, and the manual process was not clearly documented. This confusion led to non-production values being promoted to the production environment. As customer cache validity values expired at Spreedly’s CDN, customers began to receive the problematic code. Over the course of two hours, a number of customers received the problematic code and experienced failures in processing based on the iFrame process. Once Spreedly Engineers were engaged, the issue was identified, the version rolled back, and all customer integrations were restored over a 30 minute period.

Next Steps

Internal documentation was clarified and expanded, automation was expanded to reduce reliance on manual steps, and additional tests were added to the CI/CD pipeline in order to ensure that only valid production values were configured for production environments.

Posted Nov 19, 2021 - 17:07 EST

Resolved
iFrame integrations experienced errors from approximately 2021-11-11 11:05 UTC to 2021-11-12 01:58 UTC.

A fix has been deployed and all systems appear to be stabilized and functioning. The incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.

We apologize for any inconvenience and disruption to service.
Posted Nov 11, 2021 - 21:27 EST
This incident affected: Core Transactional API.