July 27, 2021 — Transaction failures

Spreedly released an update to the Stored Credentials framework that resulted in a partial outage of Spreedly’s transactional API of approximately 11 minutes. Credit, Void, and Capture transactions performed during the outage received a failure response, but may have been successfully processed. Spreedly engineers identified the adverse code deployment and reverted back to the previous version of the API, resolving the issue.

What Happened

At 16:59 UTC (12:59 PM EDT) on Tuesday, July 27, 2021, Spreedly released a new version of the Core Transactional API with feature enhancements to the Stored Credentials framework. At 17:04 UTC (01:04 PM EDT) Spreedly’s automated systems detected an elevated number of “500 Internal Server Error” responses returned from the Core Transactional API. Customers attempting to perform transactions received a failure response but the Gateway may have successfully processed the transactions. Incident response teams were alerted and mobilized. Spreedly engineers reverted to the previous version of the API at 17:10 UTC (01:10 PM EDT) and the system stabilized.

Next Steps

Spreedly is investigating integration testing improvements to prevent future regressions from feature enhancements.