Checkout V2: All Transactions Marked Successful
Incident Report for Spreedly
Postmortem

Between 13:50UTC April 30 and 16:31UTC May 1 code was deployed that caused all transactions at the Checkout.com gateway to report success regardless of actual results.

What Happened

At approximately 13:50UTC on April 30th an update was applied to the production system containing API version upgrade changes to the Checkout.com gateway. This update had insufficient logic to determine whether certain transactions failed, resulting primarily in authorizations which failed at the gateway, but that Spreedly indicated were successful. Followup transactions based on these erroneous authorizations also failed and were continually retried. Code fixing this update was applied 16:31UTC on May 1st for a total duration of 26.5 hours.

The Checkout.com API changes introduced a particularly tricky error case that we did not account for before deploying the code. Tests did not account for the combination of factors that led to this error and were not available to help diagnose the issue as it was happening. In addition, our monitoring processes did not report this situation as an issue due subtlety of its expression and we were only aware of the problem when reported by customers.

Next Steps

We will evaluate the following activities to prevent this situation from happening in future: 1) perform more in depth review of API upgrade documentation to ensure that the impact of ActiveMerchant changes are sufficiently understood, 2) construct and deploy more robust tests to account for both success and failure detection, and 3) implement additional monitoring processes to detect this subtle type of issue rather than relying on customer reporting.

Conclusion

We apologize for any disruption this incident may have caused. You rely on Spreedly to report transaction status accurately and we will take steps to ensure you can conduct your business confidently.

Posted 4 months ago. May 09, 2019 - 13:06 EDT

Resolved
Since deploying and monitoring the aforementioned changes, we have seen transactions properly marked as succeeded and failed. A post-mortem will be published within three business days. Please reach out to support@spreedly.com with any additional questions.
Posted 5 months ago. May 01, 2019 - 14:52 EDT
Monitoring
Changes to resolve the identified issue are now in production and we expect transactions to be properly marked as succeeded or failed, based on the response from the gateway. We are monitoring to ensure the fix has resolved the issue completely.
Posted 5 months ago. May 01, 2019 - 12:42 EDT
Identified
On Tuesday April 30 at approximately 15:56 UTC, Spreedly introduced changes to upgrade the Checkout.com gateway to V2. Since then, we have identified an issue that marks all transactions as successful, regardless of the state of the transaction at the gateway. Checkout.com has identified the issue and is working on a fix. When the fix has been deployed and verified we will post an update
Posted 5 months ago. May 01, 2019 - 11:31 EDT
This incident affected: Core Transactional API.