Increase in 500 Errors
Incident Report for Spreedly
Postmortem

Between 7:23 pm UTC and 9:08 pm UTC on September 25th, 2019, the Worldpay gateway experienced errors on refund and void transactions which were propagated back to Spreedly customers as 500 response codes.

What Happened

At 7:23 pm UTC, the Worldpay gateway began returning responses for Refund and Void transactions indicating “Temporary Failure, please Retry”. This response from Worldpay was returned to Spreedly in a format unexpected by our systems and resulted in Spreedly returning HTTP 500 response codes. At 9:08 pm UTC, while Spreedly was investigating the issue, the responses from Worldpay returned to the expected format, and the system returned to normal functionality.

Any transactions that experienced this failure are safe to retry.

Next Steps

We are taking the following steps to help prevent issues like this in the future:

  1. We are working to modify our systems to better handle unexpected responses and to ensure they do not cause errors in the future.
  2. We will be auditing our systems to see if we can find other locations where unexpected responses like this might result in errors so that these areas can also be addressed.

Conclusion

We apologize for any disruption this incident may have caused and are taking steps to ensure that our systems continue to be resilient for our customers.

Posted 13 days ago. Sep 30, 2019 - 16:31 EDT

Resolved
The errors have been resolved. This incident is closed.
Posted 18 days ago. Sep 25, 2019 - 17:20 EDT
Identified
The elevated error rates have been identified as affecting the Worldpay gateway. The investigation is ongoing.
Posted 18 days ago. Sep 25, 2019 - 16:45 EDT
Investigating
We are currently investigating elevated error rates.
Posted 18 days ago. Sep 25, 2019 - 16:38 EDT
This incident affected: Core Transactional API.