Spreedly API Errors
Incident Report for Spreedly
Postmortem

On February 28th at 18:30 UTC, a deployment resulted in a required database table to become locked and unavailable to the Spreedly API. This resulted in Spreedly’s transactional API returning elevated 500 and 502 error responses. A total of 9% of overall API requests were affected for a duration of approximately 34 minutes. The core API recovered after the database was returned to normal operations.

What Happened

At 18:30 UTC as part of a code deployment, schema updates were made to a database which automatically locked the database; this disruption prevented core Spreedly APIs from performing functions required to process transactions nominally. Errors with core API began at 18:31 UTC, and the Spreedly team was alerted via automated monitoring at 18:33 UTC, and identified the issue. After rolling back the deployment and removing the database lock, no further errors were observed after 19:05 UTC.

Next Steps

Spreedly has updated their internal roadmap with specific items intended to lessen the reliance on secondary systems for the transactional API and improve monitoring and resiliency when disruptions to secondary systems occur.

Posted Mar 03, 2022 - 17:27 EST

Resolved
After deploying the fix, all systems appear to be stabilized and functioning. The incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.

We apologize for any inconvenience and disruption to service.
Posted Feb 28, 2022 - 14:24 EST
Monitoring
A fix has been implemented addressing the intermittent 500 errors.

We are actively monitoring the results.
Posted Feb 28, 2022 - 14:07 EST
Identified
We have identified the cause of the API errors.

We are currently working on implementing a fix.

Updates will be provided as they become available.
Posted Feb 28, 2022 - 13:58 EST
Investigating
We have identified an issue causing intermittent 500 errors on Spreedly's Core API.

This is impacting all transactions and requests to Spreedly's API.

Updates will be provided as they become available.
Posted Feb 28, 2022 - 13:47 EST
This incident affected: Core Transactional API.