On February 24th, 2022 at 16:25 UTC, an upstream service provider declared an incident that caused Spreedly’s transactional API to return elevated 500 and 502 error responses. A total of 7% of overall API requests were affected for a duration of approximately 61 minutes. Normal operations resumed after the upstream provider resolved their incident.
What Happened
An upstream service provider disruption prevented core Spreedly APIs from performing functions required to process a subset of transactions nominally. The disruption first presented as access issues to the Spreedly Dashboard service, and at the same time Spreedly identified failed transactions (around 16:41 UTC). The Spreedly team began investigation and noted that the upstream providers disruption’s timing coincided with the timing of failed transactions on Spreedly’s systems. Once the upstream disruption subsided at 17:26 UTC, Spreedly’s systems returned to a normal functioning state.
Next Steps
Spreedly has updated their internal roadmap with specific items intended to lessen the reliance on upstream service providers for the transactional API and improve resiliency when disruptions to 3rd party services occur.
We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.
We apologize for any inconvenience and disruption to service.
Since that resolution, we have not seen recurrence of the previously reported issues with Dashboard and Core API.
We are continuing to monitor to ensure that service has been fully restored.
We have also determined that requests to the Payment Methods List Transactions endpoint of the Core API are impacted by these errors.
Updates will be provided as they become available.
We will provide updates as they become available.