Dashboard Unavailable
Incident Report for Spreedly
Postmortem

On February 24th, 2022 at 16:25 UTC, an upstream service provider declared an incident that caused Spreedly’s transactional API to return elevated 500 and 502 error responses. A total of 7% of overall API requests were affected for a duration of approximately 61 minutes. Normal operations resumed after the upstream provider resolved their incident.

What Happened

An upstream service provider disruption prevented core Spreedly APIs from performing functions required to process a subset of transactions nominally. The disruption first presented as access issues to the Spreedly Dashboard service, and at the same time Spreedly identified failed transactions (around 16:41 UTC). The Spreedly team began investigation and noted that the upstream providers disruption’s timing coincided with the timing of failed transactions on Spreedly’s systems. Once the upstream disruption subsided at 17:26 UTC, Spreedly’s systems returned to a normal functioning state.

Next Steps

Spreedly has updated their internal roadmap with specific items intended to lessen the reliance on upstream service providers for the transactional API and improve resiliency when disruptions to 3rd party services occur.

Posted Mar 03, 2022 - 17:29 EST

Resolved
After a period of monitoring, all systems are stabilized and functioning. The incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.

We apologize for any inconvenience and disruption to service.
Posted Feb 24, 2022 - 13:47 EST
Monitoring
We have identified that an issue with an upstream service provider has been resolved.

Since that resolution, we have not seen recurrence of the previously reported issues with Dashboard and Core API.

We are continuing to monitor to ensure that service has been fully restored.
Posted Feb 24, 2022 - 12:56 EST
Identified
We have identified the cause of the Dashboard errors.

We have also determined that requests to the Payment Methods List Transactions endpoint of the Core API are impacted by these errors.

Updates will be provided as they become available.
Posted Feb 24, 2022 - 12:41 EST
Investigating
We are currently experiencing degraded performance on the Spreedly Insights Dashboard. The Dashboard may be unavailable for some users.

We will provide updates as they become available.
Posted Feb 24, 2022 - 12:13 EST
This incident affected: Core Transactional API and Supporting Services (Dashboard).