Spreedly API Errors
Incident Report for Spreedly
Postmortem

On March 3rd, 2022 at 15:37 UTC a configuration error in internal DNS caused core transactional API calls to reference a service that was not yet operational. This resulted in Spreedly’s transactional API returning elevated 500 and 502 error responses. A total of 9% of overall API requests were affected for a duration of approximately 4 minutes. The core API recovered after the DNS configuration was corrected.

What Happened

At 15:37 UTC as part of a service re-deployment, DNS updates were made to direct traffic to a cluster which was not yet operational; this prevented core Spreedly APIs from performing functions required to process transactions. Errors with core API began at 15:37 UTC, and the Spreedly team immediately identified the issue. After correcting the DNS entries, no further errors were observed after 15:41 UTC.

Next Steps

Spreedly has updated their internal documentation and processes with specific items intended to reduce the likelihood of this type of issue.

Posted Mar 04, 2022 - 15:13 EST

Resolved
From 15:37:00 UTC to 15:41:00 UTC (4 minutes) the Spreedly Core API returned 500 errors for approximately 1% of transactions.

The underlying issue has been determined and a post incident review will be published.

We apologize for any inconvenience and disruption to service.
Posted Mar 03, 2022 - 11:43 EST
This incident affected: Core Transactional API.