On March 3rd, 2022 at 15:37 UTC a configuration error in internal DNS caused core transactional API calls to reference a service that was not yet operational. This resulted in Spreedly’s transactional API returning elevated 500 and 502 error responses. A total of 9% of overall API requests were affected for a duration of approximately 4 minutes. The core API recovered after the DNS configuration was corrected.

What Happened

At 15:37 UTC as part of a service re-deployment, DNS updates were made to direct traffic to a cluster which was not yet operational; this prevented core Spreedly APIs from performing functions required to process transactions. Errors with core API began at 15:37 UTC, and the Spreedly team immediately identified the issue. After correcting the DNS entries, no further errors were observed after 15:41 UTC.

Next Steps

Spreedly has updated their internal documentation and processes with specific items intended to reduce the likelihood of this type of issue.