Spreedly API Errors
Incident Report for Spreedly
Postmortem

December 22nd, 2021 — Spreedly API Errors

On December 22nd at 12:10PM UTC a major disruption in service at Spreedly’s cloud hosting provider resulted in a period of increased errors for customers attempting to use the Spreedly API.
The Spreedly API was returning error codes for a period of approximately 1 hour.
Normal operations resumed once the issue with the cloud provider’s physical infrastructure was resolved.

What Happened

Spreedly services are hosted and run on a common public cloud computing platform. A significant issue at this provider resulted in core Spreedly services being unavailable, resulting in customer requests failing and receiving a 503 or 502 error response code in most cases. Spreedly engineers were engaged and began undertaking steps to remediate the issue. However, during this time the cloud provider had begun to resolve the issue on their end and Spreedly services became available again once that issue was resolved.

Next Steps

Internal monitoring and alerting systems will be updated in order to be able to more quickly detect and react to upstream service issues in the future. In addition, Spreedly engineers are investigating system architectural changes with the goal to improve system resiliency during external issues like this.

Posted Dec 30, 2021 - 14:08 EST

Resolved
This incident has been resolved.
Posted Dec 22, 2021 - 10:33 EST
Monitoring
Spreedly became aware of intermittent connectivity issues impacting our Core API due to issues our cloud-based hosting provider experienced.

Our service provider has implemented a fix addressing the API errors. These errors were primarily manifesting as 503 HTTP errors, but may be related to other 5xx errors as well.

We are actively monitoring the results.
Posted Dec 22, 2021 - 08:44 EST
Investigating
We have identified an issue causing intermittent 500 errors on Spreedly's Core API.

This is impacting all transactions and requests to Spreedly's API.

Updates will be provided as they become available.
Posted Dec 22, 2021 - 08:11 EST
This incident affected: Core Transactional API.