Spreedly logo
  • Operational
  • Degraded Performance
  • Partial Outage
  • Major Outage
  • Maintenance
Spreedly API Errors
Incident Report for Spreedly
Postmortem

February 18, 2022 — Elevated number of 502 responses from the Core API

On February 18th, 2022 at 3:11 PM UTC, Spreedly experienced a minor disruption of service due to an application infrastructure change that reduced API request handling capacity. The service disruption resulted in a period of increased errors for customers attempting to use the Spreedly API.

The Spreedly transactional API returned 500 level error responses for around 1% of total API request volume for a duration of approximately 100 minutes. Normal operations resumed after a configuration setting was corrected.

What Happened

Spreedly engineers attempted to deploy an update to application infrastructure at 3:11 PM UTC to improve resilience and scalability. At 3:51 PM UTC, Spreedly engineers were notified of potential disruption of service in the core transactional API related to an increase in 502 error response codes and began investigating. Spreedly engineers discovered that a configuration setting did not propagate along with the application infrastructure update. At 4:52 PM UTC, the configuration setting was corrected and redeployed, resolving the issue.

Next Steps

Configuration settings used in the deployment of application infrastructure will be updated to remove conditional logic that is no longer needed and which contributed to the disruption of service.

Posted Feb 24, 2022 - 16:14 EST

Resolved
After deploying the fix, all systems are stabilized and functioning. This incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.

We apologize for any inconvenience and disruption to service.
Posted Feb 18, 2022 - 12:02 EST
Monitoring
A fix has been implemented addressing the API errors.

We are actively monitoring the results.
Posted Feb 18, 2022 - 11:46 EST
Investigating
We are investigating an issue causing intermittent 502 errors on Spreedly's Core API.

This is impacting some transactions and requests to Spreedly's API.

Updates will be provided as they become available.
Posted Feb 18, 2022 - 11:31 EST
This incident affected: Core Transactional API.