On February 18th, 2022 at 3:11 PM UTC, Spreedly experienced a minor disruption of service due to an application infrastructure change that reduced API request handling capacity. The service disruption resulted in a period of increased errors for customers attempting to use the Spreedly API.
The Spreedly transactional API returned
500 level error responses for around 1% of total API request volume for a duration of approximately 100 minutes. Normal operations resumed after a configuration setting was corrected.
Spreedly engineers attempted to deploy an update to application infrastructure at 3:11 PM UTC to improve resilience and scalability. At 3:51 PM UTC, Spreedly engineers were notified of potential disruption of service in the core transactional API related to an increase in 502 error response codes and began investigating. Spreedly engineers discovered that a configuration setting did not propagate along with the application infrastructure update. At 4:52 PM UTC, the configuration setting was corrected and redeployed, resolving the issue.
Configuration settings used in the deployment of application infrastructure will be updated to remove conditional logic that is no longer needed and which contributed to the disruption of service.