Increased timeouts for a small number of transactions

Incident Report for Spreedly

Postmortem

Summary

On 9/20/23, a third-party CDN provider had internal issues connecting to AWS data centers in Columbus, OH. This caused increased timeouts for transactions originating from these locations.

What Happened

20:37 Spreedly declares SEV-2 incident and notifies customers
21:01 - Fastly - Status posted and subscribers notified of elevated errors in CMH POP
21:21 - Fastly - Remediation plan for third party service provider issue identified
21:26 - Fastly - Remediation plan deployed
21:30 - Fastly - Incident confirmed to be fully mitigated
22:26 - Spreedly considers incident to be resolved

Based on Fastly’s Statuspage resolution on this incident:

“During this incident, customers would have experienced increased latency and elevated 5xx errors in our Columbus (CMH) POP, as well as, elevated errors when attempting to access configuration management and API services through our Control Plane from 18:40 to 21:30 UTC.”

Next Steps

No additional steps are required at this time

Conclusion

This is an unfortunate third-party service provider outage, and Spreedly has been in communication with the provider to determine the reason for the delay in notification.

Posted Sep 29, 2023 - 12:26 EDT

Resolved

This incident has been resolved.

Posted Sep 20, 2023 - 21:10 EDT

Update

We have closely monitored the situation and received official confirmation from our third-party provider that the issue has been resolved. As a result, we consider the incident resolved and do not anticipate any further impact on our customers.

We apologize for any inconvenience or disruption.

Posted Sep 20, 2023 - 20:47 EDT

Monitoring

Since 2023-09-20 2:41 PM ET this afternoon, there has been an issue outside of our platform detected with regional network service.

Traffic has been rerouted in this region, and we continue to monitor traffic. Please reach out to our Support team if you need any additional help.

Posted Sep 20, 2023 - 18:15 EDT

Investigating

Spreedly has discovered an issue that could potentially impact our services, leading to Increased timeouts for a small number of transactions.

We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available.

Thank you for your patience.

Posted Sep 20, 2023 - 16:37 EDT

This incident affected: Core Transactional API.