May 28, 2022 — Circuit Breaker Triggered at Lower than Intended Threshold for Adyen Gateway

In normal operations, Spreedly employs the use of ‘Circuit Breakers’ to manage traffic flow when a downstream gateway is producing too many failures, or consuming too many resources. This allows normal customer traffic to continue in cases where issues with a downstream gateway would otherwise impact other customers' requests.

What Happened

On March 28,2022 between 18:45 - 19:22 UTC, the Circuit Breaker for the Adyen Gateway erroneously activated, causing some customers to receive an HTTP 503 error in response to their request.

We discovered an errant configuration in the Circuit Breaker and took steps to address it, returning systems to normal operation

Next Steps

We’ve undertaken a tech initiative to reexamine how Circuit Breakers are configured and their optimal use going forward.

Posted Apr 08, 2022 - 12:18 EDT

Resolved

After a period of monitoring, we have detected no further service degradation and are considering this incident resolved. We will perform a root cause analysis and publish a post-mortem afterwards.

Posted Mar 28, 2022 - 18:12 EDT

Monitoring

There was an issue with our gateway performance monitoring service that caused a degradation of service with our Adyen gateway from 2:45 to 3:22 Eastern Daylight Time. We have resolved the degradation, are monitoring the situation, and are investigating the root cause.

Posted Mar 28, 2022 - 17:34 EDT

This incident affected: Gateway Integrations (Gateways).