Spreedly Transaction Errors
Incident Report for Spreedly
Postmortem

Spreedly experienced an issue within an automated deployment process, which resulted in a degraded performance for the API with a partial outage for gateways.

What Happened

On January 4, 2021, Spreedly engineers released a new version of the application, the deployment of which failed due to an issue within an automated process. This made a negative impact to the production servers. Spreedly’s internal systems recognized communication issues with some payment gateways. Internal response teams immediately started to investigate the issue. While investigating, it became clear that the communication issues were not limited to communication with some gateways, but also degraded performance for the Spreedly API.

Spreedly engineers released an update to the deployment process which fixed the issue with the internal automated process. As a result, our internal systems showed signs of the API and gateway connections returning to normal.

Next Steps

  • Investigating how to ensure changes to automated deployment processes will not affect production traffic.
  • Proactively add notices to Status Page when the Spreedly team has discovered degradation.
Posted Jan 19, 2021 - 10:42 EST

Resolved
After deploying the fix, all systems appear to be stabilized and functioning. The incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and will publish a post-mortem.

We apologize for any inconvenience and disruption to service.
Posted Jan 04, 2021 - 15:52 EST
Monitoring
A fix has been implemented addressing the Spreedly API errors.

We are actively monitoring the results.
Posted Jan 04, 2021 - 14:59 EST
Investigating
We are investigating an issue causing a high volume of errors with the Spreedly API.

This is impacting a subset of transactions and requests to Spreedly's API.

Updates will be provided as they become available.
Posted Jan 04, 2021 - 14:19 EST
This incident affected: Core Transactional API and Gateway Integrations.