November 22, 2021 — Partial outage of data access via API

What Happened

During a routine failover between Spreedly’s active and passive Riak clusters, an error occurred which turned off realtime replication between the two data stores. This meant that data being written to one cluster was not being synced to the other in real time. Because of this discrepancy, some customers received a not found error when attempted to access stored info.

On 11/23, Spreedly deployed a patch that allowed customers to receive data that had been temporarily unreachable after the failover.

Next Steps

Spreedly will complete an internal review of documentation and processes that led to the realtime replication failure between clusters. Spreedly has also built tooling that will allow for real-time patching in the future should a similar incident occur.

Posted Dec 01, 2021 - 17:24 EST

Resolved

After deploying the fix, any Spreedly tokens that were inaccessible should now be available. The incident is being considered resolved.

We are still investigating to understand the specific causes of the incident and any residual impact. A post incident review will be published.

We apologize for any inconvenience and disruption to service.

Posted Nov 23, 2021 - 15:21 EST

Update

Spreedly tokens created between 3:00-5:15 EST may be temporarily inaccessible. Customers may experience failed API calls that reference these tokens. We are working to restore access.

Posted Nov 23, 2021 - 10:25 EST

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 22, 2021 - 21:42 EST

Identified

The issue has been identified and a fix is being implemented.

Posted Nov 22, 2021 - 20:59 EST

Investigating

We have identified an issue causing intermittent errors on Spreedly's Core API.

This is impacting some transactions and requests to Spreedly's API for select customers.

Updates will be provided as they become available.

Posted Nov 22, 2021 - 19:45 EST

This incident affected: Core Transactional API.