Spreedly logo

Core 500s

Incident Report for Spreedly

Postmortem

Postmortem: September 30, 2025Payment Method Services Inaccessible

During a routine database configuration change, we lost connection to the Payment Methods storage table. This prevented the use of payment methods in transactions and the creation of new payment methods.

What Happened

During a routine database change to decommission extra nodes, a corruption occurred with our internal reference data. This corruption caused the database query optimizer to error when trying to process SQL queries on the payment methods storage, which resulted in the inability to access that storage table and led to errors from the application layer when trying to read, update, or create payment methods.

Next Steps

Payment methods successfully created during this period have had their fingerprints regenerated. Please contact Spreedly Support if you need the updated fingerprint values to utilize within your business processes and we will be happy to provide them.

Spreedly is also working with our database service provider on a more detailed root cause analysis. Moving forward, Spreedly will be proactively engaging with our database provider when performing these types of configuration changes.

-The Spreedly Team

Posted Oct 02, 2025 - 17:44 EDT

Resolved

The incident impacting operations involving payment methods has been resolved. All services are now operating normally.

During routine database maintenance, we encountered issues that affected tokenization and transactions using payment methods. We’ll share more details in the upcoming post-mortem.
Posted Sep 30, 2025 - 15:45 EDT

Monitoring

We have identified and resolved the issue and and monitoring for any additional impact.
Posted Sep 30, 2025 - 15:07 EDT

Identified

We are implementing a mitigation that should restore the ability to perform new tokenizations and operations on those payment methods. Some functionality, such as fingerprinting, may remain limited. Use of vaulted payment methods may remain limited. We’ll provide updates as we progress.
Posted Sep 30, 2025 - 14:55 EDT

Update

Currently we are experiencing issues with services that use payment methods
Posted Sep 30, 2025 - 14:25 EDT

Investigating

We are currently investigating a high number of 500 errors
Posted Sep 30, 2025 - 14:22 EDT
This incident affected: Core Transactional API.