CVV Caching Error

Incident Report for Spreedly

Postmortem

December 10th, 2024 — CVV Caching Issue

At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from storing CVVs for payment methods when a /recache request was made. The adverse change resulted in a partial outage of Spreedly’s transactional API between 15:36 UTC and 17:07 UTC. Service was fully restored at 17:07 UTC. After a period of monitoring, Spreedly’s systems are stable, and no further impact is expected on customers.

What Happened

At approximately 15:36 UTC on December 10th, 2024, Spreedly Engineering attempted a code change to an internal service that inadvertently prevented Spreedly from extending the time a CVV exists in the cache. The adverse change resulted in a subset of transactions failing for customers who rely on Spreedly’s CVV recaching mechanism to complete a transaction. Spreedly Engineering team quickly mobilized and identified the adverse code deployment as the issue and immediately began reverting the change. The code reversion was completed at approximately 17:07 UTC, with customers reporting no additional impact after the code reversion.

Next Steps

Spreedly Engineering is reviewing internal observability to determine how to automate the detection of such an issue in the future.

Posted Dec 12, 2024 - 17:35 EST

Resolved

After closely monitoring and confirming that all systems are stabilized and functioning as expected, this incident is considered resolved. No further customer impact is expected.

We are completing our investigation with regard to the causes of the incident and any residual impact. A post-mortem review will be published.

We apologize for any inconvenience or disruption.

Posted Dec 10, 2024 - 15:22 EST

Monitoring

Spreedly experienced an issue with Caching the CVV. Once identified, we immediately began working on a resolution. We’ve implemented a fix and stabilized the system.

Our team is monitoring the situation closely to ensure there are no additional impacts.

Please reach out to our Support team if you need any additional help.

Posted Dec 10, 2024 - 12:16 EST

Identified

Spreedly has discovered an issue that could potentially impact our services, leading to errors with caching the CVV.

We are taking all necessary measures to investigate the matter and determine the extent of the situation. We provide an update as soon as more details are available.

Thank you for your patience.

Posted Dec 10, 2024 - 11:59 EST

This incident affected: iFrame.