May 14, 2025 — HTTP 500 Errors and Google Pay Degradation

‌High-Level Summary

On May 14, 2025 at 03:18 pm UTC, a configuration change was deployed to enhance application fault tolerance which, resulted in a brief period of HTTP 500 errors. While the breaking change was reverted within a minute of deployment, there were still instances of cached data which extended the impact for a total of 4 minutes of partial service outage, until 03:22 pm UTC.

A secondary issue occurred as a result of the timing of this deployment coinciding with the regeneration of the Google Pay verification keys for decryption, which resulted in a 51 minutes degradation of Google Pay transactions until 04:09 pm UTC.

What Happened

A configuration change intended to enhance application fault tolerance to Redis failures inadvertently introduced serialization issues when accessing Redis keys. This resulted in new cache failing to be successfully read from the existing application instances, which resulted in HTTP 500 errors.

Additionally, at the same time this change was deployed, the Google Pay verification keys were regenerated. Google Pay decryption stopped working due to the cached keys not having serialized properly. These keys then had to be deleted from the cache to enable the system to auto generate a new pair.

Next Steps

• Spreedly has implemented additional observability improvements via our automated alerting system to enhance monitoring of Google Pay services.