Around 17:53 UTC on Friday, October 6th, and for a period of about two hours, due to an upstream service provider rotating shared credentials, we became unable to reach the event pipeline service that is responsible for propagating changes out to secondary systems. This resulted in a temporary gap in reporting data for our query API endpoints such as listing payment methods and transactions, as well as a gap in data in Insights. At no point were revenue-affecting transactions down or inaccessible.
We will be looking closely at how to more quickly propagate changes in our upstream provider’s configuration. However, due to the secondary nature of the affected systems, we will also be willing to accept some amount of downtime or maintenance. For these situations we will look at investing in our recovery tool chain to more quickly fill in associated gaps in data and completely restore functionality.
The transactional service of Spreedly is architected to be as isolated as possible from other secondary services. We prioritize the uptime of revenue-affecting API calls over other secondary concerns like reporting and visualization, though we also acknowledge that many businesses do rely on these abilities for some of their business processes. While we do still consider these secondary functions to be sacrificial when experiencing systems instability, we also hold ourselves accountable for the operation and uptime of these functions and apologize for the inconvenience posed by their degradation.