Spreedly logo
  • Operational
  • Degraded Performance
  • Partial Outage
  • Major Outage
  • Maintenance
Delays in listing transactions using the API listing endpoints.
Incident Report for Spreedly

May 30, 2024 —Delays in listing transactions using the API listing endpoints

Executive Summary

During a routine maintenance event, a piece of software responsible for reporting activity to systems used to generate customer reports and API listing endpoints silently failed to start. During the 40 minutes in which the software was non-functional, API calls for listing transactions and payment methods created in this timeframe would not have been reported. This reporting issue did not affect the outcome of customer transactions, only the ability for Spreedly systems to report that data to customers. After correcting the issue, Spreedly backfilled all available payment method and transaction data to reporting systems.

What Happened

On 2024-05-30 at 15:15 UTC, production traffic was switched to a newly updated system responsible for generating customer activity data. This system silently failed to produce the necessary information, which resulted in a data gap. Upon automated alerting at 15:37, the team halted their maintenance work and reverted the system to a known good state by 15:50. After confirming that all systems were functioning normally and that no transactions were impacted, the team gathered all available data and replayed the data into the system so that reporting systems would accurately reflect that data.

Next Steps

Spreedly will continue to work to eliminate the possibility of silent failures of this functionality, testing to prevent non-functional reporting to receive production traffic, and introduce additional automated monitoring that will more quickly identify any loss of reporting volume.


Spreedly values the trust our customers place in us to handle one of the most critical aspects of their business, and welcome the opportunity to discuss any questions, concerns, or comments you have regarding this incident. Please reach out to our support staff or your account management team with your concerns.

Posted May 31, 2024 - 17:43 EDT

The Spreedly team has identified and corrected the cause for the gap in reporting data for some customers.

Reporting is affected for both API and Dashboard users during the period of time from 2024-05-30 15:15 - 2024-05-30 15:55 UTC. This gap did not affect the state or success of customer transactions, but reporting during this time frame may not reflect those transactions.

We have confirmed that reporting outside of the specified time range is fully functional. We will communicate when reporting data has been corrected via the post-mortem for this incident.
Posted May 30, 2024 - 14:37 EDT
We have identified a gap in reporting for some customers listing recent transactions using the API listing endpoints, which may affect /v1/transactions/ and list Payment Methods /v1/payment_methods. We are aware of the issue and are working to implement a fix.
Posted May 30, 2024 - 12:10 EDT
This incident affected: Core Secondary API.