A high load of test transactions from a customer caused the post-transaction data pipeline populating the Dashboard data source to drop a subset of its messages.
On April 7th, 2020 at 10:10 am EDT, Spreedly began receiving alerts from our core systems which indicated that they were under unusually high load. The requests which caused this load were test transactions originating from a customer.
At 10:24 am EDT, messages being passed within the post-transaction data pipeline began to fail. This led to data in the Dashboard system becoming out of date. A subset of these messages continued to fail until 12:00pm EDT when Spreedly fully recovered the service. The failed messages were replayed over the next few hours, and the Dashboard data source was fully repopulated at 4:24pm EDT.
Transaction processing was not impacted during these events. The issue only manifested as a temporary delay in delivery of data to Dashboard.
Spreedly will evaluate test transaction rate limiting, and will re-assess the data pipeline’s load balancing mechanisms.