Starting from approximately 1600 UTC on September 10th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected the following day around 1700 UTC, September 11th. Once identified the issue was tracked down to a credential being rotated that the Dashboard uses to communicate with other internal systems, and that credential being unexpectedly (and silently) invalidated after being put into usage. We were able to roll back to the pre-rotation credential to get the system working again, and all the missing data filled in automatically in less than an hour.
There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.
Two primary next steps have been queued up:
As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via firstname.lastname@example.org if you have any follow-up questions or concerns.