Starting from approximately 1600 UTC on September 10th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected the following day around 1700 UTC, September 11th. Once identified the issue was tracked down to a credential being rotated that the Dashboard uses to communicate with other internal systems, and that credential being unexpectedly (and silently) invalidated after being put into usage. We were able to roll back to the pre-rotation credential to get the system working again, and all the missing data filled in automatically in less than an hour.
There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.
Two primary next steps have been queued up:
- We’ll be re-examining and making changes to the process we use for inter-application credentials.
- We’ll be making tweaks to how we handle errors in the Dashboard - too many non-critical notifications caused us to miss the errors that this incident was generating.
As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via support@spreedly.com if you have any follow-up questions or concerns.
We’ll publish a postmortem in the next few days with more details about what happened.