Starting from approximately 2000 UTC on September 19th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected two days later around 1930 UTC, September 21st. Once identified the issue was tracked down to a new feature deploy for the Dashboard, and while we’re currently still investigating we currently believe the error was due to unexpected cache misses blocking data ingestion. Once we identified the new feature as the cause, it was reverted, and metrics were able to fill in with no issue.
There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.
Two primary next steps have been queued up:
As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via firstname.lastname@example.org if you have any follow-up questions or concerns.