Starting from approximately 2000 UTC on September 19th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected two days later around 1930 UTC, September 21st. Once identified the issue was tracked down to a new feature deploy for the Dashboard, and while we’re currently still investigating we currently believe the error was due to unexpected cache misses blocking data ingestion. Once we identified the new feature as the cause, it was reverted, and metrics were able to fill in with no issue.

There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.

Two primary next steps have been queued up:

1. We’ll be making tweaks to how we handle errors in the Dashboard so that we detect this type of issue sooner. This is work that is already queued up from our incident ten days earlier.
2. We’re continuing to research what caused metrics ingestion to fail with the new feature in place, and will monitor closely once we have the issue fixed and re-deploy it.

As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via support@spreedly.com if you have any follow-up questions or concerns.