Spreedly Dashboard - Insights not displaying data
Incident Report for Spreedly
Postmortem

Starting from approximately 2000 UTC on September 19th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected two days later around 1930 UTC, September 21st. Once identified the issue was tracked down to a new feature deploy for the Dashboard, and while we’re currently still investigating we currently believe the error was due to unexpected cache misses blocking data ingestion. Once we identified the new feature as the cause, it was reverted, and metrics were able to fill in with no issue.

There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.

Two primary next steps have been queued up:

  1. We’ll be making tweaks to how we handle errors in the Dashboard so that we detect this type of issue sooner. This is work that is already queued up from our incident ten days earlier.
  2. We’re continuing to research what caused metrics ingestion to fail with the new feature in place, and will monitor closely once we have the issue fixed and re-deploy it.

As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via support@spreedly.com if you have any follow-up questions or concerns.

Posted 2 months ago. Oct 05, 2018 - 11:21 EDT

Resolved
We've identified the cause as a change that was deployed which broke data ingestion. We've reverted the change to allow us to investigate the issue.

Data should be fully restored to Insights within 1-2 hours.

A postmortem will be published next week once we've had a chance to fully investigate.
Posted 3 months ago. Sep 21, 2018 - 16:41 EDT
Investigating
We are currently investigating an issue causing the Spreedly Dashboard to not show transaction data after September 19th 11PM UTC. Transactions and the API are not impacted, this is strictly an issue with visualizing the data in the Dashboard.
Posted 3 months ago. Sep 21, 2018 - 15:46 EDT
This incident affected: Supporting Services (Dashboard).