Spreedly Dashboard - Insights not displaying data
Incident Report for Spreedly
Postmortem

Starting from approximately 1600 UTC on September 10th, 2018, metrics stopped flowing into the Spreedly Dashboard. This was detected the following day around 1700 UTC, September 11th. Once identified the issue was tracked down to a credential being rotated that the Dashboard uses to communicate with other internal systems, and that credential being unexpectedly (and silently) invalidated after being put into usage. We were able to roll back to the pre-rotation credential to get the system working again, and all the missing data filled in automatically in less than an hour.

There was no impact on transaction processing from this event - it was localized to metrics ingestion in the Dashboard itself.

Two primary next steps have been queued up:

  1. We’ll be re-examining and making changes to the process we use for inter-application credentials.
  2. We’ll be making tweaks to how we handle errors in the Dashboard - too many non-critical notifications caused us to miss the errors that this incident was generating.

As always, we apologize for any impact the incident may have had on you - it’s never fun (or reassuring) to go looking for data you need and find it missing. Thank you for your patience as we continue improving and scaling Spreedly, and as always, let us know via support@spreedly.com if you have any follow-up questions or concerns.

Posted 3 months ago. Sep 14, 2018 - 15:15 EDT

Resolved
Insights data has been fully restored.

We’ll publish a postmortem in the next few days with more details about what happened.
Posted 3 months ago. Sep 11, 2018 - 15:33 EDT
Monitoring
We’ve identified the cause of the issue and have implemented a fix. It will take up to a couple of hours for historic data to fill in. We’ll continue to monitor and close out the incident once all data is filled. As always, we’ll publish a postmortem in the next few days with more details about what happened.
Posted 3 months ago. Sep 11, 2018 - 14:55 EDT
Investigating
We are currently investigating an issue causing the Spreedly Dashboard to not show transaction data after ~4pm UTC. Transactions and the API are not impacted, this is strictly an issue with visualizing the data in the Dashboard.
Posted 3 months ago. Sep 11, 2018 - 13:39 EDT
This incident affected: Supporting Services (Dashboard).