2017-02-08 Ruby Gem errors

Issue Summary

On February 8, 2017 at 16:33 EST, we deployed code that removed the eligible_for_card_updater field from the response body of some of our API calls. It's a field that was added over 2 years ago as the start of a feature that never went live. In retrospect, the field should never have been added.

Removing a field from a response body is a breaking change and we at Spreedly should only remove such fields in a new API version, since customers could be dependent on a field being there.

Unfortunately, the Spreedly Ruby gem is brittle and the mere absence of the unused field caused errors for a small set of customers. For those customers, transactions were being run, but the code to parse the response after Spreedly responded was raising an exception. Some customers were retrying those transactions causing more duplicated transactions.

We deployed the fix at 19:11 EST, approximately 2.5 hours after the start of the incident.

What Happened

At 17:30 EST, a customer contacted us about errors involving duplicate charges and multiple failed transactions. We began researching, and at 18:39 EST, we became aware that a few other customers were reporting similar issues. We brought in additional team members to help us address the issue.

At 19:04 EST we reverted the most recent set of changes, recognizing that they weren't absolutely necessary and knowing that they could be related. By 19:08 EST, we learned the true cause (the eligible_for_card_updater field removal) and we were able to revert that change at 19:11 EST.

For the customers affected, the error was happening in the parsing of the response, causing their record of the actual transaction result to be different from the actual result. This means that our failure caused significant work for some of our customers. In particular, our customers needed to reconcile the transactions that happened during the time window with Spreedly's result and then refund the duplicated transactions.

Next Steps

It took us over one hour from the time a single customer contacted Spreedly about the issue until we realized the issue was affecting additional customers. This is unacceptable and we're putting measures in place for us to improve in correlating common issues being reported from independent channels. Although we have an on-call schedule and incident management process, this incident revealed some ways we need to improve in this area too. In particular we'll be implementing improvements to better include additional teams when an incident occurs.

Commensurate with these process improvements, we also need to be more explicit with the Spreedly Ruby gem’s place in our ecosystem. In the short term we will be making small improvements to its operation so it can handle such situations in a more resilient fashion. We will also consider how to include testing of the gem itself on each platform update. For the long term we need to decide if maintaining a single client library provides value for our customers and, if not, what the migration path looks like for current users of the gem.

Conclusion

Evolving and improving an API means that fields will be renamed and others will be removed. We need to ensure that such changes only happen in a new version of the API such that we don't break customers using the current version of our API. Our customers expect non-breaking improvements and rightly so. We're so sorry we failed to meet our customers' expectations in this case and that we increased their workloads by necessitating a reconciliation process after the incident.