During a scheduled language version upgrade of the Spreedly transaction processing service, Apple Pay payment methods could not be created. The error message was “The encrypted portion of the payment data could not be decrypted, most likely due to other validation errors.”

What Happened

On February 5, 2020 at 12:02pm EST, Spreedly Engineers released a new version of the application, which included a major programming language version update. This programming language update included changes to the way decryption parameters must be specified. The Apple Pay decryption library had not been updated to use this new parameter. Automated and manual testing did not catch the issue, because both types of tests used data that did not exercise the actual decryption path. In the production environment, this caused decryption of all Apple Pay payment_data tokens to fail, which occurs when creating payment methods.

Spreedly Engineers determined there was a systemic issue with Apple Pay payment methods, and began investigating at 5:50pm EST. Spreedly Engineers updated the decryption library used for Apple Pay to work with the new language version, and deployed a fix at 7:13pm EST.

Next Steps

We have completed the following activities to prevent this situation from happening in the future:

1. Added metrics to count attempted, successful, and failed Apple Pay decryptions over time.
2. Trained engineers on the use of existing Apple Pay metrics.
   

We will evaluate the following activities to prevent this situation from happening in the future:

1. Update automated tests for Apple Pay to verify decryption is functioning.
2. Create manual tests for Apple Pay payment method decryption.
3. Set automated alerting thresholds for Apple Pay decryption failures.

Conclusion

We apologize for any disruption this incident may have caused and are taking steps to ensure that our systems continue to be resilient for our customers.