As part of a major framework update to the Core API, a code dependency was updated. This dependency was not backwards compatible and another codebase responsible for sending transactions to gateways failed to properly send transactions for a single gateway type. Because this behavior presented in only a single gateway type and traffic at the time was very low, automated alerting failed to detect the regression. Upon being alerted by a customer of the issue, Spreedly engineering downgraded a code dependency to a backwards compatible version and the issue was resolved.
As part of a major framework update for Core, a number of code dependencies were updated, including one for XML parsing. The newer version of this dependency parsed and generated XML in a way that was not fully backwards compatible. The mapping code in the TransFirstTransactionExpress gateway in the Active Merchant adapter improperly formatted a nested XML attribute, causing the gateway to reject requests.
While Spreedly actively monitors for anomalous gateway failures, due to the nature of typical transaction flows, Gateway rejections are common. To avoid excessive alerting, Spreedly uses “squelching”; in other words, we require a minimum number of overall transactions to be present in any given time period of monitoring before applying anomaly detection. Unfortunately, this particular gateway's transaction count fell below the “squelch” minimum, so the failure rates were not detected in an automated fashion.
After a customer notification was escalated to Engineering, the team quickly identified the cause of the issue and downgraded the XML parsing dependency to a stable version that was backwards compatible with the existing Core and Active Merchant codebases. Upon deployment of this dependency version, transactions were once again accepted by the gateway.