Spreedly logo
  • Operational
  • Degraded Performance
  • Partial Outage
  • Major Outage
  • Maintenance
TransFirstTransactionExpress Transaction Issues
Incident Report for Spreedly
Postmortem

As part of a major framework update to the Core API, a code dependency was updated. This dependency was not backwards compatible and another codebase responsible for sending transactions to gateways failed to properly send transactions for a single gateway type. Because this behavior presented in only a single gateway type and traffic at the time was very low, automated alerting failed to detect the regression. Upon being alerted by a customer of the issue, Spreedly engineering downgraded a code dependency to a backwards compatible version and the issue was resolved.

What Happened

As part of a major framework update for Core, a number of code dependencies were updated, including one for XML parsing. The newer version of this dependency parsed and generated XML in a way that was not fully backwards compatible. The mapping code in the TransFirstTransactionExpress gateway in the Active Merchant adapter improperly formatted a nested XML attribute, causing the gateway to reject requests.

While Spreedly actively monitors for anomalous gateway failures, due to the nature of typical transaction flows, Gateway rejections are common. To avoid excessive alerting, Spreedly uses “squelching”; in other words, we require a minimum number of overall transactions to be present in any given time period of monitoring before applying anomaly detection. Unfortunately, this particular gateway's transaction count fell below the “squelch” minimum, so the failure rates were not detected in an automated fashion.

After a customer notification was escalated to Engineering, the team quickly identified the cause of the issue and downgraded the XML parsing dependency to a stable version that was backwards compatible with the existing Core and Active Merchant codebases. Upon deployment of this dependency version, transactions were once again accepted by the gateway.

Next Steps

  • Spreedly will update Active Merchant to support the latest XML parsing dependency
  • Spreedly will investigate better “squelch” control for low volume gateways
Posted Jul 28, 2022 - 16:37 EDT

Resolved
After a period of monitoring, we have observed that issues with the TransFirstTransactionExpress Gateway have been resolved.

We apologize for any inconvenience and disruption to service.
Posted Jul 13, 2022 - 21:18 EDT
Monitoring
We have implemented a change and are seeing successful transactions being sent to TransFirstTransactionExpress gateway. We are monitoring to ensure the issue is fully resolved.
Posted Jul 13, 2022 - 20:58 EDT
Identified
We have identified an issue with transactions being sent to TransFirstTransactionExpress gateway. Our Engineering team is reviewing a potential resolution. We will be updating as this investigation continues.
Posted Jul 13, 2022 - 20:15 EDT
This incident affected: Core Transactional API and Gateway Integrations (Gateways).