Spreedly logo
  • Operational
  • Degraded Performance
  • Partial Outage
  • Major Outage
  • Maintenance
Gateway Errors
Incident Report for Spreedly
Postmortem

From 2021-10-20 at 20:15 UTC to 22:59 UTC, some merchant-initiated transactions were sent from Spreedly Core to Adyen with an invalid placeholder value sent in the networkTxReference field. This unexpected value caused Adyen to reject the transactions with this placeholder value.

What Happened

On 2021-10-20 at 20:15 UTC, Spreedly engineers released a change that was intended to automatically send network transaction IDs (NTIDs) to Adyen when performing merchant-initiated transactions, while falling back to the preexisting network_transaction_id gateway-specific field if it is sent. However, because the NTID is not available for some Adyen reference transactions, Spreedly Core stored the placeholder value of "none" in the NTID field, and incorrectly sent this message on to Adyen.

The transaction message for these failures was Field "networkTxReference" is missing or not valid. Because these transactions were rejected by Adyen, they can be safely retried.

Spreedly engineers rolled back the change by 22:59 UTC.

Next Steps

Spreedly Engineers have re-deployed an update to ignore the placeholder value on Adyen transactions, alongside other improvements to this new functionality.

Spreedly Engineering also plans to make the Stored Credentials framework more resilient, to avoid future incidents like this one.

Posted Nov 03, 2021 - 15:39 EDT

Resolved
Errors were identified in a recent change to the Adyen gateway integration. That change has since been rolled back and transactions are flowing normally again.
Posted Oct 20, 2021 - 19:17 EDT
This incident affected: Gateway Integrations (Gateways).