April 27th, 2023 — Temporary resource contention in our environment resulted in 500 response errors
A code merge inadvertently prompted creation of a large queue which resulted in transitory transaction errors.
What Happened
A code merge inadvertently impacted batch processing services which created a large queue and which put pressure on a critical path database service. This resulted in some transitory errors for transactions, the majority of which were re-tried successfully.After the issue was resolved by restarting the processing service and the queues returned to normal, we refined our queue monitoring process to include additional monitoring conditions.
Next Steps
Work has been started to optimize our database performance in order to prevent similar issues in the future.
Conclusion
We apologize for this disruption to service, and will continue to drive internal improvements to avoid similar impact in the future.