Spreedly was contacted regarding transactions failing with the error “invalid signature” against the Clearhaus gateway.

What Happened

On Friday March 27th, 2020 at 7:21am EDT, the keys required for secure communication between Spreedly and Clearhaus were updated. In doing this, the previous keys were revoked however the new key was not added to Spreedly’s application. At this point, any transaction against the Clearhaus gateway was failing with an “invalid signature” error.

On Saturday March 28th at 12:25pm EDT, Spreedly was informed of failed transactions on the Clearhaus gateway and immediately mobilized to understand the issue. The cause was identified and the new key was added to Spreedly’s application at 3:29pm EDT and the incident was considered fixed at this time. The team began monitoring for further failures at 4:32pm EDT. The first successful transaction since the start of the incident was recorded at 5:59pm EDT.

Next Steps

Spreedly will evaluate implementing additional alerts and monitors to catch these types of errors.

Note about Resolution Time and Severity

Originally it was reported that the resolution time for this incident was at 8:50pm EDT on Saturday March 28th as that was the times the team was able to verify transactions were successful. In actuality, the fix was applied at 3:29pm EDT and the first recorded successful transaction was at 5:59pm EDT. The team started to actively monitor the site to ensure that no more failures started Due to that, the resolution time will be update to reflect when the team began to monitor the fix which was at 4:32pm. In addition, this incident was originally classified as “major”. The classification has been adjusted to “minor” due to the limited number of customers affected and relatively low usage on this gateway.