Agents unable to respond to conversations 10/24/19

Summary

At approximately 7:46am EDT on October 24, 2019, several organizations reported that users were unable to respond to conversations in the Kustomer application, instead receiving warning errors that their role was unauthorized. These errors persisted until 8:18am EDT, at which point the Kustomer engineering team had identified the cause and resolved the issue.

Impact

During the time of the incident, approximately 6,378 requests across the Kustomer platform failed with 401 internal error. This only affected users with custom roles and permissions trying to respond to conversations.

Root Causes

A release of new code related to roles caused an internal role validation to run on every request.

Detection

The issue was detected when it was reported minutes after the release. The ensuing issue was addressed immediately and rollback of the code commenced shortly thereafter.

Resolution

The incident was completely resolved by the code rollback.

Lessons Learned & Action Items

Additional monitoring is being put in place to test for scenarios such as this one, in which application behavior is affected by the Kustomer API.