Chat messages failing
Incident Report for Kustomer
Postmortem

Summary

On May 1, 2020, beginning at 4:06 PM ET, Kustomer chat became partially unavailable for both Kustomer production instances. This outage persisted for 35 minutes until a rollback of the impacted code was triggered at 4:40 PM ET.

Impact

During the incident window, customers were unable to create new chat sessions. Customers attempting to do so were presented with an error in the Kustomer chat SDK upon attempting to start a new chat. Both the US and EU data centers were affected. The impact was on new chat sessions only, as customers were still able to send and receive messages for existing chat sessions, i.e. those that had been created before the start of the incident at 4:06 PM ET.

Root Cause

At 4:04 PM ET, a code change to our chat API was released to our production environments.

Trigger

Kustomer was alerted to the errors at 4:39 PM ET by reports of customers unable to begin a new chat session. These reports were immediately escalated to the engineering team.

Resolution

At 4:40 PM ET, the Kustomer engineering team rolled back the offending code in our production chat services. Once this rollback was complete, full functionality was restored to Kustomer chat.

Lessons Learned & Action Items

[IN PROGRESS] Expand alerting escalation policies to more readily surface system-critical errors in Kustomer chat API.

[IN PROGRESS] Increase automated end-to-end testing coverage to catch similar types of errors in lower environments before production release.

Posted May 05, 2020 - 11:57 EDT

Resolved
The issues with chat messages failing is now resolved. All systems are performing consistently and as expected.

If you have additional questions, please reach out to our Support team by using our contact form here: https://help.kustomer.com/contact/support-requests-HkXXApp6X.
Posted May 01, 2020 - 17:04 EDT
Monitoring
Our engineering team has identified and resolved the issue with chat messages failing. We are continuing to monitor the situation and will provide an update shortly.
Posted May 01, 2020 - 16:51 EDT
Investigating
Kustomer is currently experiencing an issues with inbound chat messages failing. We are working to resolve the issue as quickly as possible. During this time you may experience errors with customers being able to use the chat widget.

Please reach out to our Support team with any additional questions. You can reach us by using our contact form here: https://help.kustomer.com/contact/support-requests-HkXXApp6X.
Posted May 01, 2020 - 16:46 EDT
This incident affected: Prod2 (EU) (Channel - Chat) and Prod1 (US) (Channel - Chat).