High Latency across Kustomer platform

Incident Report for Kustomer

Postmortem

Summary

Beginning at 00:02 Eastern Time on May 12, 2021, Kustomer engineering was alerted that there was high error rates and latency loading timeline data for clients in region us-east-1, due to database pressure from an abnormally large spike in requests resulting in a significant increase in database operations.

Impact/Alerts

There was general high latency and error rates as requests for timeline data timed out.

Root Cause

The database storing timeline data experienced a significant increase in operations which caused a rippling effect to other parts of the platform, resulting in high latency for all requests being processed in the us-east-1 region at that time..

Resolution

At approximately 00:35 ET, the spike in requests subsided and the database operations normalized.

Lessons/Improvements

[TODO] Review and tune throttling policies for database intensive actions

Posted May 17, 2021 - 02:46 EDT

Resolved

This issue has been resolved. The platform should be running normally now.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.

Posted May 12, 2021 - 00:51 EDT

Update

We are continuing to investigate this issue.

Posted May 12, 2021 - 00:25 EDT

Investigating

Kustomer is currently experiencing platform wide latency. We are working to resolve the issue as quickly as possible. During this time you may experience pages failing to load.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.

Posted May 12, 2021 - 00:25 EDT