Kustomer Loading Errors
Incident Report for Kustomer
Postmortem

Summary

On May 17th 2021 beginning at 11:29 AM EST until 11:37 AM EST users visiting the Kustomer platform were shown a “403 Forbidden / Access Denied” in their browser. After the configuration was reverted, users were intermittently unable to load parts of the Kustomer platform until 12:50 PM EST.

Impact / Alerts

Services impacted:

  • Kustomer Website as a whole

Root Cause

The root cause was due to a misconfigured security policy applied to the Kustomer Web platform. The misconfigured policy was initially applied at 11:27 AM EST. Health check notifications in test environments failed to reach the internal team resulting in the security policy being applied to production. The error was discovered on production at 11:29 AM EST through health check notifications. We immediately triggered the revert which completed at 11:37 AM EST. Full recovery was delayed due to our aggressive caching policies which cached the error response. At 12:43 PM EST, we issued invalidation for all the caches involved, leading to full recovery by 12:50 PM EST.

Resolution

Revert the misconfigured security policy and invalidate affected caches.

Lessons/Improvements

  • [COMPLETE] Fix health checks notification for test environments
  • [TODO] Optimize caching policies to exclude caching error responses
Posted May 18, 2021 - 15:28 EDT

Resolved
At this time the Kustomer platform is operating normally. You might have experienced residual effects related to the 403 error, that may have prevented parts of the Kustomer platform from loading correctly.

The Kustomer team released an update to purge cache so all clients can see the most current version of the Kustomer platform. We are no longer seeing outages related to this error.

Please reach out to our Support team with any additional questions or if you are still experiencing this issue. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.
Posted May 17, 2021 - 13:28 EDT
Update
Kustomer clients might be experiencing issues loading parts of the Kustomer platform. We are working to resolve the issue as quickly as possible. During this time you may experience issues loading different areas of the Kustomer Platform and seeing page crashing such as Settings.

The Kustomer team is releasing an update to purge cache so all clients can see the most current version of the Kustomer platform. We are still actively monitoring the situation.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.
Posted May 17, 2021 - 12:41 EDT
Monitoring
Kustomer is currently experiencing issues with loading areas of Kustomer Platform. We are working to resolve the issue as quickly as possible. During this time you may experience issues loading different areas of the Kustomer Platform such as Search and Settings.

Please reach out to our Support team with any additional questions. You can reach us by going to https://help.kustomer.com/ and clicking "Contact Support" at the top of the page.
Posted May 17, 2021 - 12:20 EDT
This incident affected: Prod1 (US) (Web Client) and Prod2 (EU) (Web Client).