Summary

On July 18th, 2023 at 12:50 PM EDT, we received reports from a small subset of customers indicating that they were unable to access the Kustomer platform.  We identified our CDN provider, Amazon CloudFront as the source of these errors and began discussing remediation procedures.  By  1:17 EDT, we began to receive reports of recovery and at 1:26, Amazon posted an update on their status page acknowledging the issue and confirming that it had been resolved.  

Root Cause

Some users who were not already logged in to Kustomer  were unable to access Kustomer from 12:37 PM EDT till 1:13PM due to failures in AWS Cloudfront preventing them from loading our front end assets.

Timeline

July 18, 2023

12:50 PM EDT Kustomer’s support team began receiving reports of a portion of agents being unable to access the platform

12:58 PM EDT The issue was escalated to our on-call engineers, kicking off an investigation

1:10 PM EDT The issue was identified as originating from CloudFront and on-call engineers began planning for remediation

1:17 PM EDT Kustomer support received first reports of recovery from clients

1:26 PM EDT Amazon posted an update acknowledging the issue and confirming that it had been resolved 

Lessons/Improvements

• We are prioritizing additional improvements to our disaster recovery infrastructure to improve response time for temporarily transitioning traffic off of CloudFront to a self-hosted solution in the event of a future CDN issue