On May 17th 2021 beginning at 11:29 AM EST until 11:37 AM EST users visiting the Kustomer platform were shown a “403 Forbidden / Access Denied” in their browser. After the configuration was reverted, users were intermittently unable to load parts of the Kustomer platform until 12:50 PM EST.
Services impacted:
The root cause was due to a misconfigured security policy applied to the Kustomer Web platform. The misconfigured policy was initially applied at 11:27 AM EST. Health check notifications in test environments failed to reach the internal team resulting in the security policy being applied to production. The error was discovered on production at 11:29 AM EST through health check notifications. We immediately triggered the revert which completed at 11:37 AM EST. Full recovery was delayed due to our aggressive caching policies which cached the error response. At 12:43 PM EST, we issued invalidation for all the caches involved, leading to full recovery by 12:50 PM EST.
Revert the misconfigured security policy and invalidate affected caches.