At 2:06 PM EST, an issue was discovered with search and reporting components that affected all instances on the Kustomer platform. A recently released update to query validation inadvertently allowed reports with unsafe time-range limits. These queries caused an overload of all elasticsearch client nodes, causing search and reporting to go down for Kustomer users.

At 2:18 PM EST, the client nodes were restored and search and reporting were once again accessible. Monitoring of the issue continued until 3:42 PM EST. During that time, individual client nodes failed 12 times and each time were brought back online within a minute. This caused intermittent errors, though search and reporting features were otherwise accessible during this time.

At 4:40 PM EST, an update was released that prevented unsafe queries, fully resolving the incident. We have confirmed there was no impact to Kustomer data.