Search Unavailable
Incident Report for Kustomer
Postmortem

At 2:06 PM EST, an issue was discovered with search and reporting components that affected all instances on the Kustomer platform. A recently released update to query validation inadvertently allowed reports with unsafe time-range limits. These queries caused an overload of all elasticsearch client nodes, causing search and reporting to go down for Kustomer users.

At 2:18 PM EST, the client nodes were restored and search and reporting were once again accessible. Monitoring of the issue continued until 3:42 PM EST. During that time, individual client nodes failed 12 times and each time were brought back online within a minute. This caused intermittent errors, though search and reporting features were otherwise accessible during this time.

At 4:40 PM EST, an update was released that prevented unsafe queries, fully resolving the incident. We have confirmed there was no impact to Kustomer data.

Posted about 1 month ago. Mar 06, 2019 - 18:01 EST

Resolved
This incident has been resolved and search and reporting components are stable.
Posted about 1 month ago. Mar 06, 2019 - 17:31 EST
Monitoring
We have identified the issue with search and found reporting was also affected. A fix has been rolled out, and we are monitoring all systems currently.
Posted about 1 month ago. Mar 06, 2019 - 14:21 EST
Investigating
Search functionality is currently unavailable, and we are investigating the cause.
Posted about 1 month ago. Mar 06, 2019 - 14:14 EST
This incident affected: Search.