On October 10, 2024, customers experienced intermittent failures when loading search and reporting features. Some users may have seen 503 Service Unavailable or 504 Gateway Timeout errors for search and report API requests.
Root Cause
Invalid search queries originating from one customer caused a high CPU load on some search nodes. This led to some searches displaying errors or timeout.
Timeline
Oct 10, 2024
11:44 AM EDT: Some users experienced intermittent “Service Unavailable” and “Gateway Timeout” errors upon viewing the dashboard, search, or reporting pages. Oncall engineers begin investigating.
12:54 PM EDT: Engineers identified a high CPU load on the search system and continued the investigation into the root cause. Mitigations to temporarily disable search were applied to reduce the load.
1:03 PM EDT: All search and reporting functionality returned to normal for majority of customers.
1:25 PM EDT: Temporary mitigations to disable search were removed. All search and reporting functionality was restored for all customers.
Lessons/Improvements
[IN PROGRESS] Migrate customers to new search hardware for improved performance
Posted Dec 30, 2024 - 11:27 EST
Resolved
Kustomer has resolved an event affecting Searches that may cause searches to be unavailable within the platform for some orgs.
After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.
Posted Oct 10, 2024 - 13:43 EDT
Monitoring
Kustomer has resolved an event affecting Searches that may cause searches to be unavailable within the platform for some orgs. The Behavior you may see are 503 error messages when accessing searches.
Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.
Posted Oct 10, 2024 - 13:16 EDT
Identified
Kustomer is aware of an event affecting Searches that may cause searches to be unavailable within the platform for some orgs. The Behavior you may see are 503 error messages when accessing searches.
Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect additional updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.