Summary

An organization’s automated process running a high volume of complex search queries strained our search system, causing reduced search functionality for multiple Prod2 organizations over the span of roughly 3 hours.

Root Cause

A high volume of complex queries in a short amount of time put excessive strain on our infrastructure. This negatively impacted the health of 2 nodes, affecting organizations that also rely on this same infrastructure.  

Timeline

Jan 12, 2025

6:00 AM - EST - A single organization’s automated user ramps up activity to significantly higher levels than normal

11:38 AM EST - Engineers start receiving alerts that various charts/reporting endpoints are generating errors

12:16 PM EST - Multiple clients report broken search functionality, we declare an incident

12:22 PM - Engineers identify that 2 of our nodes are reaching maximum CPU utilization 

12:33 PM EST - Engineers identify the organization, user, and query that are causing the issue

1:15 PM EST - Engineers temporarily blocked the organization generating the problematic queries, restoring functionality to all organizations except the blocked organization.

3:15 PM EST - Engineers disabled the block, full functionality is restored for all organizations

Lessons/Improvements

• Better communications - This incident highlighted the need for us to establish better protocol and communications with organizations whose activity may be impacting infrastructure or other organizations. With such a protocol in place we can hopefully reduce the need to block a single organization.
• Tune our blocking - In response to this incident, we implemented the ability to block a single user (rather than block an entire organization) that is disrupting our systems.
• Better alerts - By more finely tuning our alerting system, we can get earlier notification of incidents such as this without having to wait for customers to report impaired functionality.
• Rate limiting machine users - We are exploring a reasonable way to rate-limit activity by machine users so that no single machine user can overwhelm our systems, as seen in this incident.