Summary

On April 24, 2026, customers in our prod1 environment experienced a service event that caused elevated errors and latency in messaging-related workflows. The broad cross-customer impact was limited to approximately 41 minutes, from 3:26 PM ET to 4:07 PM ET. During that window, some customers saw failed or delayed messaging operations.

The immediate platform impact was resolved the same day, and overall system health returned to normal. We then completed follow-up mitigation to stop the underlying event source and reduce the risk of recurrence.

Impact

• Customers in prod1 experienced elevated API errors and latency in messaging-related workflows.
• The broad cross-customer impact lasted about 41 minutes.
• A subset of messaging workflows failed or were delayed during that period.

Timeline

• ~3:25 PM ET — A newly enabled automation began processing a large backlog of historical conversations for one tenant.
• 3:26 PM ET — Elevated errors and latency began affecting shared messaging workflows in prod1.
• 4:01 PM ET — We published a status update for the production issue.
• 4:07 PM ET — Broad cross-customer impact ended as the affected services stabilized.
• ~5:17 PM ET — We disabled the triggering automation configuration for the affected tenant.
• ~5:23 PM ET — The remaining retry activity stopped.

Root cause

The event was triggered when a newly enabled automation for one tenant processed a much larger set of eligible conversations than intended. That sudden volume overloaded a shared downstream service and caused elevated errors and timeouts in dependent workflows.

The incident was amplified by missing safeguards in how this automation handled backlog volume and retries. In particular, the system did not sufficiently limit the number of conversations processed at once or prevent the same failed work from being retried too aggressively.

Resolution

We restored platform stability during the incident by allowing the affected services to recover under increased capacity, then disabled the triggering automation configuration and cleared the remaining retry backlog. System health is currently normal.

Preventative actions

We are treating the following preventative actions as a priority bug effort. These actions are expected to be resolved by the end of May in accordance with our SLOs:

• Prevent newly enabled automation settings from processing large historical backlogs unintentionally.
• Add stronger batch limits and tenant-level throttling for this workflow.
• Reduce retry amplification by improving how failed work is tracked and re-queued.
• Improve error handling so rate-limit conditions are classified correctly and handled with the right retry behavior.