On October 8, 2025, Kustomer experienced a period of degraded performance in our prod2 EU environment that affected business rules, workflow execution and routing responsiveness. The elevated load stemmed from an internal feedback loop in workflow processing, which caused increased memory usage and instability in a downstream automation component.
Systems serving automation traffic became rate-limited as they approached memory thresholds, resulting in delays across related services. The issue was mitigated by reducing processing throughput and rebalancing service traffic, allowing impacted systems to stabilize. Full recovery was confirmed later that day.
Customer Impact:
While existing safeguards are already in place, we are implementing additional safety measures to better detect anomalous workflow patterns and further enhance the stability of our automation systems under high-load scenarios. We’re also investing in expanded observability to improve early detection of similar issues in the future.