Please find our full postmortem from the May 15, 2019 System Delay here.
Thank you for your understanding during this incident.
Posted May 16, 2019 - 18:19 EDT
Resolved
Update: Functionality of the Kustomer platform has been restored to normal behavior and throughput. Inbound and outbound messages that had previously not been sent or received have been recovered and delivered. At this time, CSAT for Kustomer chat is unavailable as we complete our investigation and post-mortem, which will be shared as soon as possible. For ongoing support or questions regarding this issue, please reach out to support@kustomer.com
Posted May 15, 2019 - 22:22 EDT
Update
Update: As of 3:40pm EST, the process for sending messages that were unable to be sent during the incident window has begun. Messages and notes created but not sent from 8:19am to 12:28pm EST will now be delivered, and can be expected to appear in your customer timelines. This will not occur all at once, and we will continue provide updates as the process continues and when it has been completed. This covers all channels: email, chat, SMS, Twitter, Facebook, and notes.
Posted May 15, 2019 - 15:48 EDT
Monitoring
As of 12:28pm, Kustomer’s throughput capacity has been restored to normal levels. From that point on, we do not expect any errors or delays to occur, and normal behavior should be expected for sending and receiving messages and the routing of conversations.
Our first priority is to ensure that all messages sent by your customers are received, and that all responses you sent to them during the time of this incident are delivered. It may take some time for all messages to be sent and received, but we are working to resolve this as quickly as possible.
We continue to monitor this situation, and are working to resolve all outstanding issues. For example, actions taken between 8:19am and 12:28pm EST may take some time to be reflected in search. We have identified the cause of this incident, and are working on a detailed report for all of our customers.
Posted May 15, 2019 - 14:21 EDT
Update
As of 12:28pm, Kustomer’s throughput capacity has been restored to normal levels. From that point on, we do not expect any errors or delays to occur, and normal behavior should be expected for sending and receiving messages and the routing of conversations.
Our first priority is to ensure that all messages sent by your customers are received, and that all responses you sent to them during the time of this incident are delivered. It may take some time for all messages to be sent and received, but we are working to resolve this as quickly as possible.
We continue to monitor this situation, and are working to resolve all outstanding issues. For example, actions taken between 8:19am and 12:28pm EST may take some time to be reflected in search. We have identified the cause of this incident, and are working on a detailed report for all of our customers.
Posted May 15, 2019 - 14:20 EDT
Update
Update: As of 12:25pm EST, our cloud service provider has restored our throughput to allow for the passing through of queued requests. This will substantially alleviate delays and all symptoms, but we continue to monitor the situation and work to improve all issues. Improvement to the incident has already been seen, and we are seeing a restoration of performance and decrease in errors.
Posted May 15, 2019 - 12:33 EDT
Update
Update: As of 11:00am EST, processing of events is still backed up and customers can expect delays across the platform. Areas most affected are:
- Chat - Queues and Routing - Workflows - Search
We are in direct contact with our cloud service provider to resolve this issue as swiftly as possible and will continue to share updates as they become available and progress is made.
Posted May 15, 2019 - 11:06 EDT
Update
Update: As of 8:40am EST, processing of events has begun returning to normal. While some delays may still be seen, customers should begin seeing normal behavior. Search indexing is still recovering, so search is still impacted at this time but is recovering. We continue to work actively to return all platform functions to normal.
Posted May 15, 2019 - 08:43 EDT
Update
Update: We have located the cause of the issue causing errors and delays on the Kustomer platform. Customers may continue to experience delays to routing, workflows, chat, and other platform features. We are actively working to deploy a fix that addresses the cause and will continue to post updates as they are released.
Posted May 15, 2019 - 08:25 EDT
Investigating
Kustomer is experiencing a system wide backup of requests, and all customers may encounter errors and/or delays for requests being made on the platform. We are actively investigating and will continue to post updates.