Update - As of 7:00 AM EST(4:00 AM PST) Amazon has restored all traffic to Kinesis data streams. Kustomers systems should feel operational. We are still monitoring all of our systems as we start to receive traffic from AWS.

We have engineers working over the holiday to ensure full recovery and operating capacity of the platform and surrounding services
Search was the main component that was impacted, but there can be other services that users might be experiencing a degraded performance during this recovery period such as data streams, i.e. Kinesis Data stream.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 26, 07:43 EST

Update - As of 7:00 AM EST(4:00 AM PST) Amazon has restored all traffic to Kinesis data streams. Kustomers systems should still feel operational. We are still monitoring all of our systems as we start to receive traffic from AWS.

Amazon CloudWatch metrics remain delayed until Kinesis throttles are fully lifted. We are able to confirm that data is flowing normally into Kustomers systems again but at a delayed and staggered rate as reflected by the AWS update. We added additional functionality to continuously update the search indexes and drive messages while the throttles are lifted by AWS.

We have engineers working over the holiday to ensure full recovery and operating capacity of the platform and surrounding services
Search was the main component that was impacted, but there can be other services that users might be experiencing a degraded performance during this recovery period such as data streams, i.e. Kinesis Data stream.

Search page indexing was temporarily halted from 5:58pm to 6:40pm EST due to an issue in a database process. The problem was resolved and indexing resumed immediately after. Users may still experience degraded performance in search pages while we’re working towards full recovery.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 26, 07:35 EST

Update - Amazon has provided an update that the errors within Kinesis has been fully mitigated but are only incrementally restoring traffic as to not overwhelm their systems. Amazon CloudWatch metrics remain delayed until Kinesis throttles are fully lifted. We are able to confirm that data is flowing normally into Kustomers systems again but at a delayed and staggered rate as reflected by the AWS update. We added additional functionality to continuously update the search indexes and drive messages while the throttles are lifted by AWS.

We have engineers working over the holiday to ensure full recovery and operating capacity of the platform and surrounding services
Search is the main component that has been impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, i.e. Kinesis Data stream.

Search page indexing was temporarily halted from 5:58pm to 6:40pm EST due to an issue in a database process. The problem was resolved and indexing resumed immediately after. Users may still experience degraded performance in search pages while we’re working towards full recovery.

We are monitoring the situation and will continue to share information here as it becomes available.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 23:13 EST

Update - Kustomer is observing steady signs of improvement of error rates for Kinesis from the current AWS outage and we are back to normal levels in our Kinesis event processing.

We will continue to work with Amazon Web Services to return our services to fully operational. Search is the main component that is being impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, i.e. Kinesis Data stream.

Search page indexing was temporarily halted from 5:58pm to 6:40pm EST due to an issue in a database process. The problem was resolved and indexing resumed immediately after. Users may still experience degraded performance in search pages while we’re working towards full recovery.

We are monitoring the situation and will continue to share information here as it becomes available.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 20:27 EST

Update - Kustomer is now driving all Gmail emails that might not have been created in Kustomer while disconnection occurred due to this AWS outage. We will continue to provide updates once this has been completed.

Kustomer has implemented internal updates to account for AWS outage issues we are seeing. Functionality of the Kustomer platform should feel operational but you may notice some degraded performance.

We have been informed that Amazon has identified the root cause of the Kinesis Data Streams API outage and is working towards a resolution. We are seeing an improvement in error rates for Kinesis, but per Amazon a full recovery for this service may still take up to a few hours.

Kustomer Search should feel operational but you may notice some degraded performance and up to a 5-minute delay during these AWS outages. We will continue to work with Amazon Web Services to return our services to fully operational. Search is the main component that is being impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, i.e. Kinesis Data stream.

We are monitoring the situation and will continue to share information here as it becomes available.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 15:36 EST

Update - Kustomer has implemented internal updates to account for AWS outage issues we are seeing. Functionality of the Kustomer platform should feel operational but you may notice some degraded performance.

We have been informed that Amazon has identified the root cause of the Kinesis Data Streams API outage and is working towards a resolution. We will be working with Amazon Web Services to continue work towards returning our services to be fully operational. Search is the main component that is being impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, i.e. Kinesis Data stream.

Kustomer was able to re-index the search pages every 30 seconds of the last 5 minutes of search activity. Kustomer Search should feel operational but you may notice some degraded performance during these AWS outages.

Kustomer is working to re-drive Gmail emails that might not have been created in Kustomer while disconnection occurred due to this AWS outage

We are monitoring the situation and will continue to share information here as it becomes available.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 14:24 EST

Update - The Kustomer team has identified the issue to be an outage with an AWS service. We will be working with Amazon Web Services to continue work towards returning our services to operational. Search is the main component that is being impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, i.e. Kinesis Data stream.

Gmail emails authentication could have been affected and will need to be re-authenticated. Kustomer was able to re-authenticate Gmail addresses that were disconnected. Kustomer is working to re-drive Gmail emails that might not have been created while disconnection occurred.

We are monitoring the situation and will continue to share information here as it becomes available. This is only affecting US servers.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 11:48 EST

Update - The Kustomer team has identified the issue to be an outage with an AWS service. We will be working with Amazon Web Services to continue work towards returning our services to operational. Search is the main component that is being impacted, but there can be other services that users might be experiencing a degraded performance with such as data streams, Kinesis Data stream, Gmail emails authentication could have been affected and will need to be re-authenticated. We are working internally to address this issue.

We are monitoring the situation and will continue to share information here as it becomes available. This is only affecting US servers.

Please review the Amazon status page for updates to their services as well. https://status.aws.amazon.com/
Nov 25, 11:02 EST