tag:status.kustomer.com,2005:/historyKustomer Status - Incident History2024-03-18T19:33:23-04:00Kustomertag:status.kustomer.com,2005:Incident/202363252024-03-13T18:11:31-04:002024-03-13T18:11:31-04:00Customer Assist AI Currently Not Available Prod1<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>18:11</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Customer Assist AI Response and Agent Suggested Response features on Prod1.<br />After careful monitoring, our team has determined that all affected areas are now fully restored. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>16:11</var> EDT</small><br><strong>Update</strong> - After careful monitoring, our team has determined that the AI service is now fully functional, and our engineering team is redriving articles for a small number of customers. During this redriving period some recently updated articles may not appear in AI responses. Please reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>15:27</var> EDT</small><br><strong>Update</strong> - Kustomer is continuing work on addressing the event affecting Customer Assist AI Response and Agent Suggested Responses features on Prod1 that caused those features to not be usable.<br /><br />The issue has stabilized and our team is currently working to backfill the affected articles used for responses to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:55</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has implemented an update to address an event affecting Customer Assist AI Response and Agent Suggested Responses features on Prod1 that caused those features to not be usable.<br /><br />Our team is currently monitoring this update to ensure the issue is fully resolved. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>14:37</var> EDT</small><br><strong>Identified</strong> - Kustomer is aware of an event affecting Customer Assist AI Response and Agent Suggested Responses features on Prod1. These features are not currently able to provide suggested answers. <br /><br />Our team has identified the issue and is working to implement a resolution. Please expect further updates within the next 30 minutes, and reach out to Kustomer support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/201679742024-03-05T14:30:23-05:002024-03-05T14:30:24-05:00Reported Third Party Event - Facebook/WhatsApp<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>14:30</var> EST</small><br><strong>Resolved</strong> - Kustomer has been notified that the third Party vendor Meta has resolved an event affecting Business Messaging services. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>11:17</var> EST</small><br><strong>Monitoring</strong> - Kustomer is aware of an event reported by third party vendor Meta affecting Business Messaging services within the platform. Our team will continue to monitor the status with Meta. To follow along a link to their status page is here: https://metastatus.com/. Please expect further updates within 3 hours and reach out to support if you have additional questions or concerns.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>11:16</var> EST</small><br><strong>Investigating</strong> - Kustomer is aware of an event reported by third party vendor Meta affecting Business Messaging services within the platform. Our team will continue to monitor the status with Meta. To follow along a link to their status page is here: https://metastatus.com/. Please expect further updates within 3 hours and reach out to support if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/201334622024-03-01T12:47:58-05:002024-03-01T12:47:59-05:00[FACEBOOK] Facebook Pages Not Showing for Connection in Instance Dropdown (PROD1 and PROD2)<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>12:47</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>12:23</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>11:04</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>10:22</var> EST</small><br><strong>Investigating</strong> - Kustomer is aware of an event affecting Facebook pages in the Facebook section of some client instances that may cause pages connected to an integrated account not appearing. Not all customers will be affected by this issue. Please expect further half-hourly updates and reach out to Support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/199417292024-02-08T16:45:32-05:002024-03-05T15:29:53-05:00[Conversational Assistants] Conversational Assistants Transferring Early on Prod 1<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>16:45</var> EST</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting conversational assistants on PROD 1 causing assistants to transfer early at higher rates. After careful monitoring, our team has found that all affected areas are operating at normal levels. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:44</var> EST</small><br><strong>Monitoring</strong> - Kustomer has implemented remediation measures to address error rates causing conversational assistants to transfer early. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 1 hour and please reach out to support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:12</var> EST</small><br><strong>Update</strong> - Kustomer's team is monitoring error rates and investigating issues with conversational assistants transferring early within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:10</var> EST</small><br><strong>Update</strong> - Kustomer's team is still investigating the issues that may cause conversational assistants to transfer early within the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>13:34</var> EST</small><br><strong>Investigating</strong> - Kustomer is aware of an event affecting Conversational Assistants in PROD 1 that may cause conversational assistants to transfer to agents before fully executing. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/196586572024-01-10T19:00:47-05:002024-01-10T19:00:47-05:00Planned Kustomer Maintenance - January 10th and 11th 6pm to 9pm ET<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>19:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>16:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 9</var>, <var data-var='time'>12:07</var> EST</small><br><strong>Scheduled</strong> - Kustomer will be performing planned maintenance to PROD1 and PROD2 platforms from 6pm - 9pm ET January 10th, and January 11th 2024. During this time, we anticipate no downtime for users of the platform. Please reach out to support at support@kustomer.com if you have any questions or concerns.</p>tag:status.kustomer.com,2005:Incident/196132972024-01-04T21:00:40-05:002024-01-04T21:00:40-05:00Planned Kustomer Maintenance - January 4th 6pm to 9pm ET<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>21:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>18:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>11:44</var> EST</small><br><strong>Scheduled</strong> - Kustomer's planned maintenance to PROD1 and PROD2 platforms will be continuing 6pm - 9pm ET today, January 4th. During this time, we anticipate no downtime for users of the platform. Please reach out to support at support@kustomer.com if you have any questions or concerns.</p>tag:status.kustomer.com,2005:Incident/194731342024-01-02T21:00:13-05:002024-01-02T21:00:13-05:00Planned Kustomer Maintenance - January 2nd and 3rd 6pm to 9pm ET<p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>21:00</var> EST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'> 2</var>, <var data-var='time'>18:00</var> EST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>14:09</var> EST</small><br><strong>Scheduled</strong> - Kustomer will be performing planned maintenance to PROD1 and PROD2 platforms from 6pm - 9pm ET January 2nd, and January 3rd 2024. During this time, we anticipate no downtime for users of the platform. Please reach out to support at support@kustomer.com if you have any questions or concerns.</p>tag:status.kustomer.com,2005:Incident/190581622023-11-08T11:57:54-05:002023-11-08T11:57:54-05:00Reported Third Party Event - OpenAI<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>11:57</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved by our vendor.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:44</var> EST</small><br><strong>Investigating</strong> - Kustomer is aware of an event reported by one of our third party vendors, OpenAI that may cause increased error rates when using Agent Co-Pilot features within the platform. To follow along a link to their status page is here: https://status.openai.com/. Our team will continue to monitor the status with our vendor. Please expect further updates within 2 hours and reach out to support if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/184919462023-09-13T13:08:21-04:002023-09-13T13:08:21-04:00[WhatsApp] Delays or errors in sending outbound WhatsApp Messages from Prod 1 and 2<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>13:08</var> EDT</small><br><strong>Resolved</strong> - All errors have been redriven. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>12:31</var> EDT</small><br><strong>Monitoring</strong> - Kustomer is monitoring an issue impacting the WhatsApp API and outbound sends from our platform. Errors will be redriven when delays and errors resolve.</p>tag:status.kustomer.com,2005:Incident/184797622023-09-12T10:07:38-04:002023-09-12T10:07:38-04:00[Workflows] Workflow latency causing latency on platform in prod1<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>10:07</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Workflows causing latency in the platform. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>09:43</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has identified and implemented a change to address an issue affecting Workflows causing latency in the platform. The latency has returned to normal levels. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>09:22</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate the issues related to latency in the platform. Please expect further updates within 30 minutes and reach out to support over chat or at support@kustomer.com if you have additional questions or concerns.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>08:50</var> EDT</small><br><strong>Investigating</strong> - Kustomer is investigating an event affecting workflows that may cause latency in the platform and delays in messages being ingested. Our team is currently working to identify the issue. Please expect further updates within 30 mintues and reach out to support over chat or at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/183095712023-08-30T08:56:10-04:002023-08-30T08:56:10-04:00[Instagram Comments] Instagram Comments Not Ingested In Prod 1 and Prod 2<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>08:56</var> EDT</small><br><strong>Resolved</strong> - After careful monitoring, our team confirms the issue with IG comments has been resolved. Please reach out to our Support team at support@kustomer.com if you have additional questions or concerns.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>02:36</var> EDT</small><br><strong>Update</strong> - We continue to await 3rd party resolution on Instagram comments not being ingested in the platform. Please reach out to support@kustomer.com if you have questions and expect further updates.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>17:53</var> EDT</small><br><strong>Identified</strong> - We have identified the problem and are currently awaiting 3rd party resolution. Please reach out to support@kustomer.com if you have questions, and expect further updates within 1 hour.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>16:28</var> EDT</small><br><strong>Update</strong> - We have identified the problem and are currently awaiting 3rd party resolution. Please reach out to support@kustomer.com if you have questions, and expect further updates within 1 hour.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>15:20</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue. For any questions please reach out to support@kustomer.com. Please expect further updates within 1 hour.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>13:59</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue. For any questions please reach out to support@kustomer.com. Please expect further updates within 1 hour.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>12:58</var> EDT</small><br><strong>Investigating</strong> - Kustomer has identified an event affecting Instagram comments on Prod 1 and 2 that may cause comments to not be ingested into the platform. Our team is currently working on identifying the issue. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/183085982023-08-29T11:46:02-04:002023-08-29T11:46:02-04:00Elevated Whatsapp errors - Prod 1 & Prod2<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>11:46</var> EDT</small><br><strong>Resolved</strong> - Kustomer has identified that the errors were a result of an issue with our third party vendor, Whatsapp. The elevated error rates have now resolved on their end and all Whatsapp channels are operational.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>11:36</var> EDT</small><br><strong>Identified</strong> - We're investigating an issue with elevated Whatsapp errors. Our team is currently working to identify the cause of this issue. Please expect further updates within 30 minutes and reach out to support via Chat if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/180326032023-08-02T12:04:23-04:002023-08-02T12:04:23-04:00Reported Third Party Event - Whatsapp<p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>12:04</var> EDT</small><br><strong>Resolved</strong> - Kustomer is now seeing Whatsapp messages successfully send and Meta has communicated their team has deployed a fix to the issue. While the issue is resolved, our engineering team will redrive messages that failed to send during this time. Please feel free to reach out to support@kustomer.com should you have any additional questions.</p><p><small>Aug <var data-var='date'> 2</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Monitoring</strong> - Kustomer is aware of an event reported by one of our third party vendors affecting Whatsapp that may cause messages to not send within the platform. Our team will continue to monitor the status with our vendor. Please expect further updates within 1 hour and reach out to support if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/179895872023-07-28T18:08:07-04:002023-07-28T18:08:07-04:00Searches Not Loading in Prod 1<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>18:08</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Searches in Prod 1 that was causing searches not to load within the platform. Searches should now be working properly.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>17:35</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has implemented an update to address an event affecting searches not loading in the platform. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>17:19</var> EDT</small><br><strong>Investigating</strong> - Kustomer is aware of an event affecting Searches in Prod 1 that may cause searches not to load within the platform. Our team is investigating the cause of this issue to work on resolving. Conversations and customers are being created in all orgs while searches are catching up. Please expect further updates within 30 minutes if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/179772022023-07-28T09:47:19-04:002023-08-04T12:18:12-04:00Timelines and API not returning data in Prod 1<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>09:47</var> EDT</small><br><strong>Resolved</strong> - While we are still investigating some minor issues, all access to the Kustomer platform has been restored. For any questions related to this incident, please reach out to Kustomer Support via Chat or at support@kustomer.com</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>08:38</var> EDT</small><br><strong>Update</strong> - Our engineers are continuing to ensure efficient cleanup efforts and are working closely to ensure precision in our execution.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>05:25</var> EDT</small><br><strong>Update</strong> - Data integrity checks and clean-up efforts are still continuing in the background. Please expect further updates and reach out to Support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>02:45</var> EDT</small><br><strong>Update</strong> - Issues related to timelines, searches, and other areas in the platform are now resolved. Data integrity checks and clean up efforts will continue in the background.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>23:18</var> EDT</small><br><strong>Update</strong> - The Kustomer team is continuing to resolve errors related to timelines, searches, and other areas of the platform, resulting from an extended event that impacted the platform. Affected data is being identified, grouped, checked against our database, restored and re-driven or reimported where needed.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>22:02</var> EDT</small><br><strong>Update</strong> - An issue affecting Kustomer's ability to receive and display data has been resolved, and is being monitored. Errors may still be encountered as our team works to reimport data affected during the incident. Please look for the next update within an hour.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>21:02</var> EDT</small><br><strong>Monitoring</strong> - All channels and systems have been restored to full functionality. As we monitor the this resolution, errors may still continue to present in various areas, including timelines, searches, and workflows. Our team is working to clean up these remaining residual issues as we reimport data affected during the incident. Please look for additional updates within the next hour.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>20:03</var> EDT</small><br><strong>Update</strong> - As recovery of channels is underway and eventually completes, we do expect some errors to appear in timelines, searches, and workflows. We expect those errors to resolve when the clean-up of data during the period of outage fully completes. That clean up will be prioritized once the backlog of data has been processed and the system maintains stability.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>19:06</var> EDT</small><br><strong>Update</strong> - Timelines and searches are loading as expected. Our team is starting to turn channels back on. Please expect further updates in 1 hour</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>18:07</var> EDT</small><br><strong>Update</strong> - Timelines and searches are loading more consistently. However, they are still in recovery. As systems scale we expect improvements. Please expect further updates within 1 hour.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>17:04</var> EDT</small><br><strong>Update</strong> - Timelines and searches are starting to load more consistently. Please expect further updates within 1 hour.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>16:00</var> EDT</small><br><strong>Update</strong> - Our team is currently scaling services to meet request traffic within the web app to resolve continued issues connecting to customer timelines and loading searches. Please expect further updates within 1 hour</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>15:01</var> EDT</small><br><strong>Update</strong> - Our team is currently working on restoring connectivity to timelines and re-indexing searches with the latest updates. Upon completion, we will restore channel connections. Please expect further updates within 1 hour.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>14:04</var> EDT</small><br><strong>Update</strong> - Engineering is currently taking measures to restore connectivity in prod1. Channels have been temporarily turned off to prioritize the recovery of the affected system. There is currently no ETA yet for recovery, and will provide one as soon as possible. Please expect further updates within 1 hour</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Update</strong> - Engineering is currently taking measures to restore connectivity in prod1. Channels have been temporarily turned off to prioritize the recovery of the affected system. There is currently no ETA yet for recovery but will provide one as soon as possible. Please expect further updates within 1 hour</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>13:11</var> EDT</small><br><strong>Update</strong> - The engineering team has identified the issue affecting the connectivity to the platform and are working towards recovery. Please expect further updates within 30 minutes and reach out to support@kustomer.com if you have any questions.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>12:40</var> EDT</small><br><strong>Update</strong> - Our team is continuing to work to resolve the issue with timelines not loading and the API not returning data. Please expect further updates within 30 minutes and reach out to support@kustomer.com if you have any questions.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>12:08</var> EDT</small><br><strong>Identified</strong> - Kustomer is aware of an event affecting Timelines and API that may cause errors when loading customer timelines within the platform. Our team is currently working to implement a resolution. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>11:53</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>27</var>, <var data-var='time'>11:44</var> EDT</small><br><strong>Investigating</strong> - Kustomer is investigating an event affecting Timelines and the API that may cause errors when loading the app in Prod1. Our team is investigating this issue. Please expect further updates within 30 Minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/179321802023-07-22T19:58:08-04:002023-07-22T19:58:08-04:00Third Party Vendor Event - Elastic Cloud<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>19:58</var> EDT</small><br><strong>Resolved</strong> - An incident affecting Search & Reporting in POD1 has been resolved. The third-party Elastic Cloud has implemented a fix on their end, and the Kustomer team has reindexed platform data where needed. Please reach out to Kustomer's support team with any questions or concerns.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>18:55</var> EDT</small><br><strong>Update</strong> - Kustomer is monitoring the ongoing recovery of an incident affecting Search & Reporting in POD1. A fix has been implemented by the third-party vendor Elastic Cloud, and Kustomer is reindexing platform data to stabilize the affected features as needed. Please look for the next update within 1 hour.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>18:14</var> EDT</small><br><strong>Update</strong> - Kustomer is monitoring an event with third-party vendor Elastic Cloud that may result in a degraded Search and Reporting experience in POD1. Elastic Cloud has reported that a fix has been identified and is being rolled out. Please look for the next update within 1 hour.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>15:01</var> EDT</small><br><strong>Update</strong> - Kustomer is monitoring an event with third-party vendor Elastic Cloud that may result in a degraded Search and Reporting experience in POD1. We are currently monitoring a fix implemented by Elastic Cloud to ensure there are no further issues.<br /><br />Elastic Cloud's Status Page: https://status.elastic.co</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>13:41</var> EDT</small><br><strong>Update</strong> - Kustomer is monitoring an event that may result in a degraded Search and Reporting experience in POD1. This event is the result of an issue with our third-party vendor Elastic Cloud, who is working to implement a resolution. Please expect our next update within 1 hour, and reach out to support if you have additional questions or concerns.<br /><br />Progress can also be followed on Elastic Cloud's Status Page: https://status.elastic.co</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>13:09</var> EDT</small><br><strong>Monitoring</strong> - Kustomer is aware of an event reported by one of our third party vendors affecting POD1 that may cause a degraded Search and Reporting experience in Kustomer. Our team will continue to monitor the status with our vendor. Please expect further updates within 30 minutes and reach out to support if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/179138772023-07-20T12:49:18-04:002023-07-20T12:49:18-04:00Knowledge Base Forms 502 Errors Prod 2<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>12:49</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Knowledge Base Forms in Prod 2. After monitoring, our team has found that affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>12:44</var> EDT</small><br><strong>Investigating</strong> - Kustomer is investigating an event affecting Knowledge Base Forms that may cause 502 errors for those forms. Our team is currently investigating this issue. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/179046742023-07-19T17:08:02-04:002023-07-19T17:08:02-04:00WhatsApp Conversation Failures Prod 1<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>17:08</var> EDT</small><br><strong>Resolved</strong> - Kustomer is redriving all messages to WhatsApp shortly. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>16:54</var> EDT</small><br><strong>Identified</strong> - Kustomer is aware of an event affecting WhatsApp that may cause issues with messages being received and sent within the platform. All messages to be sent to WhatsApp will be queued and sent once the Meta WhatsApp API becomes available. To follow along with updates we recommend viewing the status page here: https://metastatus.com/whatsapp-business-api. We are following along as well.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>16:32</var> EDT</small><br><strong>Investigating</strong> - Kustomer is investigating an event affecting WhatsApp Conversations on Prod 1 and Prod 2 that may cause WhatsApp messages to not be created or fail to send from within the platform. Please expect further updates within 30 Minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/178914052023-07-18T13:39:14-04:002023-08-18T12:55:45-04:00[Platform] - Issues Loading Platform<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>13:39</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting platform access. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>13:19</var> EDT</small><br><strong>Investigating</strong> - Kustomer is aware of an event affecting access to the platform. Our team is currently working to identify the cause of this issue in an effort to implement a resolution. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/178361672023-07-12T16:27:21-04:002023-07-12T16:27:21-04:00[Reporting] - Delays in Queue and User Work Session Report Data - Prod 1 and 2<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>16:27</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Reporting in Prod 1 and 2. After careful monitoring, our team has found that affected areas are restored. The final updates of data to catch the reports up to date are in progress. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>13:16</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has implemented an update to address an event affecting Reporting in Prods 1 and 2. This issue caused delays in reports related to Queues and User work-sessions from updating with the latest data. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 4 hours and reach out to support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/178125092023-07-10T10:03:13-04:002023-07-10T10:03:14-04:00Issues with Reporting and Searches (PROD2)<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>10:03</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event affecting Search and Reporting. Please reach out to the Kustomer Support team if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>09:09</var> EDT</small><br><strong>Update</strong> - Our team has found that all affected areas are stable. We are continuing to monitor the fixes put in place to ensure continued stability with Search and Reporting in Prod2.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>07:38</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has implemented an update to address the degraded performance to Searches and Reporting in Prod2. Our team is continuing to monitor the fix we've implemented to ensure that the issue is fully resolved. Please expect further updates within 30 minutes and reach out to the Kustomer Support team if you have any additional questions or concerns.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>06:47</var> EDT</small><br><strong>Update</strong> - We continue to investigate the event affecting Reporting and Searches in PROD2 instances causing search delays and errors when accessing reports. Please expect further updates and reach out to Support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>05:47</var> EDT</small><br><strong>Investigating</strong> - Kustomer has identified an event affecting Reporting and Searches in PROD2 instances that may cause search delays and errors when accessing reports. Our team is currently working to design and implement a resolution. Please expect further updates in the next hour and reach out to Support at support@kustomer.com if you have additional questions or concerns.</p>tag:status.kustomer.com,2005:Incident/175638842023-06-13T17:56:05-04:002023-06-16T17:45:26-04:00Platform Latency in Prod1<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:56</var> EDT</small><br><strong>Resolved</strong> - Kustomer has resolved an event causing platform latency and searches not updating in the platform. After careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>17:03</var> EDT</small><br><strong>Monitoring</strong> - Kustomer has received an update that addresses the event affecting searches not updating within the platform. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:27</var> EDT</small><br><strong>Identified</strong> - Kustomer is aware of an event that may cause searches to not update within the platform. Our team is currently working to resolve this issue in an effort to implement a resolution. Please expect further updates within 1 hour and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:55</var> EDT</small><br><strong>Investigating</strong> - Kustomer has identified an event that may cause Platform Latency across prod1. Our team is currently working to design and implement a resolution. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns. This appears to be related to degradation in performance in AWS https://health.aws.amazon.com/health/status</p>tag:status.kustomer.com,2005:Incident/175274952023-06-09T20:42:50-04:002023-06-16T17:00:34-04:00Reported Third Party Event - ElasticCloud<p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>20:42</var> EDT</small><br><strong>Resolved</strong> - Kustomer has received an update from our third party vender ElasticCloud which has rolled out a fix to the issue that has been affecting searches and reporting in the platform. The team is seeing the platform recovered though it is worth noting that larger searches will take longer to reindex. Duplicating a search should allow you to see proper results immediately. Please reach out to support if you have any further questions.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>20:13</var> EDT</small><br><strong>Update</strong> - Kustomer has received updates from our third party vender ElasticCloud which has identified the issue impacting their service and they are currently working toward a resolution. Please expect further updates within 30 minutes.<br />Link to ElasticCloud Status Page: https://status.elastic.co/</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>19:37</var> EDT</small><br><strong>Monitoring</strong> - Kustomer is aware of an event reported by one of our third party vendors affecting POD1 that is causing Kustomer Searches to not return results within the platform and also causing issues with returning results for reporting. Our team will continue to monitor the status with our vendor. Please expect further updates within 30 minutes and reach out to support if you have additional questions or concerns.<br />Link to ElasticCloud Status Page: https://status.elastic.co/</p>tag:status.kustomer.com,2005:Incident/172194802023-05-11T13:04:05-04:002023-05-11T13:27:18-04:00Delay in processing Whatsapp API - Prod 1 & Prod 2<p><small>May <var data-var='date'>11</var>, <var data-var='time'>13:04</var> EDT</small><br><strong>Resolved</strong> - Whatsapp has reported that this issue is now resolved and that all outgoing message traffic is fully recovered. Our engineers are working to redrive Whatsapp messages to ensure successful delivery for Kustomer. Please reach out to support at support@kustomer.com if you have additional questions or concerns.<br /><br />Update: All messages have been redriven</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>12:15</var> EDT</small><br><strong>Identified</strong> - Kustomer is aware of an event affecting WhatsApp across Prod 1 that may cause delays in sending and receiving WhatsApp Messages within the platform. Most of these errors are still in our Prod1 environment. However, we are seeing a few errors in our Prod2 environment as well. Please expect further updates within 4 hours and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>11:14</var> EDT</small><br><strong>Investigating</strong> - Kustomer is aware of an event reported by Whatsapp that may cause a delay in receiving and sending Whatsapp messages. While most of these errors are in our Prod1 environment, we are seeing a few errors in our Prod2 environment as well. Our team will continue to monitor the status with Whatsapp and redrive events when the issue is resolved.<br /><br />Please refer to the issue by visiting the WhatsApp status page here: https://metastatus.com/whatsapp-business-api</p>tag:status.kustomer.com,2005:Incident/160187882023-02-03T14:36:04-05:002023-02-03T14:36:26-05:00Reported Third Party Event - Postmark<p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>14:36</var> EST</small><br><strong>Resolved</strong> - Our vendor has resolved their incident and after careful monitoring, our team has found that all affected areas are fully restored. Please reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>14:06</var> EST</small><br><strong>Monitoring</strong> - Kustomer has been made aware that our third-party vendor is no longer experiencing issues. Our team will continue to monitor this update to ensure the issue is fully resolved. Please expect further updates within 30 minutes and reach out to support at support@kustomer.com if you have additional questions or concerns.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>11:50</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue. Our team will continue to monitor the status with our vendor. Please expect further updates within 3 hours and reach out to support if you have additional questions or concerns. Additional information can be found here: https://status.postmarkapp.com/</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>11:11</var> EST</small><br><strong>Investigating</strong> - Kustomer is aware of an event reported by one of our third party vendors affecting Email on Prod1 and Prod 2 that may cause delays in outbound emails sent and delays in notifications within the platform. Our team will continue to monitor the status with our vendor. Please expect further updates within 1 hour and reach out to support if you have additional questions or concerns.</p>