Postmark Email Delays
Incident Report for Kustomer
Postmortem

Postmark has shared the following Postmortem to the email delays that occurred on February 25th, 2020 & February 27th, 2020:

Last week, the Postmark APIs became sporadically unavailable. The root cause of this was due to saturation of some of our underlying physical infrastructure. We have been in the process of migrating our core systems to a cloud provider in order to better respond to these types of spikes. This process is on-going, and the API has not yet been migrated.

To respond to the increased load to the Postmark API last week, we have done the following:

  • We doubled our API capacity.
  • We have shifted lower-priority workloads to our cloud provider to reduce saturation on our physical infrastructure.
  • We implemented additional traffic-shaping rules in our load balancers to isolate traffic spikes so that they cannot cause our API cluster to cascade into failure.

In addition, we have been planning to add to new public endpoints for our API. This work was already planned, but we are prioritizing. If your application restricts access to public IPs, please ensure that the following IPs are whitelisted:

  • 3.137.63.180
  • 18.216.134.80

We intend to start moving API traffic to these new IPs on March 16, 2020, so please make sure these IPs are allowed by any firewall or routing rules you may be using.

The complete status update can be found here: https://status.postmarkapp.com/incidents/92707

Posted Mar 04, 2020 - 16:44 EST

Resolved
The intermittent latency issue with the Postmark email platform has been resolved. Inbound and outbound emails are working as expected.

Please reach out to support via our web-form at help.kustomer.com and select the "Submit Issue" link with any questions.

Thank you for your patience.
Posted Feb 25, 2020 - 13:07 EST
Update
We are continuing to investigate this issue.
Posted Feb 25, 2020 - 12:47 EST
Investigating
The Postmark email platform is currently experiencing intermittent latency (inbound and outbound emails). We are researching to determine a root cause and will continue to monitor. We will share updates here on our Status Page.

Please reach out to support via our web-form at help.kustomer.com and select the "Submit Issue" link with any questions.
Posted Feb 25, 2020 - 12:40 EST
This incident affected: Prod1 (US) (Channel - Email, CSAT, Notifications) and Prod2 (EU) (Channel - Email, CSAT, Notifications).