Summary

On August 29, 2025, a security policy update unintentionally restricted access to web assets, resulting in HTTP 403 errors for the site for 26 minutes. The API was unaffected. During this window, some users were unable to log in; users already logged in were generally unaffected. The update was then reverted and service was restored.

Root Cause

A security policy update inadvertently restricted access required for public web content. Reverting the update restored normal behavior.

Timeline (Eastern)

1:27 pm – Security policies updated in production.

1:28 pm – Monitoring detected elevated 403 errors.

1:34 pm – Confirmed impact limited to web; API unaffected.

1:53 pm – Update reverted;  recovery observed.

Lessons/Improvements

While there are safeguards and automated checks for access-related changes,  this incident exposed a narrow gap; to cover such edge cases, the following actions are being taken:

• [IN-PROGRESS] Broaden existing change-review rules for updates that could affect platform access.
• [IN-PROGRESS] Extending our automated policy validations and guardrails to detect and block these edge cases before rollout.