Incident Report 

‌

Date: Sep 15, 2025

Date of Incident: Sep 16, 2025

Description: RCA for Admin Portal Access Failures

‌

Summary:

On September 15, 2025, between 08:59 UTC - 09:11 UTC customers experienced failures when attempting to access the Admin Portal.  Users and services attempting to authenticate received a loading error message during that window.

‌

Root Cause:

The incident originated during an update to support a new API for our SaaS Management Service. The update was designed to allow a new console subdomain to handle traffic, in addition to the existing SaaS subdomain.

‌

While this change successfully passed all testing in our pre-production environment, our automated deployment system failed to apply the full configuration in production. It applied the new console subdomain but, due to a misconfiguration, it omitted a critical path prefix. This resulted in an issue where traffic intended for a different service was routed incorrectly, causing a brief service outage.

‌

As soon as the errors appeared, the team immediately rolled back the change, and service was restored. We have identified the root cause in our automated deployment process and are implementing a fix to prevent this from happening again.