Date: 2022-04-07
Date of Incident: 2022-03-31
Description: RCA for JumpCloud Agent Dispatch Delay
Summary:
On 2022-03-31, some JumpCloud customers experienced delays in Agent dispatch times. These delays started around 08:00 UTC lasting until approximately 15:56 UTC. Similar delays with Agent dispatch also occurred in smaller windows on 2022-03-19, and 2022-03-30.
Root Cause:
In our efforts to improve the overall efficiency with Agent functionality and move towards IoT messaging, we ran into two issues with the deployment of these changes. First, the code to publish MQTT messages was missing a timeout parameter for long operations which slowed down some of the longer operations. Second, we suspect, and are still investigating possible throttling of these messages by our Messaging Service Provider(1). We have rolled back all changes while we work through the corrective actions.
Corrective Actions / Risk Mitigation:
1 We are still investigating quota, or rate limiting issues with this service imposed by our service provider. We will update this as we uncover those results.