JumpCloud Protect Degraded Service (Android Devices)

Incident Report for JumpCloud

Postmortem

JumpCloud Incident Report

Date: 2022-06-02

Date of Incident: 2022-05-24

Description: RCA for JumpCloud Protect Failing on Android 12 Devices

‌

Summary:

Starting at approximately 02:15 MST on 2022-05-24, users accessing JumpCloud Protect Push MFA version 1.15.0 on Android 12 devices would have experienced failures opening the application and would have been required to use TOTP if enabled. These failures lasted until approximately 08:40 MST on 2022-05-24.

‌

Root Cause:

The 1.15.0 version of JumpCloud Protect for Android exposed a bug in one of the androidx libraries specific to Android 12, causing the Protect application to fail on startup. Unfortunately, this conflict passed both our internal cross-platform mobile testing, as well as Google’s Play Store testing. The first attempt to roll back this version was successfully pushed to Google Play Store, but did not contain an incrementally positive number causing user devices to reject that version. The second deploy attempt did correctly iterate that version forward which ultimately fixed the issue.

Corrective Actions / Risk Mitigation:

Production deployments are temporarily paused until we have required changes and coverage in place.

Release a new version for Android removing the library conflict - DONE
Increase our physical testing coverage for all platforms - Target 06/2022
Expanded alerting around 3rd party store metrics - Target 06/2022
Improved rollback procedure across platforms - Target 06/2022

Posted May 31, 2022 - 16:18 MDT

Resolved

This issue is now considered resolved. An updated version of JumpCloud Protect for Android (v.1.15.1 ) is available for download in the Play Store. Users may need to manually update their JumpCloud Protect Android app if their version is not at v.1.15.1 or higher. Users who are still seeing issues with their JumpCloud Protect app for Android will want to manually force an update on the app.

To force an update for an Android app:
1. Open the Google Play Store app
2. Search for "JumpCloud Protect"
3. Select the JumpCloud Protect App
4. Tap Update for the JumpCloud Protect app.
https://support.google.com/googleplay/answer/113412?hl=en

If users see v1.14.0 OR v1.15.0 they should update to the most recent JumpCloud Protect Android app.

Posted May 24, 2022 - 11:08 MDT

Monitoring

A fix has been implemented and an updated version of JumpCloud Protect should be available for download. Users may need to manually update their JumpCloud Protect Android app if their version is not at v.1.15.1 or higher. Users who are still seeing issues with their JumpCloud Protect app for Android will want to manually force an update on the app.

To force an update for an Android app:
1. Open the Google Play Store app
2. Search for "JumpCloud Protect"
3. Select the JumpCloud Protect App
4. Tap Update for the JumpCloud Protect app.
https://support.google.com/googleplay/answer/113412?hl=en

We will continue to monitor the results of the fix.

Posted May 24, 2022 - 09:03 MDT

Update

Our team is working to get an updated version of the JumpCloud Protect app in the Google Play Store for Android users. Once completed, users will be able to update to a fully functional version. We will provide another update when the new version is available for Android users.

Posted May 24, 2022 - 08:01 MDT

Update

We are continuing to work on a fix for this issue.

Posted May 24, 2022 - 06:43 MDT

Identified

The issue has been identified and JumpCloud is working on implementing a fix.

Posted May 24, 2022 - 06:11 MDT

Update

We are continuing to investigate issues with the JumpCloud Protect app for Android devices The JumpCloud Protect app for iOS devices does not appear to be impacted.

Posted May 24, 2022 - 06:00 MDT

Investigating

We are currently investigating reports of issues with JumpCloud Protect for Android devices. We will report back as soon as we have an update.

Posted May 24, 2022 - 05:12 MDT

This incident affected: TOTP / MFA / JumpCloud Protect.