EU Vault API errors

Incident Report for Keeper Status Page

Postmortem

The Engineering team found and resolved the cause of the API errors which occurred on Jan 28, 2021, described in this postmortem report.

It was determined that a particular “SCIM PATCH” for Group association requests from the Microsoft Azure portal was returning an invalid response from the Keeper servers based on the number of users within the group. The size of the response grew exponentially in size based on the number of users within the group. This issue caused memory exceptions to occur on all affected application servers, which subsequently generated API errors for login requests.

We have corrected the SCIM PATCH response in our Backend release 15.2.1 which went live on Saturday, Jan 30, 2021.

The SCIM specification https://tools.ietf.org/html/rfc7644#section-3.5.2 allows returning 204 as a result of GROUP's patch, which can greatly improve performance for large groups.

Jira Ticket: KA-3782

Posted Feb 01, 2021 - 21:29 PST

Resolved

For approximately 5 minutes, from 2:25PM PST to 2:30PM PST there were increased API errors in the EU region. This issue was also related to the API errors on the previous night. The problem was identified as a bug that was a result of certain SCIM messages sent from Azure. The engineering team released a fix over the weekend and the issue is now resolved.

Posted Jan 29, 2021 - 14:00 PST