6 July 2022

Root Cause Analysis – Automation Cloud Outage on June 21 2022

6 July 2022

Root Cause Analysis – Automation Cloud Outage on June 21 2022

⬅  SEE ALL THE ARTICLES

 

Customer Impact 

 

On June 21, 2022, from 6:27 UTC until 7:42 UTC, almost all users received errors when trying to access any UiPath cloud product. While some users in some regions were still able to get through, we are treating it as a total outage.

 

Root Cause

 

UiPath relies heavily upon Cloudflare. In addition to DNS, we use it to proxy all of our traffic to cloud.uipath.com and related domains. This allows us to take advantage of Cloudflare’s DDOS protection, Web Application Firewall, caching, and more.

 

At the same times listed above, Cloudflare had a major outage in 19 of their data centers. These data centers handle the vast majority of all UiPath traffic, and as a result we were unavailable.

 

Cloudflare has posted an excellent deep dive into exactly what happened here.  

 

Detection & Response

 

Our automated alerting system notified on call engineers about the problem within minutes of the outage. They could quickly identify the root cause and post it to status.uipath.com. Unfortunately, due to the nature of our dependency on Cloudflare, there was no action they could take to mitigate the problem. They saw that Cloudflare had already identified the problem and were working on a fix, so they waited until that fix was available. At that time, they confirmed that all UiPath services became available again with no further problems.

 

Follow Up

 

UiPath selected Cloudflare, in part, due to its very high reliability with an SLA of 100%. We are currently working with engineers at Cloudflare to see if there is anything more we can do to protect ourselves from these sorts of outages. For the time being, taking a dependency on industry-standard services such as Cloudflare is still better than any alternative.

 

 

Kevin Schmidt is Senior Director, Site Reliability Engineering


by Kevin Schmidt

Show sidebar