Major Cloud Providers – Monthly Outage Recap November

MS Azure

11/20/19 RCA – Multiple Services – Downstream impact from Azure Front DoorBetween 00:56 and 03:40 UTC multiple services across Microsoft including Azure, Microsoft 365 and Microsoft Power Platform leveraging the Azure Front Door (AFD) service experienced availability issues resulting from high request failure rates. During this event, some impacted services were able to divert traffic away from the AFD service to mitigate impact for them.

11/19/19 DownDetector

Problems at Microsoft Azure Microsoft Azure is having issues since 8:33 PM EST. Most reported problems

  • 67% Cloud services
  • 28% Website hosting
  • 3% Virtual machines

11/7/19 RCA – Storage – West EuropeBetween 02:40 and 10:55 UTC a subset of customers using Storage in West Europe experienced service availability issues. In addition, resources with dependencies on the impacted storage scale units may have experienced downstream impact in the form of availability issues, connection failures, or high latency.

11/5/19

–RCA – Virtual Machines and Storage – West US 2Between 02:33 and 05:30 UTC, a subset of customers in West US 2 may have experienced service availability issues for resources hosted in this region due to an underlying Storage issue.

-RCA – Azure Databricks – Unable to connect to workspaces Between 14:19 and 15:34 UTC Azure Databricks may have intermittently experienced the error “This URL is temporarily unavailable. Please try again later.” when connecting to Databricks workspaces.

11/4/19 RCA – Azure SQL Database – West EuropeBetween 05:00 UTC and 11:30 UTC  you may have received failure notifications when performing service management operations – such as create, update, delete – for resources hosted in this region.

 AWS

11/22/19 VMware Cloud on AWS Service Availability issue User may not be able to access the service or may experience trouble while accessing the service. This incident affected: VMware Cloud on AWS (VMware Cloud on AWS, DRaaS, HCX).

Start Time: November 22, 2019 04:23 UTC
End Time: November 22, 2019 04:32 UTC

11/19/19 Degraded Performance for VMware Cloud Services Account creation for USA customer may fail for VMware Cloud services. This incident affected: VMware vRealize Automation Cloud (VMware Cloud Assembly, VMware Service Broker, VMware Code Stream), VMware Cloud on AWS (VMware Cloud on AWS), and VMware Cloud PKS, VMware Hybrid Cloud Extension (HCX), VMware Skyline, VMware Learning Platform, VMware Pulse IoT Center, VMware Secure State, VMware Cloud Marketplace.

Start Time: November 19, 2019 12:50 UTC
End Time: November 19, 2019 13:25 UTC

DownDetector

11/12/19

–Problems at Amazon Web Services Amazon Web Services is having issues since 12:36 PM EST.Most reported problems

  • 53% AWS Console
  • 30% EC2
  • 15% IAM

–VMware Cloud on AWS Service Performance Degradation issue Users in the Europe Central region might experience failures while performing Add host operation, Network and security tab might fail to load.This incident affected: VMware Cloud on AWS (VMware Cloud on AWS, DRaaS, HCX).

Start Time: November 12, 2019 09:43 UTC
End Time: November 12, 2019 12:10 UTC

DownDetector

11/8/19 Problems at Amazon Web Services Amazon Web Services is having issues since 6:00 PM EST. Most reported problems

  • 50% AWS Console
  • 40% EC2
  • 10% VPC

11/6/19 Change Propagation Delays Between November 5 9:52 PM and November 6 6:00 AM PST, we experienced intermittent delays in propagating CloudFront distribution changes to some edge locations.

11/3/19

–VMware Cloud on AWS: SDDC provisioning failures in US East (N. Virginia) region Users may experience issues when provisioning new SDDCs in US East (N. Virginia). This incident affected: VMware Cloud on AWS SDDC

Start Time: November 2, 2019, 23:43 UTC
End time: November 3, 2019, 06:28 UTC

Other Platforms

11/29/19 SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 10:45 UTC until 12:30 UTC, Business Rules cannot be created, viewed, or modified. Business Rules components are completely inaccessible

11/27/19 SAP Cloud Platform Europe (Amsterdam) [neo-eu3] – Service Advisory Since approximately 06:43 UTC until 07:11 UTC, Mobile Service for SAP Fiori is unavailable.

11/26/19

-SAP Cloud Platform US West (WA) [cf-us20] – Service Advisory Since approximately 18:20 UTC on 25 Nov 2019 until approximately 13:52 UTC on 26 Nov 2019, Audit log service broker is not able to handle any requests. Each customer that tries to create a new audit log service instance will not succeed. Everyone that already has an audit log instance will not be affected and they can write audit logs successfully.

11/25/19

-SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 08:15 UTC until approximately 10:49 UTC, Workflow Service is unavailable.

11/24/19 SAP Cloud Platform US East (VA) [cf-us10] – Service Advisory Since approximately 00:32 UTC until 01:13 UTC, Applications may be unavailable Intermittently and Lifecycle management operations cannot be executed Intermittently.

11/22/19

–VMware Cloud on AWS Service Availability issue User may not be able to access the service or may experience trouble while accessing the service. This incident affected: VMware Cloud on AWS (VMware Cloud on AWS, DRaaS, HCX).

Start Time: November 22, 2019 04:23 UTC
End Time: November 22, 2019 04:32 UTC

–VMware vRealize Log Insight Cloud outage and log availability issue During the outage, logs ingested into the product where not processed or persisted. Only logs collected during the outage was impacted. The impact was across all log types.

Start time: November 15, 2019 20:10 UTC
End time: November 15, 2019 23:00 UTC

-SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 19:30 UTC until approximately 20:42 UTC, Lifecycle management operations cannot be executed. Applications may be unavailable.

11/19/19 Degraded Performance for VMware Cloud Services Account creation for USA customer may fail for VMware Cloud services. This incident affected: VMware vRealize Automation Cloud (VMware Cloud Assembly, VMware Service Broker, VMware Code Stream), VMware Cloud on AWS (VMware Cloud on AWS), and VMware Cloud PKS, VMware Hybrid Cloud Extension (HCX), VMware Skyline, VMware Learning Platform, VMware Pulse IoT Center, VMware Secure State, VMware Cloud Marketplace.

Start Time: November 19, 2019 12:50 UTC
End Time: November 19, 2019 13:25 UTC

11/18/19 VMware Skyline Service Availability issue Users may experience issues when registering collectors and using log assist.

Start Time: November 18, 2019 15:48 UTC
End Time: November 19, 2019 04:30 UTC

11/15/19 SAP Cloud Platform Europe (Frankfurt) [neo-eu2] – Service Advisory Since approximately 02:23 UTC until approximately 02:37 UTC, Mobile Services for Development and Operations is unavailable. Europe (Frankfurt)(All Advisories [neo-eu2], Region Access [neo-eu2], Cloud Cockpit [neo-eu2], Lifecycle Management [neo-eu2], Web IDE [neo-eu2], Portal Service [neo-eu2], Mobile Services [neo-eu2]).

11/13/19 SAP Cloud Platform US East (VA) [cf-us10] – Service Advisory Since approximately 03:54 UTC approximately 06:00 UTC, Applications may be unavailable Intermittently and Lifecycle management operations cannot be executed Intermittently.

11/12/19

–VMware Cloud on AWS Service Performance Degradation issue Users in the Europe Central region might experience failures while performing Add host operation, Network and security tab might fail to load.This incident affected: VMware Cloud on AWS (VMware Cloud on AWS, DRaaS, HCX).

Start Time: November 12, 2019 09:43 UTC
End Time: November 12, 2019 12:10 UTC

-SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 07:40 UTC until approximately 13:00 UTC, A general disruption is impacting the availability of applications and services. This incident affected: Europe (Frankfurt) [cf-eu10] (All Advisories [cf-eu10], Lifecycle Management [cf-eu10], Portal Service [cf-eu10], Mobile Services).

11/11/19

–SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Since approximately 00:18 UTC on 09 Nov 2019 until approximately 16:07 UTC on 11 Nov 2019, Cloud Platform Cockpit users cannot browse or enable services. This issue occurs when new Subaccount creation fails and the behavior is intermittent. Enabling Cloud Foundry Organization in the existing Subaccounts or adding new members to the existing Subaccounts might also be impacted. Platform users might not be able to fetch tokens.

–Google Cloud Functions Incident #19009 We are experiencing an issue with multiple products.Incident began at 03:33 and ended at  04:23 (US/Pacific) lasting 50 minutes.

-Cloud Memorystore Incident #19003 We are experiencing an issue with multiple products. Incident began at 03:23 and ended at 04:27 (US/Pacific) lasting 1 hour 3 minutes.

–Google Cloud SQL Incident #19005 We’ve received a report of an issue with Cloud SQL. Incident began at 03:15 and ended at 04:45 (US/Pacific).

–Google Cloud Networking Incident #19022 We’ve received a report of an issue with Cloud Networking. Incident began at 02:40 and ended at 03:50 (US/Pacific) lasting 1 hour 10 minutes.

–Google Cloud Datastore Incident #19006 Google’s internal key management system (KMS), suffered a failure which began to cause user facing impact in the us-east1, us-east4, and southamerica-east1 regions at 02:39 US/Pacific and recovered by 03:27. Some services took longer to fully recover and continued to experience issues outside of this period. Incident began at 03:15 and ended at 05:36 (US/Pacific) lasting 2 hours 21 minutes.

–Google Cloud Storage Incident #19009 We are experiencing an issue with multiple products.Incident began at 03:02 and ended at 06:27 (US/Pacific) lasting 3 hours 25 minutes.

–Google Cloud Datastore Incident #19007 We’ve received a report of an issue with Cloud Firestore.Incident began at 02:09 and ended at 04:32 (US/Pacific) lasting 2 hours 22 minutes.

11/8/19

–VMware Cloud PKS Intermittent Availability issue Users may not be able to create/delete clusters in US-East-1 and US-West-2.

Start Time: November 08, 2019 16:55 UTC
End Time: November 08, 2019 18:10 UTC

-SAP Cloud Platform Europe (Rot) [neo-eu1] – Service Advisory Since approximately 10:10 UTC until approximately 10:29 UTC, OData Provisioning is unavailable.

11/6/19

–VMware Skyline Service Availability issue VMware Skyline Service is experiencing login issues.

Start Time: November 06, 2019 08:02 UTC
End Time: November 06, 2019 08:40 UTC

-SAP Cloud Platform Europe (Rot) [neo-eu1] – Service Advisory Since approximately 10:05 UTC until approximately 11:03 UTC, OData Provisioning is unavailable. Downtime would mean that the service is not available for all the customers using this service. The Fiori applications that use OData provisioning will be impacted as a result of the said downtime.

11/3/19

–VMware Cloud on AWS: SDDC provisioning failures in US East (N. Virginia) region Users may experience issues when provisioning new SDDCs in US East (N. Virginia). This incident affected: VMware Cloud on AWS SDDC

Start Time: November 2, 2019, 23:43 UTC
End time: November 3, 2019, 06:28 UTC

-SAP Cloud Platform Europe (Frankfurt) [neo-eu2] – Service Advisory Since approximately 00:45 UTC until 01:12 UTC, A general disruption is impacting the availability of applications and services. This incident affected: Europe (Frankfurt) [neo-eu2] (All Advisories [neo-eu2], Region Access, Cloud Cockpit [neo-eu2], Lifecycle Management [neo-eu2], Web IDE, Portal Service [neo-eu2], Mobile Services [neo-eu2]).

11/2/19 VMware Cloud PKS Intermittent Availability issue Users may not be able to create and delete cluster in US-East region. 

Start Time: November 02, 2019 07:01 UTC
End Time: November 02, 2019 10:25 UTC