Major Cloud Providers – Monthly Outage Recap March
MS Azure
3/29/19 RCA – SQL DatabaseBetween 16:45 and 22:05 UTC on 29 Mar 2019, a subset of customers may have experienced the following:
- Difficulties connecting to SQL Database resources in the East US, UK South, and West US 2 regions
- Difficulties connecting to Service Bus and Event Hubs resources in the East US and UK South regions
- Failures when attempting service management operations for App Service resources in the UK South and East US regions
- Failures when attempting service management operations for Azure IoT Hub resources
3/28/19 RCA – Data Lake Storage / Data Lake AnalyticsBetween 22:10 on 28 Mar 2019 and 03:23 UTC on 29 Mar 2019, a subset of customers using Data Lake Storage and/or Data Lake Analytics may have experienced impact in three regions:
- East US 2 experienced impact from 23:40 UTC on 28 Mar to 03:23 UTC on 29 Mar 2019.
- West Europe and Japan East experienced impact from 22:10 to 23:50 UTC on 28 Mar 2019.
Impact symptoms would have been the same for all regions:
- Customers using Azure Data Lake Storage may have experienced difficulties accessing Data Lake Storage accounts hosted in the region. In addition, data ingress or egress operations may have timed out or failed.
- Customers using Azure Data Lake Analytics may have seen U-SQL job failures.
3/27/19 RCA – Service Management Failures – West EuropeBetween approximately 15:20 UTC on 27 Mar 2019 and 17:30 UTC on 28 Mar 2019, a subset of customers may have received failure notifications when performing service management operations such as create, update, deploy, scale, and delete for resources hosted in the West Europe region.
AWS
3/29/19 Amazon CloudFrontBetween 8:50 AM and 12:59 PM PDT, we experienced longer than usual propagation times for changes to CloudFront configurations.
3/7/19 DownDetector
Problems at Amazon Web ServicesAmazon Web Services is having issues since 10:48 AM EST. Most reported problems:
- EC2 (51%)
- Log-in (38%)
- S3 (9%)
Other Platforms
3/29/19 IBM Cloud Platform Issues with accessing cloud foundry applications via console Users were experiencing issues with provisioning/restaging cloud foundry application and logging in via console. Running applications also impacted. START TIME March 29, 2019 7:15 PM PDT
END TIME March 29, 2019 8:01 PM PDT
3/26/19 SAP Cloud Platform Europe (Amsterdam) [neo-eu3] – Service Advisory Extensions for SAP SuccessFactors cannot be managed. Since approximately 13:40 UTC to 14:17 UTC.
3/25/19
-SAP Cloud Platform Japan (Tokyo) [cf-jp10] – Service Advisory Lifecycle management operations cannot be executed. Since approximately 07:07 UTC to 08:59 UTC.
-IBM Cloud Platform IAM adopters seeing sporadic 404s from identity service Customers may see intermittent errors from services that rely on IAM for authentication/authorization. Washington DC
START TIME March 25, 2019 7:00 AM PDT
END TIME March 25, 2019 4:20 PM PDT
3/24/19
–IBM Cloud Public – US-South: Staging and restaging of applications are failing. Dallas
– Staging and restaging applications failing
– Running applications are not impacted.
START TIMEMarch 24, 2019 2:46 PM PDT
END TIMEMarch 24, 2019 6:03 PM PDT
–IBM Cloud Public – United Kingdom: Applications may fail to stage and restage London
START TIME March 24, 2019 2:00 AM PDT
END TIME March 24, 2019 8:45 AM PDT
3/22/19 IBM Cloud Platform Customers cannot access running applications accessing the Cloud Foundry service. Customers may experience difficulties in accessing the Cloud Foundry service and Cloud Foundry applications. Dallas.
START TIME March 22, 2019 3:39 PM PDT
END TIME March 22, 2019 5:18 PM PDT
3/21/19
-SAP Cloud Platform Japan (Tokyo) [cf-jp10] – Service Advisory Customers may have experienced a disruption on the SAP Cloud Platform. Since approximately 06:29 UTC until 10:07 UTC
–VMware Cloud Assembly – Provisioning failure Users may encounter Internal Server Error when navigating through the User Interface.
Start Time: March 21, 2019 01:38 AM UTC
End Time: March 21, 2019 07:10 AM UTC
– IBM Cloud Platform Cloud Foundry users experienced problems authenticating to the service. Cloud Foundry users experienced problems authenticating to the service. Running applications not impacted. Sydney Frankfurt London.
START TIME March 21, 2019 8:25 AM PDT
END TIME March 21, 2019 9:25 AM PDT
– IBM Cloud Platform Service provisioning failed on Sydney region BSS Provisioning. Sydney.
START TIME March 21, 2019 4:08 AM PDT
END TIME March 21, 2019 6:45 AM PDT
3/20/19 SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory SAP Cloud Platform Internet of Things Service for the Cloud Foundry Environment is not working – Customers cannot maintain IoT scenarios. This includes CRUD operations for metadata such as devices, sensors, measures, or user management. – Customer’s devices cannot ingest data in the cloud or push data from the cloud to the devices. Since approximately 00:59 UTC on 20 Mar 2019 until 06:57 UTC
3/19/19 IBM Cloud Platform Timeouts and fails for logins for IBMid Intermittent service interruption and response time delays in logging in to Cloud Console with IBM ID.Dallas Sydney London Frankfurt Washington DC
START TIME March 19, 2019 6:30 AM PDT
LAST UPDATE March 19, 2019 8:48 AM PDT
3/16/19 VMware Cloud Services Console Availability issue User may not be able to access or experience trouble when logging into the the VMware Cloud Services.
Start Time: March 16, 2019 12:25 AM UTC
End Time: March 16, 2019 12:28 AM UTC
3/15/19 VMware Skyline Service Performance Degradation issue
Start Time: March 15, 2019 08:40 PM UTC
End Time: March 15, 2019 09:22 PM UTC
3/14/19
-SAP Cloud Platform Europe (Rot) [cf-eu1] – Service Advisory Applications protected by Authorization & Trust Management (XSUAA) are not accessible since approximately 09:42 UTC end time 10:45 UTC
-IBM Cloud Platform Issues accessing Cloud Foundry applications and services Frankfurt Sydney
START TIME March 14, 2019 6:22 AM PDT
END TIME March 14, 2019 9:30 AM PDT
3/13/19 SAP Cloud Platform US East (Ashburn) [neo-us1] – Service Advisory Applications and services are unavailable since approximately 06:25 UTC end time 06:36 UTC
3/12/19
-Google Cloud Storage Incident #19002 Incident began at 18:40 and ended at 22:50 (US/Pacific). On Tuesday 12 March, to reduce resource usage, SREs made a configuration change which had a side effect of overloading a key part of the system for looking up the location of blob data. The increased load eventually lead to a cascading failure. lasting 4 hours 9 minutes
–VMware Cloud Services – Backend Service Intermittent Availability Issue Users are able to login to the consoles of our VMware Cloud Services
Start Time: March 12, 2019 02:45 PM UTC
End Time: March 12, 2019 03:45 PM UTC
3/11/19
-Google Cloud Dataflow Incident #19001 Incident began at 2019-03-11 10:33 and ended at 2019-03-12 06:13 (US/Pacific). We’ve received a report of an issue with increased system lag in some Google Cloud Dataflow. lasting 19 hours 39 minutes
-SAP Cloud Platform US West (Chandler) [neo-us2] Service Advisory Lifecycle management operations for Java applications cannot be executed since approximately 19:39 UTC on 11 Mar 2019 end time 01:58 UTC on 12 Mar 2019
-Google Cloud Functions Incident #19002 Incident began at 15:02 and ended at 16:19 (US/Pacific). We’ve received a report of an issue with Google Cloud Functions deployments seeing increased errors. lasting 1 hour 16 minutes
-Google Cloud Console Incident #19001 Incident began at 09:58 and ended at 16:31 (US/Pacific). Affected users may receive a “failed to load” error message when attempting to list resources like Compute Engine instances, billing accounts, GKE clusters, and Google Cloud Functions quotas. lasting 6 hours 33 minutes
–VMware Skyline Service Performance Degradation issue User may experience slow or unresponsive service. Start Time: March 11, 2019 07:30 AM UTC End Time: 08:30 AM UTC
–VMware Cloud Services – Backend Service Intermittent Availability Issue Activation or Deactivation of Multi-Factor Authentication is impacted intermittently. Start Time: March 11, 2019 06:15 AM UTC End Time: 08:30 AM UTC
3/9/19
–VMware Skyline Service Intermittent Availability Issue Start Time: March 09, 2019 09:35 AM UTC End Time: 10:05 AM UTC
–VMware Skyline Service Intermittent Availability Issue Start Time: March09, 2019 06:25 AM UTC
End Time: 07:35 AM UTC
–SAP Cloud Platform Europe (Netherlands) [cf-eu20] – Service Advisory Lifecycle management operations cannot be executed since approx 06:11 UTC end time 08:19 UTC
3/7/19
-Cloud Machine Learning Incident #19001 Incident began at 2019-03-07 20:30 and ended at 2019-03-08 04:02 (US/Pacific). We are investigating an issue with Google Cloud Dialogflow – customers will experience 502 error messages. lasting 8 hours 36 minutes
-SAP Cloud Platform Europe (Rot) [neo-eu1] – Service Advisory Customers might experience issues with authentication of technical users since approximately 08:45 UTC end time 11:45 UTC
-Google Kubernetes Engine Incident #19005 Incident began at 05:49 and ended at 07:24 (US/Pacific). Current data indicate that all GKE API requests in region europe-west4 are failing. lasting 1 hour 34 minutes
3/6/19
-Google Cloud Networking Incident #19005 issue with Cloud Routers in us-east4. Began at 2019-03-06 23:37 ended at 2019-03-07 08:14 (US/Pacific) duration of 8 hours and 34 minutes.
–VMware Skyline Service Intermittent Availability Issue Users may not be able to login to VMware Skyline Advisor via my.vmware.com.
Start Time: March 06, 2019 02:05 PM UTC
End Time: 02:37 PM UTC
–VMware Cloud Services Availability Issue Users may not be able to access, or experience trouble when logging into VMware Cloud Services.
Start Time: March 06, 2019 01:55 PM UTC
End Time: 01:59 PM UTC
3/5/19
-IBM Cloud Platform Public_London: Issues with Cloud Foundry application provision Customers in London are experiencing issues when they try to provision app and use cf ssh due to authentication issue.
START TIME March 5, 2019 4:37 PM PDT
END TIME March 5, 2019 6:49 PM PDT
–Google Cloud Networking Incident #19004 Instance connectivity issues in us-west1, us-central1, asia-east1 and europe-west1. Began at 2019-03-05 10:38 ended at 14:52 (US/Pacific) lasting 4 hours 13 minutes
-IBM Cloud Platform multiple – environments seeing DAL05 Swift connectivity latency causing staging and restaging requests to intermittently fail Dallas
START TIME March 5, 2019 9:04 AM PDT
END TIME March 5, 2019 12:40 PM PDT
3/4/19
-SAP Cloud Platform Europe (Netherlands) [cf-eu20] – Service Advisory Lifecycle management operations cannot be executed since approximately 12:50 UTC end time 13:20 UTC
– SAP Cloud Platform US East (VA) [cf-us10] – Service Advisory A network issue is causing intermittent access issues for applications and services on both US East (VA) & Europe (Frankfurt). Since approximately 14:47 UTC end time 18:22 UTC
3/3/19
–VMware Skyline Service Availability issue Users may not be able to login to VMware Skyline Advisor via My.vmware.com.
Start Time: March 03, 2019 04:25 AM UTC
End Time: March 04, 2019 01:01 AM UTC
–VMware Cloud backend service Availability issue User may not be able to access or experience trouble when logging into the the VMware Cloud Services.
Start Time: March 03, 2019 04:03 AM UTC
End Time: March 03, 2019 04:15 AM UTC
3/2/19 SAP Cloud Platform China (Shanghai) [neo-cn1] – Service Advisory Applications and services are unavailable since approximately 19:00 UTC on 02 Mar 2019 end time 11:24 UTC on 03 Mar 2019.
3/1/19
-SAP Cloud Platform Europe (Frankfurt) [cf-eu10] – Service Advisory Applications and services are unavailable, Since approximately 00:35 UTC end time 02:21 UTC
–VMware Cloud backend service Availability issue Users may not be able to access or experience trouble when logging into the VMware Cloud Services.
Start Time: 08:00 AM UTC
End Time:04 AM UTC
-IBM Cloud Platform Customers may not be able to set classic permissions in IAM UI – London Sydney Washington DC Frankfurt Dallas
START TIME March 1, 2019 12:07 PM PDT
END TIME March 1, 2019 1:20 PM PDT
-IBM Cloud Platform Application staging may fail Bluemix Cloud Foundry – Frankfurt –
START TIME March 1, 2019 12:37 AM PDT
END TIME 1:43 AM PDT
