Is Google Cloud Down?

## Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your productivity. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using [https\://cloud.google.com/support](https://cloud.google.com/support) ## Date/Time of the Issue (All time US/Pacific) Incident Start: 08 October 2026 04:00 Incident End: 08 October 2026 16:30 Duration: 12 hours, 30 minutes ## Summary On Thursday, 08 October 2026, Google Cloud Storage experienced elevated write latency and write error rates in the us-central1 region for a duration of 12 hours, 30 minutes. We are taking immediate steps to ensure this doesn’t happen again. ## Preliminary Root Cause Google Cloud Storage relies on a highly distributed backend architecture to process and store incoming data. During the incident window, specific storage backend nodes within the us-central1 region experienced a localized capacity shortfall. The sequence of events leading to customer impact was as follows: * A traffic sorting optimization intended to address localized capacity constraints caused severe resource exhaustion on specific backend storage nodes in the affected region. * This resource exhaustion caused backend services handling object data ingestion to begin timing out. * As a result, incoming upload and write requests routed to these nodes experienced elevated latency and failed to complete, resulting in errors surfaced to customers. * Concurrently, a separate issue with the underlying database service contributed to additional metadata processing errors during a portion of the incident window. Google engineers have begun a full root cause analysis (RCA) and will provide additional information once it is available. ## Remediation The issue was initially detected by Google engineering teams actively monitoring system telemetry early in the incident window. To mitigate the issue, Google engineers executed the following step-by-step strategy: * Engineers initially attempted to dynamically allocate additional emergency capacity to the affected region, but this was blocked by systemic regional capacity limits. * To relieve immediate pressure, engineers selectively drained traffic from the most overloaded backend storage nodes. * A configuration change was deployed to disable the traffic sorting optimization, significantly reducing resource overhead on the storage backend. * For the concurrent metadata issue, engineers rolled back a recent update to the underlying metadata database service. Once traffic levels stabilized and resource utilization returned to normal baseline levels, the previously drained nodes were restored to active service, successfully restoring normal operations. ## Description of Impact On 08 October, 2026, from 04:00 to 16:30 US/Pacific, customers utilizing the us-central1 region experienced elevated latency and error rates during Cloud Storage upload and write operations. * Customers primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts. * A minor impact was also observed on read operations during this timeframe. * Cloud Storage operations in all other regions, as well as all other Google Cloud services, remained unaffected by this specific event. --- [2026-10-08T20:48:22+00:00] (SERVICE_DISRUPTION) **Summary** Google Cloud Storage customers are experiencing elevated latencies and error rates when performing upload or write operations **Description** We are continuing to mitigate the issue with Google Cloud Storage in the us-central1 region that began on Thursday, 2026-10-08 05:19 PDT. The initial mitigations applied by our engineering team continue to show observable improvements in error rates. Our engineering team is actively monitoring the environment and rolling out further mitigations to fully stabilize the service. We will provide an update by Thursday, 2026-10-08 16:00 PDT with current details. **Customer Symptoms** * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors **Workaround** None at this time. --- [2026-10-08T19:38:18+00:00] (SERVICE_DISRUPTION) **Summary** Google Cloud Storage customers are experiencing elevated latencies and error rates when performing upload or write operations **Description** We are continuing to mitigate the issue with Google Cloud Storage in the us-central1 region that began on Thursday, 2026-10-08 05:19 PDT. The initial mitigations applied by our engineering team continue to show observable improvements in error rates. Our engineering team is actively monitoring the environment and rolling out further mitigations to fully stabilize the service. We will provide an update by Thursday, 2026-10-08 14:00 PDT with current details. **Customer Symptoms** * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors **Workaround** None at this time. --- [2026-10-08T18:55:19+00:00] (SERVICE_DISRUPTION) **Summary** Google Cloud Storage customers are experiencing elevated latencies and error rates when performing upload or write operations **Description** We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Initial mitigations applied by our engineering team leading to observable improvements in error rates in our internal monitoring beginning 10:57 PDT. Work remains underway to roll out further mitigations. We will provide an update by Thursday, 2026-10-08 12:45 PDT with current details. **Customer Symptoms** * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors **Workaround** None at this time. --- [2026-10-08T18:34:00+00:00] (SERVICE_DISRUPTION) **Summary** Google Cloud Storage customers are experiencing elevated latencies and error rates when performing upload or write operations **Description** We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Initial mitigations applied by our engineering team leading to observable improvements in error rates in our internal monitoring. Work remains underway to roll out further mitigations. We will provide an update by Thursday, 2026-10-08 12:00 PDT with current details. **Customer Symptoms** * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors **Workaround** None at this time. --- [2026-10-08T18:24:18+00:00] (SERVICE_DISRUPTION) **Summary** Google Cloud Storage customers are experiencing elevated latencies and error rates when performing upload or write operations **Description** We are experiencing an issue with Google Cloud Storage beginning on Thursday, 2026-10-08 05:19 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-10-08 11:50 PDT with current details. **Customer Symptoms** * Customers are experiencing elevated latencies and error rates when performing upload or write operations. * A smaller subset of read operations may also experience increased latency and errors **Workaround** None at this time.

## Addendum to Incident Report This addendum extends the Detailed Description of Impact from the previous message. Though only two clusters in a datacenter in us-central1-b and us-central1-f experienced network isolation, a few regional products with a dependency on the affected datacenter were also affected. Between 07:41 and 11:52 US/Pacific on Tuesday, September 1, 2026, the following products were affected: - **Compute Engine VMs**: VMs running in the affected data center lost network connectivity to resources located outside of their datacenter. From a Compute Engine perspective, this incident only affected a subset of VMs in us-central1-b and us-central1-f. - **Zonal and Regional Products Based on Compute Engine** Products with dependencies on VMs in the affected datacenter experienced symptoms like connection timeouts or service unavailability because the underlying VMs lost network connectivity. Examples of products that are based on Compute Engine VMs include: Apigee, Cloud SQL, AlloyDB for PostgreSQL, Dataflow, Cloud Filestore, and Looker. - **Google Kubernetes Engine (GKE)**: GKE clusters whose control plane had dependencies on Compute Engine VMs in the affected clusters experienced errors and timeouts when administrators or nodes attempted to connect to the Kubernetes API server. Consequently, deployments might have been delayed, Kubernetes API operations might have failed, and new nodes might not have registered with the control plane (causing node pool operational delays). - **Cloud Run and App Engine:** Workloads in the us-central1 region experienced elevated serving latency, 5xx aborted request errors, and outbound connectivity degradation. To prevent severe request failures, traffic was diverted to other clusters. Diverting requests to other clusters temporarily caused elevated latency or errors in those clusters.Once the impacted cluster was restored, workload demand for these new backends created a 'thundering herd' of concurrent instance initializations. This sudden surge bypassed warm caches and overwhelmed internal file-serving tiers, extending symptoms until emergency capacity was provisioned. - **Other Google APIs and Services**, in us-central1 or a multi-region that includes us-central1, were affected if they had dependencies on software tasks in the affected clusters and weren't able to use software tasks in an unaffected cluster. Symptoms included connection timeouts or service unavailability. Examples of other Google APIs and services are: Google Cloud Bigtable, BigQuery, Google SecOps SOAR, Cloud Spanner, and Cloud Firestore. For some services, symptoms were limited to temporary latency spikes and service unavailability until workloads shifted to unaffected clusters. - **Private Service Connect**: Clients (in any region) experienced connectivity issues involving PSC endpoints with dependencies in us-central1, in specific circumstances: - PSC endpoints for Google APIs were affected in the same way as noted in the Google APIs and services section. - Producer/consumer PSC endpoints were affected if the producer had load balancers or backend VMs in the affected clusters of us-central1. - **Hybrid Connectivity**: Some Cloud Router, Cloud VPN, and Cloud Interconnect VLAN attachments in us-central1 were affected in specific circumstances: - Cloud Router BGP software tasks for some Cloud Routers in us-central1 experienced a Cloud Router maintenance event if their BGP software tasks were running in the affected clusters. - Cloud VPN tunnel software tasks running in the affected clusters lost network connectivity. This caused some Cloud VPN tunnels in us-central1 to go down until a replacement tunnel task in a different cluster was assigned. - Resources (in any region) sending packets through Cloud Interconnect VLAN attachments in us-central1 might have experienced intermittent connectivity issues in certain situations where network flows weren’t already programmed. - **Envoy-based load balancers and other products**: Load balancers and other products that depended on managed Envoy tasks of a proxy-only subnet in us-central1 experienced increased latency or connectivity issues if enough Envoy tasks were in the affected clusters. The following products were affected: - Regional external application load balancers in us-central1 - Regional internal application load balancers in us-central1 - Cross-region internal application load balancers with proxies in us-central1 - Regional external proxy network load balancers in us-central1 - Regional internal proxy network load balancers in us-central1 - Cross-region internal proxy network load balancers with proxies in us-central1 - Secure Web Proxy in us-central1 --- [2026-09-10T21:20:16+00:00] (AVAILABLE) ## Incident Report ## Summary On Tuesday, 1 September 2026, some customers in us-central1 experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. ## Root Cause The event was triggered by the inadvertent physical disconnection of network fiber-optic cables during routine hardware maintenance. A technician was performing a scheduled capacity upgrade on data center routers that support a fraction of capacity in the us-central1-b zone and a small fraction of capacity in the us-central1-f zone. During the hands-on maintenance, optical fibers connecting the data center routers to the network fabric were unplugged and the optical transceivers were replaced with a transceiver that supports higher-density connectivity. This transceiver was not compatible with the transceivers still in place in the data center fabric, causing a loss of connectivity for each fiber. The network is designed with redundant routers in separate data center rooms within the same building so that failure of one router does not interrupt service. The maintenance was intended to upgrade one router at a time over several days, with traffic diversions at each step and verification between steps that the network has returned to a fully-connected state. However, a procedural error in the manually orchestrated upgrade process caused the complete list of transceiver replacements across all routers to be issued to the technician without instructions to sequence the work one router at a time. The maintenance workflow also did not include the expected human and software verification steps for detecting unintended disruption. As a result, the technician sequentially unplugged all fiber paths across the affected devices within 13 minutes. The speed and nature of the error prevented warnings of incorrect action from reaching the engineer before connectivity was lost. Additionally, a standing procedure to halt if light is detected on any fiber-optic cable after unplug was not followed. This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone. ## Remediation and Prevention The issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams. To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure. Concurrently, hardware operations technicians on site identified the disconnected optical links and physically re-inserted the original optical transceivers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service. Google is committed to preventing a repeat of this issue in the future and is completing the following actions: * Reinforce training in us-central1 region and globally around safe maintenance techniques, including the mandatory requirement to verify whether light is on every unplugged fiber. * Finish the migration of network upgrade workflows to a fully automatically sequenced orchestration system. Most common regional network workflows were completed in 2025, and zonal workflows are in progress. * Complete the deployment of work stop alerting for actions resulting in the unintended disconnection of live fiber cables in Google data centers. * Complete the deployment of the automated system that will move regional traffic away from faulty zones, reducing the time from approximately 19 minutes in this instance to around 5 minutes. ## Detailed Description of Impact On Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b and us-central1-f zones experienced severe network degradation and resource isolation. 1. Multiple products affected: The network disruption to/from a portion of us-central1-b and us-central1-f zones impacted all zonal GCP Products for some customers in those zones. 2. Regional impact: Regional products using capacity in the affected data center were affected until traffic diversions were fully in place at 08:00 US/Pacific. Google Kubernetes Engine and Cloud Run / Google App Engine had longer impacts, discussed below. 3. Error rate: Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss. **Affected Services and Features** * Google Compute Engine: Inability for users in the affected portion of us-central1-b and us-central1-f zones to access virtual machines externally, and inability for VMs to reach remote resources. * Cloud SQL, AlloyDB for PostgreSQL, Google Cloud Bigtable, Cloud Filestore: Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * Cloud Spanner, Cloud Firestore: Elevated Remote Procedure Call (RPC) error rates, write timeouts, and latency spikes for the nam5 multi-region instances due to severed connectivity to components running in the impacted zone. * Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect: Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone. * Google Kubernetes Engine (GKE): Between 07:40 and 09:15 US/Pacific on 2026-09-01, customers experienced errors and timeouts on requests to their GKE cluster control planes (Kubernetes API servers) in us-central1. This could have caused new or updated workloads to fail scheduling and Kubernetes API operations to fail. * Cloud Run / Google App Engine: Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. **Customer Impact** Customers with resources hosted in the specific affected clusters within us-central1-b and us-central1-f experienced a complete loss of network connectivity starting at 07:41 US/Pacific. During this time, virtual machines and associated services became unreachable from the internet and from other Google Cloud regions, and those internal resources could not initiate outbound connections. The issue was detected almost immediately and engineers took traffic diversion actions at 07:45 and 08:00 US/Pacific to reduce impact to regional products. By 08:50 US/Pacific, the majority of physical connections were restored and traffic began recovering, and the traffic diversion actions were removed at 09:19 US/Pacific. Full recovery across nearly all affected services and long-running operations was confirmed by 11:52 US/Pacific. Because the network isolation was strictly limited to specific clusters within us-central1-b and us-central1-f, multi-zonal deployments correctly utilizing redundancy across other zones in us-central1 were largely able to bypass the physical hardware failure and continue serving traffic. Customers with projects in both us-central1-b and us-central1-f would not have seen multi-zonal impact - at most one of their zones would have capacity in the affected data center. --- [2026-09-03T17:47:02+00:00] (AVAILABLE) ## Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business. We know how much you rely on Google Cloud, and we regret the impact on your environment. We are working to address the root cause and prevent this from occurring in the future. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support ## Date/Time of the Issue (All time US/Pacific) **Incident Start**: 1 September 2026 07:41 **Incident End**: 1 September 2026 11:52 **Duration**: 4 hours, 11 minutes ## Summary On Tuesday, 1 September 2026, some customers in us-central1-b experienced network service degradation and instance isolation for a duration of 4 hours and 11 minutes. To our customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. ## Preliminary Root Cause The immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure. An engineer was performing a scheduled capacity upgrade on datacenter routers that support a fraction of capacity in the us-central1-b zone. The network architecture for each datacenter is designed with redundancy across multiple routing devices. The system is designed to be resilient to all single device or fiber-path failures, and most double- or triple-failures do not affect customer traffic. To ensure this, the devices and fiber paths are physically separated in each datacenter, with diverse power sources. A procedural error meant that the physical maintenance action sequentially unplugged 100% of fiber paths across all devices within 13 minutes. The nature of the error, combined with the speed of the action, prevented warnings of incorrect action reaching the engineer before complete disconnection. This resulted in compute capacity in the impacted zone being isolated from the network. Customers were unable to reach their virtual machines, and those virtual machines could not establish connections outside their zone. ## Remediation The issue was detected immediately by automated network loss monitoring systems as well as proactive probes, which rapidly engaged the network engineering and incident response teams. To mitigate the immediate customer impact, engineering teams actively moved traffic away from the impacted infrastructure, automatically shifting compatible workload traffic to healthy capacity elsewhere in the region. Concurrently, hardware operations technicians on site identified the disconnected optical links and physically reseated the fibers. Once the physical links were fully restored, traffic flow rates normalized and the traffic was redirected back to return the capacity to service. ## Description of Impact On Tuesday, 1 September, from 07:41 to 11:52 US/Pacific, a portion of the us-central1-b zone experienced severe network degradation and resource isolation. * **Multiple products affected:** The network disruption to/from a portion of us-central1-b impacted multiple GCP Products for some customers in that zone. * **Error rate:** Traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, resulting in unreachable virtual machines and elevated packet loss. **Affected Services and Features** * **Google Compute Engine:** Inability to access virtual machines externally, and inability for VMs to reach remote resources. * **Google Kubernetes Engine:** Unreachable regional/zonal clusters and nodes within the affected zone, prompting delayed cluster recovery times. * **Cloud Run / Google App Engine:** Temporary latency spikes and pending queue aborts as backend workloads automatically evacuated and shifted to healthy capacity. A subset of workloads in the specifically impacted area experienced degradation until 11:52 US/Pacific. * **Cloud SQL, AlloyDB for PostgreSQL, Cloud Spanner, Google Cloud Bigtable, Cloud Filestore:** Data access and connectivity severed for instances localized strictly to the impacted infrastructure. * **Virtual Private Cloud (VPC), Cloud NAT, Apigee, Google BigQuery, Google Cloud Dataflow, Looker, Google SecOps SOAR, Hybrid Connectivity, Cloud Interconnect:** Elevated network packet loss, connectivity drops, and API timeouts in the impacted zone. --- [2026-09-01T19:12:28+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b experienced network service degradation. **Description** The issue was affecting a portion of the us-central1-b zone. The remaining zones in the region were unaffected. All the impacted products have recovered. From preliminary analysis, routine network fabric path maintenance triggered unexpected issues in one cluster in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We thank you for your patience while we worked on resolving the issue. **Customer Symptoms** Customers experienced elevated packet loss and errors. **Workaround** The issue has been mitigated. --- [2026-09-01T18:36:56+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b are experiencing network service degradation. **Description** The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered. The impacted products Cloud Run and App Engine are also seeing recovery and the errors have lowered pre-incident levels. Engineering continues to monitor and validate. Routine network fabric path maintenance triggered unexpected issues in two clusters in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. **Customer Symptoms** Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore **Workaround** * Customers with service layer retries may see successful completions as only a portion of the zone is affected. --- [2026-09-01T18:01:09+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b are experiencing network service degradation. **Description** The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. Routine network maintenance triggered issues in us-central1-b. We have recovered the network capacity that was impacted and restored all zonal and regional services. Maintenance in the region has been halted while proactive audits are carried out. We will provide more information by Tuesday, 2026-09-01 12:30 US/Pacific with current details. **Customer Symptoms** Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore **Workaround** * Customers with service layer retries may see successful completions as only a portion of the zone is affected. --- [2026-09-01T17:09:04+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b are experiencing network service degradation. **Description** The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Most of the products have recovered and the engineering team is still validating a few remaining product impacts. We will provide more information by Tuesday, 2026-09-01 11:00 US/Pacific with current details. **Customer Symptoms** Customers are experiencing elevated packet loss and errors. * Cloud Run * App Engine Following products have recovered * BigQuery * Cloud Dataflow * Cloud Bigtable * Compute Engine * Cloud Hybrid Connectivity * Cloud Filestore * Virtual Private Cloud * Cloud Spanner * Google Kubernetes Engine * Cloud SQL * AlloyDBCloud VPN * Apigee * Cloud Interconnect * Cloud NAT * Looker (Google Cloud core) * Google SecOps SOAR * Cloud Firestore **Workaround** * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones. --- [2026-09-01T16:19:12+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b are experiencing network service degradation. **Description** The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Following network-level mitigation efforts, the underlying network infrastructure has fully recovered. Product engineering teams across Cloud are currently validating the recovery status of individual services. We will provide more information by Tuesday, 2026-09-01 10:00 US/Pacific with current details. **Customer Symptoms** Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) * Apigee * Google SecOps SOAR **Workaround** * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones. --- [2026-09-01T16:03:12+00:00] (SERVICE_DISRUPTION) **Summary** Multiple products in us-central1-b are experiencing network service degradation.  **Description** We are experiencing a networking issue starting Tuesday, 2026-09-01 07:41 US/Pacific. The issue is affecting a portion of the us-central1-b zone. The remaining zones in the region are unaffected. Our network engineering team has put in mitigations and we are seeing services starting to recover. We will provide more information by Tuesday, 2026-09-01 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. **Customer Symptoms** Customers are experiencing elevated packet loss and errors. * BigQuery * Cloud Dataflow * Cloud SQL * App Engine * Cloud Run * AlloyDB * Cloud Bigtable * Cloud Spanner * Compute Engine * Cloud VPN * Cloud Interconnect * Cloud NAT * Cloud Filestore * Cloud Hybrid Connectivity * Virtual Private Cloud * Google Kubernetes Engine * Looker (Google Cloud core) **Workaround** * Customers with service layer retries may see successful completions as only a portion of the zone is affected. * Customers that are severely affected and have services available in alternate zones can failover to alternate zones.

# Incident Report ## Summary On Thursday, 20 August 2026, from 08:00 to 10:22 US/Pacific (15:00 to 17:22 UTC), multiple Google Cloud services in the us-west1 region experienced elevated latency, provisioning failures, increased error rates, and widespread service degradations for a total duration of 2 hours and 22 minutes. The incident impacted a wide range of core services — including Persistent Disk, Google Kubernetes Engine, Google Compute Engine, Cloud Run, Google Cloud Bigtable, Cloud Storage, and Identity and Access Management — across both control plane operations and data plane requests. We sincerely apologize for the disruption this incident caused to your business operations and critical workloads. We recognize the vital role Google Cloud plays in supporting your organization, and we deeply regret the impact on your business operations and critical workloads. Engineering and infrastructure teams are actively implementing measures to address the root causes and strengthen network resiliency to prevent recurrences in the future. Specifically, teams are monitoring recovery progress, refining safety checks for planned optical maintenance, and optimizing regional traffic routing mechanisms to safeguard against unexpected capacity constraints. ## Root Cause The disruption originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers within the us-west1 region. Automated rerouting mechanisms failed to properly redistribute traffic to alternate capacity, resulting in network congestion as volumes exceeded available bandwidth in the affected area. This underlying network degradation subsequently impacted higher-level service components through severe packet loss, request throttling, and increased latency across inter-campus dependencies. Core infrastructure services, including Spanner Paxos consensus and the Unified Metadata Server (UMS), experienced significant latency spikes, which cascaded into timeouts and elevated error rates for downstream dependent products such as Cloud Storage, Cloud IAM, Persistent Disk, and Google Kubernetes Engine. Consequently, both control plane operations and data plane requests failed to execute successfully across multiple Google Cloud services in us-west1 throughout the duration of the incident. ## Remediation and Prevention Internal monitoring systems initially detected widespread service anomalies and alerted Google engineers, who promptly confirmed that multiple core services operating across the us-west1 region were severely impacted. In response to the immediate operational risks, engineering teams promptly executed emergency traffic draining protocols to reroute active service workloads away from the compromised inter-campus network infrastructure and minimize further customer disruption. Once teams restored inter-campus fiber network capacity, engineers systematically reintroduced production traffic back to us-west1 through controlled validation stages, ultimately confirming full service normalization across all impacted platforms. Longer term engineering remediations and architectural enhancements are actively being finalized and assigned to owning teams. Key focus areas include reforming scheduled maintenance protocols, establishing stricter safety checks and circuit redundancy requirements prior to routine maintenance, and refining automated regional traffic failover mechanisms to automatically handle sudden capacity degradation without incurring severe network congestion. ## Detailed Description of Impact On Thursday, August 20, between 08:00 and 10:22 US/Pacific, Google Cloud customers in the us-west1 region encountered elevated latency, provisioning failures, and increased error rates. * **Impact / Error Rates:** The reduced capacity caused request throttling, latency spikes, and increased retry volume. * **Scope Exclusion:** Services and workloads operating in regions other than us-west1 remained fully operational and unaffected. **Affected Services and Features** The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) * Managed Service for Apache Spark (Dataproc) * Persistent Disk The regional incident in us-west1 followed a structured three-phase recovery dictated by platform dependency layers: While core platform connectivity and live request serving were restored by 10:22 US/Pacific, certain services took additional time to fully recover due to asynchronous backlog processing and localized control-plane state reconciliation for a very small set of customers. For event-driven and pipeline services, inbound error rates dropped to 0 immediately, but some operations experienced elevated latency while workers processed through backlogs accumulated during the outage, clearing later for Cloud Pub/Sub, Cloud Build & Deploy, Cloud Dataflow, and Cloud Storage lifecycle deletions. Concurrently, while primary read/write traffic was healthy across the region, specific long-running lifecycle workflows required extra time to clear locks and reconcile distributed state machines. Compute Engine VM provisioning in zone us-west1-c and Cloud Filestore control plane instance allocation locks and resource validation checks took longer to normalize across regional storage backends. --- [2026-08-25T06:20:02+00:00] (AVAILABLE) # Preliminary Incident Report We sincerely apologize for the disruption this incident caused to your business operations. Recognizing your reliance on Google Cloud, we express our sincere regrets for any operational impact experienced. Our engineering teams are actively addressing the underlying root cause to prevent future recurrences. Please note that the information provided herein reflects our current understanding as of the time of publication and remains subject to revision as the investigation progresses. A comprehensive Incident Report detailing preventive measures will be issued upon conclusion of our inquiry. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. ## Date/Time of the Issue (All time US/Pacific) * Incident Start: 20 August 2026 08:00 * Incident End: 20 August 2026 10:22 * Duration: 2 hours, 22 minutes ## Summary On Thursday, 20 August 2026, multiple Google Cloud services encountered elevated latency, provisioning failures, increased error rates, and service degradations lasting for a duration of 2 hours and 22 minutes. ## Preliminary Root Cause The disruption originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers within the us-west1 region. Automated rerouting mechanisms failed to properly redistribute traffic to alternate capacity, resulting in network congestion as volumes exceeded available bandwidth in the affected area. This underlying network degradation subsequently impacted higher-level service components through request throttling, increased latency, and cascading retries. Consequently, both control plane operations and data plane requests failed to execute successfully across several dependent Google Cloud services, producing the observed latency and elevated error rates. ## Remediation Internal monitoring alerted Google engineers to the incident, confirming that multiple services across us-west1 were affected. To mitigate the immediate operational impact, engineers initiated traffic draining protocols, re-routing service workloads away from the degraded network infrastructure. Following the restoration of inter-campus network capacity and the resolution of the core issue, engineers systematically restored traffic to the us-west1 region and confirmed service normalization. Long-term engineering solutions are currently being finalized and assigned to address the root cause and strengthen system resilience regarding fiber maintenance procedures. ## Description of Impact On Thursday, August 20, between 08:00 and 10:22 US/Pacific, Google Cloud customers in the us-west1 region encountered elevated latency, provisioning failures, and increased error rates. * Proportionate Impact / Error Rates: The precise scope of impact—including the proportion of affected projects and applications—and detailed error metrics will be published in the comprehensive Root Cause Analysis, as the reduced capacity caused request throttling, latency spikes, and increased retry volume. * Scope Exclusion: Services and workloads operating in regions other than us-west1 remained fully operational and unaffected. **Affected Services and Features** The following services experienced elevated latencies and/or increased error rates, across their respective data planes and control planes: * AlloyDB * Apache Kafka * Apigee Edge Public Cloud * Apigee X * Artifact Registry * BigQuery & BigQuery Data Transfer Service * Cloud Build * Cloud Data Fusion * Cloud Dataflow * Cloud Filestore * Cloud Key Management Service (KMS) * Cloud Monitoring * Cloud Run * Cloud SQL * Contact Center AI Platform * Dataproc Metastore * Google App Engine * Google Cloud Bigtable * Google Cloud Pub/Sub * Google Cloud Storage (GCS) * Google Compute Engine (GCE) * Google Kubernetes Engine (GKE) * Identity and Access Management (IAM) * Managed Airflow (Cloud Composer) *Managed Service for Apache Spark (Dataproc) Persistent Disk --- [2026-08-20T19:37:40+00:00] (AVAILABLE) **Summary** The issue causing timeouts, degradations, errors, and latencies across multiple us-west1 products has been mitigated. **Description** We have mitigated the issue impacting multiple products in our us-west1 region as of Thursday, 2026-08-20 10:22 PDT. Our engineering teams have restored capacity on a planned optical maintenance that caused unexpected congestion in Dalles, Oregon metro /us-west1 region. Our systems stabilized and services have recovered once capacity was restored. Our teams are continuing to monitor for any residual impact. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue. **Diagnosis / Customer Symptoms** Customers in us-west1 may have experienced timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** This issue is now mitigated. --- [2026-08-20T19:21:07+00:00] (SERVICE_OUTAGE) **Summary** The issue causing timeouts, degradations, errors, and latencies across multiple us-west1 products has been mitigated and we are working to recover all products. **Description** Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 13:30 PDT with details. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** No workarounds needed at this time. --- [2026-08-20T18:53:26+00:00] (SERVICE_OUTAGE) **Summary** The issue causing timeouts, degradations, errors, and latencies across multiple us-west1 products has been mitigated and we are working to recover all products. **Description** Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 12:30 PDT with details. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** No workarounds needed at this time. --- [2026-08-20T18:03:07+00:00] (SERVICE_OUTAGE) **Summary** The issue causing timeouts, degradations, errors, and latencies across multiple us-west1 products has been mitigated and we are working to recover all products. **Description** Mitigation actions have been completed by our engineering teams and we are seeing recovery from multiple products. We are continuing to work to recover the remaining products. We will provide an update by Thursday, 2026-08-20 11:45 PDT with details. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** No workarounds needed at this time. --- [2026-08-20T17:32:15+00:00] (SERVICE_OUTAGE) **Summary** We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region. **Description** Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 11:00 PDT with details. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** We recommend customers to failover to other regions where feasible. --- [2026-08-20T17:13:59+00:00] (SERVICE_OUTAGE) **Summary** We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region. **Description** Mitigation actions are implemented by our engineering teams, recovery trends have been observed across infrastructure layers. Active efforts remain underway to bring impacted cloud services back to full operation. We will provide an update by Thursday, 2026-08-20 10:45 PDT with details. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** We recommend customers to failover to other regions where feasible. --- [2026-08-20T16:44:37+00:00] (SERVICE_OUTAGE) **Summary** We are investigating an issue where customers may experience timeouts, service degradations, errors, and elevated latencies across multiple products in the us-west1 region. **Description** We are experiencing an issue with multiple products, beginning on Thursday, 2026-08-20 08:40 PDT. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2026-08-20 10:30 PDT with details. We apologize to all who are affected by the disruption. **Diagnosis / Customer Symptoms** Customers in us-west1 may experience timeouts, service degradations, errors, and elevated latencies across multiple products. **Workaround** None at this time.