Buildkite Outage History

86 incidents recorded over the last 365 days, 85 of them resolved. Durations are measured from when PulsAPI first saw the problem to when it cleared, which is usually longer than the vendor's own figure.

Live Buildkite status99.89% uptime history

Every Recorded Buildkite Incident

Buildkite incidents, newest first, with date, duration and status.
DateIncidentDurationStatus
Nov 4, 2026Elevated API latency (single customer)We received a report that API requests were taking abnormally long to process. Investigation revealed this to be isolated to a single customer shard, with impact between 2026-10-22 01:50 - 08:30 UTC.-20499 minResolved
Nov 4, 2026Increased error rates on Agent APIThis incident is resolved.3h 28mResolved
Nov 4, 2026Increased latency and error ratesWe've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.19 minResolved
Nov 3, 2026Increased latency and error ratesWe have seen recovery and latency has returned to normal.1h 29mResolved
Nov 3, 2026Elevated API error ratesAPI errors have subsided and we're seeing recovery. From 19:19-19:39 UTC we saw elevated errors on the Agent API and REST API.32 minResolved
Nov 3, 2026Increased latency and error ratesWe have seen recovery and latency has returned to normal.2h 8mResolved
Oct 29, 2026Elevated job dispatch timesWe are seeing the job dispatch times are back to normal.2h 0mResolved
Oct 28, 2026Increased latency and error ratesWe identified that the majority of the impact was for a single customer and was caused by a database failover. We are investigating the cause and we will work with the impacted customer.31 minResolved
Oct 26, 2026Elevated API latency and error ratesBetween 17:07-17:19 UTC, our API endpoints experienced elevated latencies and errors due to issues with a shared service. Our monitoring systems detected these issues, which triggered automated system remediations and…-84 minResolved
Oct 24, 2026Elevated error ratesBetween 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database…0 minResolved
Oct 22, 2026Delays with Hosted Agents jobsProcessing of Hosted Agents jobs are no longer delayed.45 minResolved
Oct 20, 2026Buildkite response to the eMudhra certificate compromiseAll services that power Buildkite have been updated to distrust the affected certificates, and updated releases are available for the components we distribute that run in customer infrastructure: Buildkite agent…24h 24mResolved
Oct 20, 2026Delayed notificationsAdding additional capacity has resolved the notifications latency.1h 8mResolved
Oct 19, 2026Delayed Hosted Agents instance creationWe received reports from a few customers that Hosted Agent jobs were delayed in starting. We investigated and found this related to a GitHub incident with webhooks that caused throttling in Hosted Agent job creation…0 minResolved
Oct 6, 2026Degraded checkout performance with Hosted AgentsThe issue is now resolved. Network maintenance from our provider caused unexpected side effects that impacted outgoing traffic from Hosted Agents.5h 40mResolved
Oct 1, 2026Small number of incoming GitHub Webhook failingThe error rate has returned to normal levels.47 minResolved
Sep 30, 2026macOS hosted agent jobs not startingBetween 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still…-3180 minResolved
Sep 30, 2026Delayed notificationsOutbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved.1h 28mResolved
Sep 22, 2026Issues with webhook ingestion, GraphQL queries## Service Impact On September 22, 2026, between 19:09 and 19:32 UTC, customers experienced failures ingesting webhooks and executing GraphQL queries. Some web application requests also failed or timed out. During…42 minResolved
Sep 21, 2026AWS us-east-1 outageWe did not experience any customer impact during this AWS us-east-1 outage.45 minResolved
Sep 21, 2026Increased latency and error ratesThis incident has been resolved.1h 10mResolved
Sep 9, 2026Buildkite service disruption## Service Impact _All times UTC unless stated otherwise._ Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00…6h 46mResolved
Sep 4, 2026Hosted Agents - Increased latency and error ratesJobs running on Hosted Agents are now being dispatched promptly.56 minResolved
Sep 4, 2026Increased latency and error ratesWe're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.5 minResolved
Sep 3, 2026Scheduling delay for hosted agentsJobs running on Hosted Agents are now being dispatched promptly.1h 5mResolved
Sep 2, 2026Delays in Hosted Agents job dispatchJob dispatch for Hosted Agents is no longer experiencing delays.1h 5mResolved
Sep 1, 2026Delivery issues with email notificationsOur outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully.1h 19mResolved
Aug 27, 2026Test Engine ingestion delayed, catching upProcessing of uploaded test execution data has caught up and is back to normal.51 minResolved
Aug 26, 2026Issues identified with buildkite-agent registrationMitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now.1h 30mResolved
Aug 25, 2026Buildkite service disruption## Service Impact Customers experienced a site-wide disruption affecting the Buildkite web interface, REST API, Agent API, job queue, and notifications. The disruption began at 22:47 UTC and continued until 23:16…1h 22mResolved
Aug 25, 2026Unexpected agent disconnection for some customersThis incident has been resolved.1h 34mResolved
Aug 17, 2026Buildkite service disruptionThis incident has been resolved.1h 57mResolved
Aug 13, 2026Slow web UIWe think the impact from the issue is over.58 minResolved
Aug 12, 2026Buildkite service disruptionFor the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully. At 23:01 11 August 2026 UTC we started routing <1% of agent traffic for a small subset of our…1h 24mResolved
Aug 10, 2026Slow web UIThis incident has been resolved.3h 3mResolved
Aug 5, 2026Pipelines latency and webhook errorsBetween 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.-73 minResolved
Aug 3, 2026Latency on Pipelines## Service Impact Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was…25 minResolved
Jul 28, 2026Delayed notificationsNotification latency for the affected shard is now under 5s.19 minResolved
Jun 18, 2026Increased latency and error ratesWe have seen a full recovery of services.1h 37mResolved
Jun 11, 2026Increased latency on REST and GraphQL APIsThe mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency.57 minResolved
Jun 11, 2026Increased latency and error rates for Agent APIBetween 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job…18 minResolved
May 31, 2026Email deliveries are delayedWe have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails. This issue has now been resolved.-2835 minResolved
May 28, 2026Delayed notifications## Service Impact Customers experienced delayed Buildkite notification delivery. The customer impact varied depending on how those notifications are used. For some customers, delayed notifications also delayed…57 minResolved
May 26, 2026Increased latency and error rates## Service Impact A subset of customers experienced elevated latency in notification delivery. ## Incident Summary While migrating a subset of our background processing services to Amazon EKS, we encountered an issue…41 minResolved
May 20, 2026Delayed notifications## Service Impact A subset of our customers experienced elevated latency in our notification delivery, build dispatch and metrics services. ## Incident Summary We are in the process of migrating our underlying…59 minResolved
May 15, 2026Delayed Test Engine ingestion processingProcessing of the backlog is complete.44 minResolved
May 13, 2026Error rates increasingAdditional capacity was added to our redis caches. This triggered a failover between UTC 15:10 - 15:14 and there was a spike of errors on the REST and GraphQL APIs. Customers would have seen some errors in the Buildkite…19 minResolved
May 12, 2026Delayed Test Engine ingestion processingThe fix was successful and the backlog has now been cleared.3h 9mResolved
May 8, 2026Delayed Test Engine ingestion processingThe delayed backlog is now cleared and Test Engine ingestion is operating normally.2h 32mResolved
May 8, 2026Delayed Test Engine ingestion processingThe Test Engine backlog is now cleared and operating normally.46 minResolved
May 8, 2026AWS us-east-1 single availability zone outage## Service impact On 8th May 2026 UTC, between 00:00 and 07:30 UTC, some customers would have seen intermittent errors and latency spikes across many areas of the platform. ## Incident Summary The AWS availability…27h 22mResolved
May 7, 2026Delays in job dispatch, webhook processing, and outbound webhooks## Service Impact On 2026-05-07 between 22:22 UTC and 00:14 UTC the next day \(~1 hour 52 minutes\), Buildkite customers experienced delays across several Pipelines features: * Inbound webhook processing was…1h 28mResolved
May 7, 2026Jobs not starting on hosted agents and agent-stack-k8s## Service Impact On May 7th between 08:14 and 09:25 UTC customers using [hosted agents](https://buildkite.com/docs/agent/buildkite-hosted), the [k8s stack](https://buildkite.com/docs/agent/self-hosted/agent-stack-k8s)…49 minResolved
May 6, 2026Test Engine: Delayed processing of test result ingestionProcessing of test execution ingestion data has successfully caught up.1h 28mResolved
May 4, 2026Delayed notificationsNotifications continue to be delivered without delay for the previously affected subset of customers. This incident is resolved.2h 33mResolved
May 4, 2026Increased latency and error ratesAn increase in requests has lead to the API service being temporarily saturated. We have updated rate limits to ensure this doesn't re-occur and will add further resources if necessary27 minResolved
May 1, 2026Buildkite service disruption### Service Impact Between 02:29 and 02:48 UTC on May 1st 2026, the Buildkite dashboard, Agent API, REST API and GraphQL were severely degraded for the majority of customers. During this period, customers were unable…49 minResolved
Apr 29, 2026Increased latency and error ratesWe have confirmed that latency and error rates have returned to normal for impacted customers.1h 4mResolved
Apr 28, 2026Increased dispatch latency and error ratesPreviously elevated loads with Hosted Agents dispatch have fully recovered.1h 15mResolved
Apr 22, 2026Auth failures with remote MCP serverThe issue is resolved.1h 40mResolved
Apr 22, 2026Delayed processing of test executionThe backlog has been cleared and all systems are fully operational. Thank you for your patience.2h 35mResolved
Apr 8, 2026Degraded performance and increased error rates## Service Impact On April 8th from 22:10 to 22:15 UTC and from 22:33 to 22:38 UTC all customers would have experienced an increase in latency browsing [buildkite.com](http://buildkite.com) and using Buildkite REST and…45 minResolved
Mar 31, 2026Hosted Agents jobs immediately cancelledWe have deployed the fix and we have confirmed customer builds are working. If you encounter any further issues please contact support.43 minResolved
Mar 27, 2026504 errors viewing buildsThe incident is now resolved. We are no longer seeing errors when viewing pipelines.1h 51mResolved
Mar 25, 2026Increased Delays with Hosted AgentsThis incident is now resolved. We are no longer seeing further networking issues with Hosted Agents, which affected delays in creating them for Jobs, resolving external traffic and interactions with Cache - affecting…1h 24mResolved
Mar 11, 2026Increased queue times on hosted agentsThis incident has been resolved.1h 24mResolved
Mar 10, 2026Increased error rates from Test Plan APIOur mitigations have resolved the elevated latency and likelihood of suboptimal fallback test plans. We have also identified and fixed a blind-spot in our automated alerting, which was previously unable to detect this…8h 13mResolved
Mar 7, 2026Elevated ingestion latency for Test EngineProcessing of test execution ingestion data has successfully caught up.44 minResolved
Mar 6, 2026Hosted Agents: Job start latency for a small subset of customersBuildkite Hosted Agents experienced degraded start-time performance due to a network partition issue in the Hosted Agents control plane. A small subset of customers may have seen delayed job starts during 04:40-04:50…-264 minResolved
Mar 5, 2026Slow artifact uploadsWith artifact upload latency continuing to be stable, we are resolving this incident.12h 9mResolved
Mar 4, 2026Elevated ingestion latency for Test EngineProcessing of test execution ingestion data has successfully caught up.8h 36mResolved
Mar 3, 2026Latency issuesWe have completed the provisioning of additional capacity mentioned in our last update, and error rates and response times have returned to normal. This incident is now resolved.7h 32mResolved
Feb 26, 2026Increased dispatch latency# Service Impact Between approximately 18:00 UTC and 22:50 UTC February 26, 2026, a subset of customers experienced increased latency when dispatching jobs to agents. Affected customers observed agents sitting idle for…7h 20mResolved
Feb 26, 2026Increased latency for secrets endpoints for some customersResponse times have returned to normal. This incident is now resolved.2h 1mResolved
Feb 17, 2026Increased dispatch latencyThe cause of pipeline upload failures and dispatch latency that was impacting some of our customers has been resolved.56 minResolved
Feb 13, 2026Some jobs not dispatching## Service Impact Between 01:33 UTC and 04:49 UTC on February 12, 2026, a subset of Sidekiq background workers experienced database connection failures, affecting several types of operations over approximately **3…-1526 minResolved
Feb 3, 2026Delays with GitHub WebhooksThis incident has been resolved.3h 7mResolved
Jan 31, 2026Test Engine services unavailableWe believe this issue is now resolved and all Test Engine systems are functioning. Ingestion of test data was delayed during this incident but we believe there has been no data loss. Customers using Buildkite Test…1h 29mResolved
Jan 24, 2026Agent unexpectedly disconnected## Service Impact Agents that were in a connected or stopping state before January 23, 06:50 UTC—and remained in that state through January 25, 08:05 UTC—were incorrectly transitioned to lost, causing them to…9h 0mResolved
Jan 16, 2026Increased dispatch latencyThis incident is now resolved.4 minResolved
Jan 11, 2026Database maintenance for subset of customersThe scheduled maintenance has been completed.4h 0mScheduled
Jan 6, 2026Slow Loading of Dashboard PagesThis incident has been resolved.1h 31mResolved
Nov 21, 2025Elevated error rate for viewing logsWe have confirmed that error rates for log retrieval have returned to nominal levels.52 minResolved
Nov 12, 2025Increased latency and error ratesWe believe the impact from the issue is over, but we will continue monitoring today.2h 23mResolved
Nov 10, 2025Increased error rate affecting Web UI and APIs## Service Impact Between 04:48 and 05:30 UTC on November 10th 2025 the Agent API, that serves interactions with the Buildkite Agent, was severely degraded for the majority of customers. This degradation continued…6h 37mResolved
Nov 7, 2025Increased 403 Errors on Package Registries package access## Service Impact On October 24th, 2025 from approximately 01:53AM UTC until 04:24AM UTC our package download service experienced an outage affecting customer access. This was confirmed to have impacted Debian and…-20156 minResolved

Hear about the next one while it is open

Everything above is a record written after the fact. We can email you when the next Buildkite incident opens instead, at the point where knowing still changes what you do.

Free, no password, no card. Unsubscribe in one click.

How to Read This Buildkite Incident Log

Each row is an incident PulsAPI observed, not a summary written afterwards. The duration is wall-clock time between the first failing check and the first clean one, so it includes the window before Buildkite acknowledged anything. Vendor post-mortems typically measure from acknowledgement, which is why their numbers are usually shorter.

Incidents still open have no duration yet and are listed as ongoing rather than being given a running total. The archive covers the last 365 days; anything older has aged out of the window rather than never having happened.

The longest single Buildkite outage in this window ran 1d 3h, against a mean recovery of 2h 41m. If you depend on Buildkite in a customer-facing path, the longest figure is the one to design around. The mean is what happens on a normal bad day; the maximum is what happens on the worst one.