Is LiveKit Down?

**Summary** On October 31, 2026, between 07:10 and 07:43 UTC, an uncommon data pattern surfaced a latent software bug which caused media servers in several regions to restart repeatedly. A small percentage of participants were disconnected and the vast majority automatically reconnected. Impact was concentrated in North America. We stopped the trigger at 07:43 UTC and deployed a fix to all regions by 08:49 UTC. We apologize for the disruption this caused. **Root Cause** Applications can open their own data channel and send raw data over it, outside of LiveKit's standard data message format. Media servers forward this raw data to other participants in the room, wrapping it with the sender's identity so that receivers know who sent it. When a room spans multiple regions, the media servers hosting it also forward this data to each other. That server-to-server step sent the raw data without the sender's identity, and the receiving server assumed the identity would be present, which caused it to crash. This gap in our server-to-server protocol had been present for some time and was not introduced by a recent deployment. Starting at 07:10 UTC, one application sent raw data over its own data channel in a room that spanned multiple regions. Every server that received the forwarded message crashed at the same time, while the server the sender was connected to kept running and kept sending. Participants on the crashed servers automatically reconnected to other servers. Those servers then began hosting the room, received the next message, and crashed as well. This cycle repeated until we stopped the trigger at 07:43 UTC. In total, media servers in 12 of our data centers restarted, some of them up to 10 times. **Timeline \(UTC\)** 2026-10-31 07:10 - An application begins sending raw data over its own data channel in a multi-region room. Media servers in US East and Canada restart within seconds. 07:13 - Our monitoring detects the restarts and an incident is opened automatically. 07:10 to 07:43 - Media servers in 12 data centers restart repeatedly as participants reconnect to new servers. 07:43 - We block the traffic triggering the bug. There are no further restarts after this point, and customer impact ends. 07:46 - A fix for the bug is ready. 08:04 - The fix begins rolling out, starting with a small set of regions. 08:49 - The fix is deployed to all regions. **Scope of Impact** Realtime sessions: * Across all regions, about 1.5% were disconnected more than once within five minutes. * In US East, US Central, US West, Canada and Mexico, about 6% were disconnected more than once within five minutes. * The median time from disconnect to rejoin was about 3 seconds. Agents: * Agents were disconnected and reconnected the same way as other participants, with the same median reconnect time of about 3 seconds. An agent that reconnects keeps its conversation state, so most conversations continued after a short gap. Fewer than 1% of active agent sessions did not recover. Ingress, Egress and SIP: * Most active WHIP ingress streams were interrupted, and reconnects typically took about 40 seconds. Some WHIP ingress update and delete requests also failed. * About 5% of active egress sessions were disconnected. Most reconnected within a few seconds. * About 1% of SIP participants were disconnected, and half of them rejoined in under a second. Some SIP call transfers also failed. APIs: * About 1.3% of Room Service API requests failed during the window. ListParticipants and GetParticipant were the most affected. How to tell if you were affected: participants in affected sessions saw a reconnecting state, or a brief loss of audio and video. In session analytics, an affected participant appears to leave and rejoin the same room with the same identity between 07:10 and 07:43 UTC. **Detection and Response** Our monitoring detected the server restarts about 3 minutes after they began, and an incident was opened automatically. Identifying the specific room and traffic triggering the restarts took about 30 minutes. Once we found it, we blocked the trigger at 07:43 UTC and the restarts stopped immediately. We deployed a fix to all regions by 08:49 UTC. We did not post to our status page while the incident was in progress, and published a notice only after it was resolved. Our original post mistakenly sent notifications as we initially over-estimated the scope of impact. **Mitigations and Follow-ups** * We fixed the bug. Media servers now handle forwarded data that arrives without the sender's identity instead of crashing. This fix was deployed to all regions by 08:49 UTC. * Raw data forwarded between media servers now includes the sender's identity, so receiving servers always know who sent it. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team. --- [2026-10-31T09:11:10.494-07:00] (resolved) This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected and continued their conversations, but a small number of agent sessions ended or restarted. Some SIP call transfers also failed. - About 1.3% of Room Service API requests failed during this period. ListParticipants and GetParticipant were the most affected. Some WHIP ingress update and delete requests also failed. The bug was triggered when a specific type of data message was relayed between servers hosting the same multi-region room. We stopped the trigger at 07:43 UTC, and there were no further restarts after that. A fix was deployed to all regions by 08:49 UTC. Apologies for the impact. We will follow up with a postmortem sharing more details on impact and RCA as soon as possible.

## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55. * **SIP outbound:** about 10% of `CreateSIPParticipant` requests in US East failed, some returning 408 `deadline_exceeded`. * **APIs:** significant server errors on Room Service, Egress, and Ingress APIs in US East. Recent versions of our server SDKs automatically retry failed API requests in other regions, which reduced the impact for customers on those versions. * **Cloud Agents:** agents in US East couldn't reach some of the model providers and other services they depend on, so agent sessions failed or degraded. Cloud Agents currently run in a single region, so they couldn't be quickly moved. * **Realtime:** some participants in US East saw failed room joins or increased packet loss. Sessions that were already connected kept working. * **Dashboard:** the LiveKit Cloud dashboard was inaccessible for part of the window. **Mexico \(23:29 to 01:01 UTC\)** Active sessions were disconnected and service was degraded until we moved traffic to nearby regions. ## Root Cause Starting at around 23:28 UTC, our cloud provider had a networking incident in its US East data center. It was a partial outage: our servers could still reach end users and our other regions, but lost access to our database and to certain external services, such as model providers. Anything that depended on those started failing. API requests and SIP call setup timed out, and agents couldn't reach the model providers they need to respond. Our automatic failover moves traffic away from a region when it becomes unhealthy. It didn't kick in here, for two reasons: 1. **Our health checks mostly look at our own software.** None of it failed. Every server in US East was up and responding. 2. **We could see that US East couldn't reach external services, but nothing acted on that automatically.** Those signals paged engineers, but they didn't feed into failover. As far as failover was concerned, US East looked healthy, so we kept sending traffic to a region that couldn't do much of the work. Cloud Agents had a separate gap. They currently run only in the data center specified in its deployment \(i.e. US East\) and we did not have a process to automatically migrate them quickly in case of a data center failure. Connectivity came back at around 23:58 UTC, once the provider mitigated the issue on their side. ## Timeline \(all times in UTC, starting October 28\) * **23:28**: Requests from US East to our database and certain external services start timing out. API errors follow shortly after. * **23:35**: Alerts fire for API errors in US East and an incident is opened. * **23:42**: Incident call starts. We initially focus on the API timeouts. * **23:45**: We remove an affected server to rule out a single bad instance. Failures continue. Since SIP, APIs, and agents are all affected, we start treating it as a regional network problem rather than a software bug. * **23:56**: Incident escalated. We start draining US East. * **23:58**: The provider's fix restores connectivity in US East. Impact in US East ends. * **00:36**: We identify degraded SIP, realtime, and API service in our Mexico region. * **01:01**: Mexico traffic moved to nearby regions, and service there recovers. ## Corrective Actions & Prevention * **Fail over based on external connectivity.** We're adding checks for whether a region can reach the services it depends on, and failover will act on them. If a region can't do its job, it should be drained automatically, even when every server in it looks healthy. * **Deploy Agents to multiple data centers.** We'll run agents in multiple data centers to have sufficient redundancy. * **Faster failover in agent dispatch.** If a server or an entire data center stops responding, agent dispatch will hand the job to another agent within seconds. * **Multi-cloud redundancy.** We're adding capacity with more than one cloud provider, so an outage at one provider can't take down a region. --- [2026-11-01T08:28:36.370-08:00] (resolved) This incident has been resolved. Between 23:30 and 23:58 UTC, a network disruption at an upstream infrastructure provider affected our US East region. During this window: Inbound SIP calls in US East failed at elevated rates, with affected calls rejected with 500 or 503 responses. Answered inbound calls were about 65% below normal across the window, and up to 90% below normal between 23:45 and 23:55 UTC. About 10% of outbound SIP call requests (CreateSIPParticipant) in US East failed, some returning 408 deadline_exceeded. Between 23:28 and 23:58 UTC, there were elevated rates of Room Service, Egress, and Ingress API request server errors in US East. Some participants saw failed room joins or increased packet loss. The LiveKit Cloud dashboard was inaccessible for part of this window. Network connectivity recovered at approximately 23:58 UTC, and error rates and call volumes returned to normal by 00:20 UTC. A related network issue also affected our much smaller Mexico region from 23:29 UTC, disconnecting active sessions and degrading service until traffic was moved to nearby regions at approximately 01:01 UTC. Sincere apologies for the impact. We will follow up with a postmortem. --- [2026-10-28T17:41:01.127-07:00] (monitoring) We've applied a mitigation for our Mexico region and services have returned to baseline. We are continuing to monitor and will update again within 15 minutes. --- [2026-10-28T17:36:48.492-07:00] (identified) While the US regions still appear to be stable, we are tracking degraded services including SIP, RTC, and APIs in our Mexico region. We will update again as soon as possible. --- [2026-10-28T17:16:41.176-07:00] (monitoring) We believe the mitigation was successful and are seeing services return to normal. The dashboard is accessible and error rates have gone down across services. We are continuing to monitor to make sure that the situation is resolved. We will update again as soon as possible. --- [2026-10-28T17:09:42.153-07:00] (investigating) We are still determining the full scope of impact, but we believe that the Room Service has also been impacted. We will update again within 15 minutes. --- [2026-10-28T17:00:03.232-07:00] (investigating) We are currently attempting a mitigation. We will follow up with more information in the next 15 minutes. --- [2026-10-28T16:57:34.153-07:00] (investigating) We believe there is also impact to inbound calls in the affected regions. We are working on identifying the cause and will update again within 15 minutes. --- [2026-10-28T16:51:41.083-07:00] (investigating) The LiveKit cloud dashboard is currently inaccessible. We are investigating this and will provide another update within 15 minutes. --- [2026-11-01T08:27:15.161-08:00] (investigating) We are currently investigating an increased rate of failing outbound calls in US East. We will post another update with more information as soon as possible.

**Root Cause** Between 2026-10-16 and 2026-10-23, some new connections to LiveKit Cloud in US Central, US East and EU Central got no response. The cause was a fault in our cloud provider's network load balancers. Some load balancers kept sending new connections to servers that were no longer in service. Most clients connected on a retry, so the main effect was a slower join. Some clients waited much longer. A routine server replacement and one of our recovery actions on October 20 also closed open connections from self-hosted agent workers. This caused agent dispatch failures for some customers. **Technical Details** Each LiveKit Cloud region has network load balancers from our cloud provider. A load balancer sends new connections only to servers that pass its health check. When servers are added or removed, the provider updates the health-check configuration of the load balancer. Our cloud provider found these causes: 1. A burst of health-check configuration updates overloaded the provider's health-check service. 2. Some of these updates failed. 3. A bug in the automatic recovery left the affected health checks inactive after the overload stopped. 4. As a result, the load balancers kept old health data. They continued to send new connections to servers that were no longer listening. Those connections got no response. The fault affected only some load balancers. Other load balancers with the same servers worked correctly. The load balancers reported no errors. **Agent dispatch failures.** Each self-hosted agent worker keeps one long-lived connection to LiveKit Cloud. The worker gets job offers on this connection. Two events on October 20 closed these connections: 1. At 16:14, a routine server replacement in US Central stopped one of our ingress servers. This closed about 485 agent worker connections at the same time. At that time, new traffic did not go to US Central, so these workers reconnected to US East. The workers of each affected customer were then split between two regions. Until about 17:40, agent dispatch failed or was slow for some customers. 2. At 23:15, we deleted the faulty load balancer in US Central. About 600 agent worker connections still used it. The workers reconnected within about 1 minute. But our servers did not detect that the old connections were closed. They continued to offer jobs on these connections until 23:32, when the connections timed out. Those job offers failed, and some jobs were not dispatched. **Timeline \(UTC\)** 2026-10-16 06:50 - Connection timeouts started in EU Central 2026-10-16 10:52 - We moved traffic away from the affected EU Central servers; the timeouts stopped. We opened a case with our cloud provider 2026-10-17 - We added probes that test each load balancer address from several regions 2026-10-20 08:00 - Slow agent joins started in US Central 2026-10-20 15:28 - We moved traffic away from US Central; join times went back to normal 2026-10-20 16:14 - A routine server replacement in US Central closed about 485 agent worker connections; agent dispatch failed or was slow for some customers until about 17:40 2026-10-20 21:30 - After traffic returned to US Central, slow joins started in US Central and US East 2026-10-20 22:30 - We moved traffic away from US Central again 2026-10-20 23:13 - We moved traffic away from US East 2026-10-20 23:15 - We deleted the faulty load balancer in US Central; about 600 agent worker connections still used it 2026-10-20 23:32 - The old agent worker connections timed out; agent dispatch went back to normal 2026-10-20 23:41 - We replaced the faulty load balancer in US Central and moved traffic back 2026-10-21 00:55 - We replaced the faulty load balancers in US East and moved traffic back 2026-10-21 03:13 - The fault occurred on another load balancer in US Central; we moved traffic away 2026-10-21 06:12 - We replaced that load balancer and moved traffic back to US Central 2026-10-21 07:02 - We replaced the faulty load balancers in EU Central and moved traffic back 2026-10-21 - Our cloud provider restored the correct health state on the 5 affected load balancers 2026-10-23 15:15 - The fault occurred at a lower rate on one load balancer in US East 2026-10-23 22:35 - Our probes detected it 2026-10-23 23:10 - We moved traffic off that load balancer; customer impact ended 2026-10-27 - Our cloud provider sent the root cause analysis **Scope of Impact** On each affected load balancer, 5% to 25% of new connections got no response. The load balancer fault did not affect connections that were already open. But a routine server replacement and one of our recovery actions on October 20 closed open connections from self-hosted agent workers \(see **Technical Details**\). * Oct 16 06:50–10:52 - Region: EU Central - About 5% of new connections timed out. This included connections for SIP calls. * Oct 20 08:00 – Oct 21 06:15 - Region: US Central - A share of new connections timed out. * Oct 20 16:14–17:40 - Region: US Central, US East - Some self-hosted agents did not get jobs or joined late. Some inbound calls were not answered. * Oct 20 21:30 – Oct 21 00:55 - Region: US East - 5% to 10% of new connections timed out. * Oct 20 23:16–23:32 - Region: US Central - Job offers to some self-hosted agent workers timed out. Some jobs were not dispatched. * Oct 23 15:15–18:15 and 20:45–23:10 - Region: US East - About 0.3–0.5% of new connections to the region timed out. The LiveKit SDKs stop a signal connection attempt after 5 seconds and try again. Most participants and agents joined on the retry, about 5 seconds later than usual. Clients with a longer connection timeout, or with no timeout, waited longer. Some self-hosted agents reported room joins that took 60–70 seconds. API calls to the affected regions could also time out or be slow. **Detection and Response** Customers reported the EU Central and US Central timeouts before our monitoring found them. Our checks tested each region as a whole, so a fault on one load balancer address was hidden. Most connections in a region still worked. On October 17, we added probes that test each load balancer address from several regions. These probes detected the October 23 fault. In each event, we moved traffic away from the affected region, replaced the faulty load balancers, and then moved traffic back. **Mitigations and Follow-ups** * Our cloud provider fixed the bug in a new release. With the fix, a failed health-check update is retried until it succeeds. The provider plans to start the deployment of this release in mid-November. We are working with the provider on when the fix reaches each region. * Each load balancer now sends connections only to the servers that run our ingress servers. Before, it included a larger group of servers, so each server that we added or removed caused a health-check update. Now the health checks change only when an ingress server moves. This change is live in most regions. We are completing it in the remaining regions, including US Central and US East. * Our probes now test each load balancer address and alert when the same connections fail again and again. * We changed how we replace a faulty load balancer. We now add a new load balancer, test it, and move traffic to it. The faulty load balancer stays in service until its open connections end, so agent workers on it are not cut off. * If an agent dispatch times out in one region, LiveKit Cloud now tries the next region. * We are working on a change that moves agent worker connections out of a region when we move traffic away from it. Then all agent workers of a customer reconnect to the same region. We sincerely apologize for the disruption this caused. We appreciate your understanding and are committed to continuously improving our platform's reliability. If you have any questions, please reach out to our support team. --- [2026-10-20T17:15:32.019-07:00] (resolved) Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience. --- [2026-10-20T17:13:27.354-07:00] (monitoring) We identified connection drops at load balancers in our US Central and US East regions that caused a subset of connection latencies to take 5 seconds or longer. We applied a mitigation at 23:15 UTC, and we are seeing connection speeds return to normal. We are continuing to monitor before marking this resolved. --- [2026-10-20T17:10:47.895-07:00] (investigating) We are currently investigating higher than normal join latencies affecting a small percentage of participants in our US East and US Central regions. The latency increase appears to have begun at 21:30 UTC. Affected participants may notice increased join latencies higher than 5 seconds. The SDK retries automatically, and most of these joins succeed on retry. If your application stops waiting for an agent before then, those calls may time out. We will share another update as we learn more.