Is LiveKit Down?

This incident has been resolved. Between 19:45 and 20:00 UTC, calls crossing the US West <-> US Central link experienced intermittent degradation in Room Service and SIP performance. The underlying network between the two regions had episodes of high packet loss during this time period. We rerouted traffic and performance has returned to baseline as of 22:00 UTC. We will follow up with a postmortem, which will include further details regarding scale and scope. --- [2026-08-07T14:57:31.659-07:00] (monitoring) We have applied a mitigation to the affected regions and are now monitoring to ensure that performance signals return to baseline. We are still working on determining impact (including how users can determine if they were affected) and will share those details as soon as possible. --- [2026-08-07T14:26:40.431-07:00] (investigating) We are currently investigating automated alerting which triggered for degraded Room service performance in the US West and US Central regions beginning around 19:45 UTC. We are working to determine if there is any customer-facing impact. If there is impact, we don't currently have reason to believe that it is widespread. We will update again within 30 minutes.

Today there was an issue with one of our GPU providers that caused it run at about 50% capacity. Requests above capacity return 503s, which we route to another, fallback provider. This fallback provider was not able to rise to the occasion and also returned 503s. In the very near future we will be onboarding additional providers to avoid these kinds of issues. --- [2026-08-06T14:13:58.336-07:00] (monitoring) A burst of requests increased the failure rate. Both primary and secondary model providers failed to fulfill the requests at the time; we are root-causing the issue. We are continuing to monitor. --- [2026-08-06T12:53:04.320-07:00] (identified) We are actively investigating elevated error rate on LiveKit Inference (google/gemma-4-31b-it) --- [2026-08-06T12:47:33.737-07:00] (identified) Between approximately 18:45 and 19:15 UTC today, a subset of LiveKit Inference requests using the google/gemma-4-31b-it model returned errors. Affected agent sessions would have seen an LLM request error on those requests. Other models and other LiveKit services were not affected. No action is needed on your part. If you continue to see errors, please reach out to support. We apologize for the disruption.

This incident has been resolved. --- [2026-08-04T13:45:37.299-07:00] (monitoring) We are monitoring elevated timeouts that affected LiveKit Inference between 14:05 and approximately 15:00 UTC today. During that window, roughly 3% of LLM requests using the google/gemma-4-31b-it model timed out. Impacted customers would have seen LLM request timeouts in their agent sessions (for example, APITimeoutError in agent logs). Other models were not affected. We have identified the cause. During a period of increased traffic, response times for this model slowed, and a defect in our health-monitoring logic incorrectly marked a backup deployment as unhealthy and removed it from rotation, preventing requests from failing over as designed. Error rates returned to baseline by approximately 15:00 UTC, and a fix for the health-monitoring defect is in progress. We are continuing to monitor before marking this resolved. No action is needed on your part. If you continue to see timeouts, please reach out to support. We apologize for the disruption.