Is Qlty Cloud Down?

Qlty services are running normally. This is a security issue in certificate trust, not an outage. We have published a security advisory with the status of each Qlty component and steps you can take now: <https://github.com/qltysh/qlty/security/advisories/GHSA-f55x-97q4-pfrc>. We will keep the advisory up to date as fixes ship, and we will stop posting updates here. --- [2026-10-21T16:08:40Z] (investigating) We are responding to the recently disclosed eMudhra certificate authority compromise: <https://emudhra.we-have-a-problem.com/>. An attacker able to intercept network traffic could impersonate trusted services to software that still accepts the affected certificates. We have released initial mitigations for our hosted analysis infrastructure. We are continuing to review and remediate Qlty services, build environments, the CLI, and bundled tooling, including installation and upgrade paths. We recommend that CLI and self-hosted CI users apply OS security updates and keep certificate verification enabled. Some developer tools use separate certificate bundles, so an OS or browser update alone may not be enough. We will publish fixed versions, upgrade instructions, and any cache refresh steps when they are available.

This incident has been resolved. --- [2026-10-08T22:17:38Z] (identified) Since 10:35 AM ET (14:35 UTC) on October 8, 2026, analysis builds fail for projects that use the Semgrep plugin. The build stops while installing Semgrep, before any analysis runs. Projects that do not use Semgrep are not affected. Coverage uploads are not affected. A workaround is available now (below). We are sorry for the disruption. **What is happening** Semgrep 1.144.0, the version Qlty installs by default, depends on a package called mcp that imports a private function from pydantic. pydantic 2.14.0, published at 10:34 AM ET (14:34 UTC) today, removed that function. Semgrep does not limit which pydantic version it installs, so every fresh install of Semgrep now picks up pydantic 2.14.0 and fails to start. The build log shows: ``` ImportError: cannot import name 'eval_type_backport' from 'pydantic._internal._typing_extra' ``` Semgrep releases 1.137.0 through 1.145.0 share this dependency and fail the same way. Releases 1.146.0 and later install correctly. **Who is affected** - Qlty Cloud: every analysis build for a project with Semgrep enabled fails while installing tools. Affected pull requests do not get an analysis result. - qlty CLI in CI, or on any machine installing Semgrep for the first time: `qlty check` fails while installing Semgrep. - qlty CLI on a machine where Semgrep was installed before 10:34 AM ET today: not affected. The installed copy keeps working until the tool cache is cleared. **Workaround, available now** in Semgrep to 1.179.0 in `.qlty/qlty.toml`: ``` [[plugin]] name = "semgrep" version = "1.179.0" ``` If your `qlty.toml` already has a `[[plugin]]` entry for `semgrep`, add or change its `version` line. Commit and push. The next Qlty Cloud build and the next `qlty check` run install 1.179.0, which works with pydantic 2.14.0. If you had already pinned Semgrep to a version between 1.137.0 and 1.145.0, you need to make this change yourself. Our fix changes the default version only and does not override a version you set. Semgrep 1.179.0 is 35 releases newer than 1.144.0. Its rules have changed in that time, so you may see new findings or lose some old ones after the switch. We chose 1.179.0 over a smaller jump because every Semgrep release between 1.146.0 and 1.178.0 depends on packages with open security advisories. **What we are doing** We are changing the default Semgrep version in the qlty CLI from 1.144.0 to 1.179.0 (https://github.com/qltysh/qlty/pull/2910). We will ship it in the next CLI release and roll it out to Qlty Cloud. After that, projects without an explicit Semgrep version build again with no change on your side. Builds that failed during the incident do not rerun on their own. Push a new commit to get a fresh result.

On October 5, 2026, between 12:18 and 12:31 PM ET, Qlty stopped completing analysis results for 13 minutes. About 120 builds across roughly 60 workspaces never completed. On the affected pull requests, the Qlty check on GitHub showed "Qlty encountered a timeout error at 15mins" while the build page on Qlty showed the build as passed. Pushing a new commit to an affected pull request produces a fresh, correct result. We are sorry for the confusion this caused. Below is what happened, who was affected, why, and what we are changing. **Timeline** (US Eastern on October 5, 2026, UTC in parentheses) • 12:17 PM (16:17): A routine data cleanup in one workspace started a large maintenance operation on our main database. • 12:18 PM (16:18): Analysis results stopped completing. Builds kept running and finishing normally, but their results were not written. • 12:30 PM (16:30): Our queue-delay alarm fired. • 12:31 PM (16:31): The operation finished and results started completing again. • 12:33 to 12:50 PM (16:33 to 16:50): The Qlty check on affected pull requests changed from pending to the 15-minute timeout error. • 12:35 PM (16:35): Result processing was back to normal. • About 1:00 PM (17:00): A customer reported a pull request stuck on the timeout error. We advised pushing a new commit. • 4:00 PM (20:00): We confirmed the root cause. **Who was affected** During the 13-minute window, every analysis result was delayed by up to 12 minutes. Most builds recovered on their own: our queue retries each result three times, and 73 builds completed on a later attempt. 118 builds did not complete after three attempts and were dropped. They span 88 pull requests and 26 default-branch pushes, across about 80 projects in roughly 60 workspaces. For each of these builds: • The Qlty check on GitHub showed the timeout error instead of a pass or fail, and the check linked to our troubleshooting docs rather than to the build. • No pull request comment or review was posted. • The build page on Qlty showed the build as passed, with no indication that its results were missing. • For the default-branch pushes, that commit is missing from the project's quality history until the next push. A smaller group of builds that did complete on a retry recorded some issues more than once, which inflated their issue counts. This affects 42 builds. We are deciding whether to correct these or leave them, since a later push replaces them. If you had a pull request open during this window and the Qlty check still shows the timeout error, push a new commit and the check will run again. **Root cause** A customer ran a routine cleanup of older data in their project. That is a normal, supported action. Behind it, Qlty removes that project's data from one of our largest tables. Because of how that table is laid out, removing one project's data required reading a large share of the whole table to find the small fraction that matched. That read did not slow the database itself as much as it slowed ClickHouse Keeper, the coordination service that tracks what data each database replica has. Every request to Keeper became about six times slower for the 13 minutes the operation ran. The final step of processing an analysis result, which computes a build's metrics, asks Keeper to confirm the replica has the latest data before the query starts. Under normal conditions that check takes under a second. With Keeper slowed, and about 40 of these steps waiting at once, each check queued behind the others and the wait grew to several minutes. Our 30-second client timeout cut each attempt short, the queue retried three times over about six minutes, and builds whose three attempts all fell inside the 13-minute window were dropped. **What we are doing** • **Making data removal a bounded operation.** Removing a project's data will cost in proportion to the data being removed, not the size of the table it lives in, so one workspace's cleanup cannot slow the service for everyone. • **Reducing the final processing step's dependence on Keeper.** We are cutting the number of coordination checks each metrics query makes, and we are designing a version of that step that does not need them at all. • **Alerting sooner.** We are adding alerts on the database timeouts that preceded the queue delay, so we see an event like this within a couple of minutes instead of twelve. If you have questions about a specific pull request or build from this window, contact us through [qlty.sh/contact/support](http://qlty.sh/contact/support) and include the pull request or build link.