Is Testsigma Down?

On October 21, 2026, the PHP project disclosed two critical vulnerabilities in the PHP runtime (CVE-2026-109618 and CVE-2026-109619, also known as "PHP2Shell"). PHP is not used in the Testsigma platform, including the application, APIs, and test execution infrastructure. At Testsigma, PHP is limited to the content management system behind our public blog. We have upgraded the PHP runtime for the blog to version 8.2.35, the patched release that fixes both vulnerabilities, and confirmed the fix is in place. We also reviewed activity on the blog infrastructure since October 17, 2026 and found no evidence of compromise. The Testsigma platform and customer data were not affected, and no action is required from customers.

This incident has been resolved. The fix has been deployed, the affected data has been cleaned up, and the platform has remained stable since. We apologise for the disruption and thank you for your patience. --- [2026-10-25T05:23:58.494Z] (monitoring) The fix has been deployed, and the platform has been stable since then. Summary: between about 02:45 UTC (08:15 IST) and 11:20 UTC (16:50 IST) on 23 Oct 2026, users in the US region may have experienced slow page loads, intermittent errors, and interruptions to some test runs. The cause was a defect that duplicated test data filter values when they were edited, combined with an inefficient way of loading those values. Opening an affected test case caused our application servers to run out of memory and restart. What we changed: Duplicate test data values are no longer created when editing filters. Loading test cases with large data filters is now much lighter on memory. The duplicated data has been cleaned up, and we've verified that affected test cases now open normally. No customer data was lost. We are continuing to monitor the platform and will mark this incident resolved shortly. --- [2026-10-25T05:23:16.500Z] (identified) We have identified the cause of the slowness and intermittent errors in the US region that began at about 02:45 UTC (08:15 IST) on 23 Oct 2026. Opening certain test cases that contain an unusually large amount of duplicated test data caused our application servers to run out of memory and restart. While servers restarted, some requests were slow, failed, or returned errors, and some test runs may have been interrupted. Our engineering team is working on a fix that prevents this data from being duplicated and makes loading such test cases much lighter. Until the fix is rolled out, users in the US region may see intermittent slowness or workload interruptions. If a test run fails to start or ends unexpectedly, please re-trigger it. EU and India regions are not affected. We will share another update once the fix is deployed. --- [2026-10-23T08:46:27.844Z] (investigating) Between 02:37 UTC (08:07 IST) and 07:46 UTC (13:16 IST) on 23 Oct 2026, customers on our US region may have experienced slow page loads, intermittent errors, and interruptions to some test runs in the Testsigma web application. Our team identified the cause as high memory usage in the application servers, triggered by a specific type of test data processing. Services have been stable since 07:46 UTC (13:16 IST) and are currently performing normally. Until a permanent fix is identified and rolled out, there may be intermittent slowness and possible workload interruptions in the US region. Our engineering team is actively working on the fix and monitoring the platform closely. Possible impact: - Slow loading or timeouts in the web application - Intermittent errors when opening test cases or test results - Some test plan runs or reruns may fail to start or end with an error Recommended action: if a test plan run or rerun doesn't start or ends unexpectedly, please re-trigger it. We apologise for the inconvenience and will post another update once the fix is rolled out.

## Summary On 14 October 2026, between 09:00 and 09:06 IST \(03:30 to 03:36 UTC\), the Testsigma web application in the US region was unavailable for approximately six minutes. A routine cleanup job caused one database server to restart, and the application servers restarted along with it. Service recovered on its own, and no customer data was lost or changed. ## Impact Only US-region accounts were affected; the EU and India regions ran normally throughout. * **What users saw:** errors or timeouts in the web app and API for about six minutes. * **Data:** nothing was lost or changed. * **Running tests:** continued without interruption, and their results were saved once the application was back. ## Timeline All times are on 14 October 2026 in IST \(UTC\+5:30\). * **09:00** — A scheduled cleanup job started removing an old test run; one US database server ran out of working memory and restarted. * **09:01** — Application health checks failed while the database was unavailable, so all US application servers restarted together. * **09:04** — The database server finished restarting and was back online. * **09:06** — All application servers were serving traffic again; service fully restored. ## Root cause The database server restarted because a single cleanup operation needed more working memory than its configured limit allowed. Our DevOps team reviewed the crash and confirmed this limit was too low for how our data is structured. **What triggered it.** Deleting an old test run also removes everything linked to it: results, settings, configuration and any re-runs, each with linked records of their own. Working through this chain in one go exceeded the database's memory limit for a single operation, which caused it to restart. **Why the whole US region was affected.** The application's health check depends on the database. While the database restarted, every application server failed that check and was restarted at the same time, turning one database restart into a short region-wide outage. **What made it more likely.** A defect in the cleanup job caused it to retry failed work many times, so these large deletes ran far more often than intended. ## Resolution Service recovered within six minutes once the database and application servers restarted automatically. The fixes below are planned and not yet applied; together they address all three causes: the low memory limit, the cleanup job's retries, and the health check that restarts every server at once. **Short-term measures.** 1. **Raise the database memory default parameter limit on all US database servers** to a level well above what our largest cleanup operations need. Because the change requires a database restart, the DevOps team will roll it out carefully: validate it first on staging databases that mirror production; apply it on a weekend during the lowest-traffic US hours, with the window announced to customers in advance; restart one database server at a time \(about 4–5 minutes each\), moving on only after the previous one is confirmed healthy; and monitor throughout, with the previous setting ready to restore if needed. 2. **Fix the cleanup job** so it no longer retries failed work repeatedly, and spread its work across the day. **Long-term measures.** 1. **Make the health check resilient** so a database issue no longer restarts every application server at once. 2. **Delete data in smaller batches** so no single cleanup operation works through a long chain of linked records. 3. **Apply the same memory-limit change in the EU and India regions**, using the same safe-window process. --- [2026-10-15T08:02:32.887Z] (resolved) On 14 October 2026, between 09:00 and 09:06 IST (03:30 to 03:36 UTC), users of the Testsigma web application in the US region may have experienced errors or timeouts in the web app and API for approximately six minutes. Test executions already running continued without interruption, and their results were saved once the application was back. The EU and India regions were not affected. No customer data was lost or changed. Cause: a routine scheduled cleanup job ran a delete that needed more working memory than the database's configured limit allowed, which caused one US database server to restart. Our automated health checks, which depend on the database, then restarted all US application servers at the same time, which briefly interrupted service for US-region accounts. Resolution: the database and application servers recovered automatically, and service was fully restored at 09:06 IST and has been stable since. The root cause has been identified. Preventive changes are planned, including raising the database memory limit during an announced low-traffic maintenance window and making our health checks resilient to a database restart. A detailed post-incident report is available on request. We are continuing to monitor the platform closely.