Is Spacelift Down?

After deploying the fix, webhooks processing has returned to normal. --- [2026-10-30T11:12:10.830Z] (monitoring) We have reverted the change that caused the problems with webhook processing and are seeing signs of recovery. We will continue to monitor it to ensure that the system has fully recovered. --- [2026-10-30T10:57:21.711Z] (investigating) We're currently investigating an issue that may result in some webhooks processing being delayed or failing to process correctly. We are working on reverting the change that caused the issue and will post an update as soon as possible.

**Summary** Between 15:31 UTC and 15:55 UTC on October 29th, runs for customers using ARM runners failed to start with the following error message: couldn't start worker: couldn't start docker container: error populating worker binary: unexpected response code when querying for worker binary at [https://downloads.spacelift.io/worker-arm64/752bab1d/spacelift-worker](https://downloads.spacelift.io/worker-arm64/752bab1d/spacelift-worker): 404 **Incident Timeline** * 15:31 UTC - A new deployment of the Spacelift backend was triggered. * 15:45 UTC - We triggered a rollback to the previous version. * 15:50 UTC - The fix began to take effect. * 15:55 UTC - The fix was fully rolled out. **Root Cause** The problem was caused by our build pipeline failing silently to upload that single file. When we triggered a new deployment of the Spacelift backend, new runs switched over to the new worker version and began to fail because the file was missing. **Mitigations** * Immediate: we rolled back to a previous working version. * Permanent: we have updated our deployment pipeline to ensure that it cannot silently fail to upload assets like this in future. --- [2026-10-29T16:40:13.386Z] (resolved) This incident has been resolved. Runs in the EU region are starting normally again. If any of your runs failed during this period, please retrigger them --- [2026-10-29T16:24:34.889Z] (monitoring) A fix has been applied and runs in the EU region are working again. We are monitoring the situation to make sure all runs start as expected --- [2026-10-29T15:48:12.658Z] (investigating) We are investigating an issue where some runs fail to start with the error "couldn't start worker: error populating worker binary". Our team is actively working on this.

**Root Cause** A change in a new launcher version caused it to connect by default to the EU endpoint if the region was not explicitly configured. Workers registered in other regions were then rejected at authentication. This was a human error. Detection was delayed for three reasons: * Our pre-release testing did not cover workers outside the EU region well enough. * Launcher authentication errors usually come from individual customer misconfiguration, so the increase was not flagged as a platform issue. * MQTT disconnections coincided with our planned MQTT-to-polling migration, which made the drop look expected. **Mitigations** * **Immediate:** We rolled back to a working version of the launcher * Before every deployment, we will now boot fresh private workers in other regions and confirm that they connect and complete a run. A failure like this should be caught before a release. * We are improving alerting so that authentication failures across multiple customers are flagged automatically as a potential platform issue. **Conclusion** A broken binary caused launchers without a configured region to target the wrong endpoint. Gaps in regional test coverage and alerting meant we did not catch it before customers did. Additional automated checks are being put in place so that this failure would be detected before release along with improved alerting so that we are notified sooner for issues that make it past our automated checks. --- [2026-10-21T01:26:09.607Z] (resolved) This incident has been resolved. --- [2026-10-21T01:10:50.651Z] (monitoring) We deployed the fix and monitoring. --- [2026-10-21T00:17:33.280Z] (identified) We have the fix implemented and we will release it shortly. --- [2026-10-21T00:03:47.073Z] (investigating) We’re seeing an increased number of errors affecting the worker polling API for customers in our US region. Our team is actively working on mitigation and will share updates as they become available.