Is Flying Circus Down?

The upstream provider has acknowledged another technical defect on the line and has fixed it. We're waiting for a more detailed RFO and will use that to decide on steps for future reliability improvements on our backup DC uplink. --- [2026-11-13T13:41:12.018+01:00] (monitoring) Customer-specified service checks have been re-enabled. --- [2026-11-13T12:50:30.042+01:00] (monitoring) As there was still no contact to our upstream network provider, we are leaving Customer-specified service checks (as shown on https://my.flyingcircus.io) disabled for now to avoid false alerts. All other services remain available for now, consider the overall situation still unstable though. --- [2026-11-13T12:41:54.410+01:00] (monitoring) The network link came back online, all affected services are available again. The upstream provider has not responded yet, the reason for outage is still unclear. --- [2026-11-13T12:25:54.465+01:00] (investigating) These are the central services affected by the network failure: * Our emergency hotline (partial, backup number still available) * Emergency emails * Customer-specified service checks (as shown on https://my.flyingcircus.io) * Our Matrix chat * gitlab.flyingcircus.io (which may affect automated deployments) * hydra.flyingcircus.io (which may affect automated deployments and system updates using "fc-manage") --- [2026-11-13T12:25:51.665+01:00] (investigating) We suspect that today's upstream provider repair was not effective and that the intermittent outages from the last days have now lead to a major impact. We have reached out and are waiting for feedback from their service team. --- [2026-11-13T12:24:25.754+01:00] (investigating) Our backup data center RZHAL is currently offline. Production workloads running in our main data center RZOB are not affected, but monitoring and alerting are affected.

This incident has been resolved. --- [2026-09-18T10:30:35.975+02:00] (monitoring) During deployment of DNSSEC a wrong key announced. This prevented further DNS resolution. The deployment was reverted. Resolution will be degraded until DNS caches have been expired. --- [2026-09-18T10:27:27.107+02:00] (investigating) We are seeing DNS resolution errors for flyingcircus.io (and only this domain). Customer domains and services are not affected.

Things have been stable and we've implemented temporary measures to compensate for the failed PSUs, with replacement and additional spare parts being on their way with express shipments. We're closing this incident. --- [2026-08-28T12:33:58.110+02:00] (monitoring) The cluster has recovered and we have performed an initial analysis: Our team was performing routine maintenance work and shut down a storage server to check for a suspected defect power supply (PSU). While reattaching the PSU after inspecting it, this triggered a circuit breaker at around 11:30 and loss of one phase in a single rack. Unfortunately a neighbouring storage server also exposed a hidden defect power supply connected to the redundant power phase and thus caused the redundancy in the storage cluster to drop below safe levels for serving IO and thus started blocking requests. We quickly rebooted the affected servers and IO resumed after around 10 minutes at 11:40. A few lagging requests needed to be unblocked manually which finally resolved around 12:10. In-depth review will happen in the next days and we'll revisit our processes for potential improvements. --- [2026-08-28T12:01:10.669+02:00] (identified) Due to an electrical issue during maintenance work, we had triggered an overcharge protection, setting multiple storage nodes out of service. Most hosts came back online, the cluster has recovered and service is restored. We are working on getting the remaining host back. --- [2026-08-28T11:46:23.532+02:00] (investigating) There was a timeout in the storage cluster causing some VMs to block on storage access.

Release 2026_048 is ready and will be rolled out during the specified timeframe. See the <a href="https://doc.flyingcircus.io/platform/changes/2026/r048.html">Changelog</a> for information about the specific changes and impact of the release. Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime. Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.