Is Cycode Down?

The system is back to being fully operational. --- [2026-08-24T12:29:48Z] (investigating) We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times. --- [2026-08-21T09:23:42Z] (resolved) Summary **Problem:** We observed a delay in processing status updates for GitHub checks due to a backlog in our event processing system. **Impact:** A small number of pull requests (PRs) experienced delays in status check updates. While the majority of the system remained functional, a specific segment of the processing queue was affected, causing a lag for those specific updates over several hours. **Cause:** The delay was caused by a high volume of retries generated by unrecoverable permission errors on a limited number of requests, which blocked a portion of our processing capacity. This occurred during a period of high event volume, which led to the database reaching maximum capacity and slowing down the overall processing workflow. **Resolution:** The issue has been resolved. We scaled up our database capacity, increased the number of processing units to handle the higher volume, and implemented logic to stop retrying updates that fail due to permanent permission errors. All backlogs have been cleared, and system performance has returned to normal for all pull requests. Key Timeline (IDT) • **August 21, 2026, 06:30 IDT:** An incident was declared as delays in status updates for a subset of requests became visible. • **August 21, 2026, 06:50 IDT:** A public status page was published to inform customers of the degradation. • **August 21, 2026, 07:15 IDT:** The database was scaled up to address processing bottlenecks. • **August 21, 2026, 07:45 IDT:** Processing capacity was doubled by increasing the number of active queue partitions. • **August 21, 2026, 08:10 IDT:** A corrective update was deployed to stop unnecessary retries on unrecoverable permission errors. • **August 21, 2026, 08:45 IDT:** The event backlog was fully drained, and processing returned to real-time. • **August 21, 2026, 09:00 IDT:** The incident was officially resolved. Root Cause The incident was caused by a combination of high event volume and a lack of "fail-fast" logic for permanent errors. Specific requests with incorrect permission settings generated a massive volume of retry attempts. Because the system treated these permanent permission errors as temporary, it repeatedly retried them, which blocked a specific processing partition and prevented updates for a small number of pull requests from being processed. This was further exacerbated by the database reaching its resource limits, creating a temporary slowdown in the update workflow. Actions Taken • **Scaled Database Resources:** Increased the capacity of the primary database to handle the increased load and improve processing speed. • **Increased Processing Parallelism:** Doubled the number of active partitions in the event queue to ensure all available processing units were utilized. • **Deployed Retry Exclusion Logic:** Updated the system to identify unrecoverable permission errors and stop retrying them, preventing queue blockages. --- [2026-08-20T16:04:54Z] (monitoring) Scan processing delays have been draining quickly, and most scans have not been affected. A small number of pull requests may still experience delayed scan processing. We’re continuing to monitor recovery and will share another update as we learn more --- [2026-08-20T15:48:17Z] (identified) GitHub scan processing and status-check updates remain delayed, affecting all scan types. We have made changes to improve processing capacity and are monitoring recovery as the backlog decreases. --- [2026-08-20T14:52:37Z] (investigating) We’re currently experiencing some delays in GitHub scan processing. Our team is investigating the issue and working to restore normal processing times.

**1. Summary** On August 19, 2026, some customers experienced slower page loading and data display, as well as delays when starting or completing pull request scans. The disruption was caused by a managed memory database entering a repeated restart cycle and not recovering automatically. This created a temporary slowdown in processing. Expected service behavior was restored after corrective actions, including replacement and increased capacity for the affected service. The provider is continuing to investigate why the service was unable to return to normal operation automatically. **2. Key Timeline (IDT)** • August 19, 2026, 16:10 IDT: We identified performance degradation affecting parts of the platform and pull request scanning. • August 19, 2026, 16:30 IDT: An initial corrective change was applied and service behavior was monitored. • August 19, 2026, 16:46 IDT: We confirmed that some delays were continuing and expanded the investigation. • August 19, 2026, 17:14 IDT: We began working with our service provider to investigate instability in the managed memory database. • August 19, 2026, 18:46 IDT: Recovery actions were in progress, with temporary mitigations in place to reduce customer impact. • August 19, 2026, 19:06 IDT: The managed memory database capacity update completed, and platform responsiveness and pull request scanning returned to expected behavior. **3. Root Cause** The managed memory database entered a repeated restart cycle and was unable to recover automatically. This caused delays in the processing systems that support platform responsiveness and pull request scans. The affected service underwent a routine replacement process, followed by a capacity increase during the recovery effort. While the capacity increase completed successfully and restored expected behavior, AWS is still researching why the service did not come back up normally after the restart cycle. Their detailed root-cause analysis is pending. **4. Actions Taken** • Applied corrective updates to reduce immediate processing demand. • Temporarily adjusted the status update process to reduce reliance on the affected service. • Restarted affected processing components to restore responsiveness. • Worked with the service provider to replace affected capacity and complete a capacity increase. • Monitored platform performance and scan processing until expected behavior was confirmed. **4b. Action Items** • Review the service provider’s detailed root-cause analysis when available. • Improve monitoring to detect similar recovery delays earlier. • Add safeguards to reduce the impact of temporary processing slowdowns. • Introduce separate Redis capacity for each service to isolate workloads and reduce the impact of an issue in one service on others. --- [2026-08-19T17:32:50Z] (monitoring) The remaining processing delays have cleared, and the Cycode application is operating normally. We are continuing to monitor the service. --- [2026-08-19T17:00:58Z] (monitoring) The Cycode application remains stable and page loading is operating normally. Some pull request scans may still be delayed while we continue clearing the remaining backlog and monitor recovery. --- [2026-08-19T16:25:44Z] (monitoring) The Cycode application is stable and page loading is operating normally. Some pull request scans are still experiencing delays. We are continuing to monitor the service and work to clear the remaining scan backlog. --- [2026-08-19T16:02:45Z] (monitoring) We are monitoring stability of the affected memory database service. It currently appears stable, but our provider has not yet confirmed full recovery. We will share another update once recovery is confirmed. --- [2026-08-19T15:54:12Z] (identified) Some customers may continue to experience slower response times across the Cycode application. This can include delays loading pages, viewing data, and starting or completing pull request scans. We have identified instability in an underlying data service that is affecting request processing. Our infrastructure provider is actively working to restore the affected service; recovery has not yet completed successfully. We have applied temporary changes to reduce demand on the service while we monitor its stability and work with the provider on a permanent resolution. We will share another update as soon as we have confirmed service stability or have additional information to report. --- [2026-08-19T14:16:44Z] (investigating) We're investigating potential issue with one of the infrastructural services with infra provider support team. --- [2026-08-19T13:28:24Z] (monitoring) System response times are back to normal. We're monitoring system state. --- [2026-08-19T13:18:38Z] (investigating) Cycode experiences increased response time for selected requests. We're investigating the issue.

GitLab component: Website, API, Git Operations, CI/CD Original GitLab incident: <https://status.gitlab.com/>