Is Phrase Down?

Changes made to branches during the incident window are now fully applied. Merged branches that were affected have been identified and impacted customers have been contacted directly. This incident has been resolved. --- [2026-08-07T10:30:52.474+02:00] (monitoring) A fix has been implemented. The backlog of missing events is now being processed. Once the queue is fully processed, unmerged branches will automatically receive the missing changes. We are monitoring the queue and results. --- [2026-08-07T10:18:44.563+02:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-08-07T09:54:34.714+02:00] (investigating) We are continuing to investigate this issue. --- [2026-08-07T09:27:46.093+02:00] (investigating) We are currently investigating an issue with branching events missing in Phrase Strings (EU), which may cause changes to not be applied when branches are merged.

The incident has been resolved and all components are back to operational. --- [2026-08-06T14:00:00.840+02:00] (monitoring) We continue monitoring the situation. --- [2026-08-06T10:43:07.355+02:00] (monitoring) We have identified the root cause and the issue has now been resolved. Connectors should be appearing as active again. We will continue to monitor the situation to ensure full stability. --- [2026-08-06T10:18:29.202+02:00] (investigating) We are continuing to investigate this issue. --- [2026-08-06T10:11:39.640+02:00] (investigating) We are currently investigating an issue affecting Connectors in Phrase TMS. Customers may be unable to access or use Connector functionality during this time. Our engineering team is actively working to identify the cause and restore full service as quickly as possible.

## Introduction We would like to share details about an incident that affected Phrase Orchestrator on July 27–28, 2026. During this period, workflow executions in the Next-Gen Workflow Engine were unable to progress and remained stuck in an "executing" state. No data was lost during the incident. This post-mortem explains what happened, when it was resolved, and the steps we have taken to prevent a recurrence. ## Timeline * **27 July 2026 at 18:55 CEST** – The workflow engine began producing errors as database query performance degraded. Workflow executions stalled and stopped progressing. * **27 July 2026 at 20:54 CEST** – The first customer report of executions stuck in "executing" was received. * **27 July 2026 at 22:39 CEST** – The incident was formally declared. * **28 July 2026 at 02:18 CEST** – A service restart provided temporary relief; workflow executions resumed. * **28 July 2026 at 05:15 CEST** – The issue recurred as the underlying database performance problem persisted. * **28 July 2026 at 09:56 CEST** – The root cause was identified and addressed. No executions were lost; however, due to partial service restarts, some actions within executions were retried, which may have caused a small number of executions to fail that otherwise would have succeeded. * **28 July 2026 at 13:46 CEST** – The full backlog of stalled executions was confirmed as cleared. The system was declared stable. * **28 July 2026 at 13:48 CEST** – Incident resolved. ## Root Cause The incident was caused by progressive bloat in database indexes used by the workflow job scheduling system. The performance of these particular indexes gradually degraded over time as they accumulated dead index entries from prior writes and updates. The job scheduling engine acquires database-level coordination locks while querying these indexes to determine which jobs to dispatch. As the index lookups grew slower, they began exceeding the database's configured statement timeout. When a lookup was canceled by the timeout, the scheduling process responsible for that work crashed and restarted. With no schedulers running, no workflow steps could be dispatched and all in-progress workflow executions became stuck. The database server itself remained healthy throughout the incident, with normal CPU and connection levels. The problem was exclusively lock and latency contention within the scheduling layer. A service restart cleared the crashed processes and temporarily restored execution. However, because the index bloat was still present, the same degradation recurred once query load resumed. A manual index rebuild fully restored performance and resolved the issue. ## Actions to Prevent Recurrence 1. **Automated index maintenance added** – Scheduled automatic index maintenance has been configured for the affected indexes. This ensures bloat cannot accumulate over time and eliminates the conditions that triggered this incident. 2. **Legacy indexes removed** – Unused legacy database indexes have been identified and removed, reducing the overall maintenance surface and simplifying future index hygiene. 3. **Monitoring coverage updated** – Our monitoring landscape is being reviewed and updated to reflect the current state of the workflow engine. This work will close gaps that allowed the degradation to go undetected before the first customer report. --- [2026-07-28T13:49:12.977+02:00] (resolved) All executions have been processed. This incident has been resolved. --- [2026-07-28T10:35:07.327+02:00] (monitoring) A fix has been implemented and we are monitoring the results. The workflow engine is processing the queue of pending executions. --- [2026-07-28T10:30:40.486+02:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-07-28T09:53:55.651+02:00] (investigating) We are continuing to investigate this issue. --- [2026-07-28T09:30:42.072+02:00] (investigating) The engineering team identified an issue with Orchestrator where the new Workflow Engine is currently not executing workflows. The problem is under investigation.