Is Siit Down?

The issue is now fixed, the postmortem of the incident is available below. -- **Summary** On the morning of October 8, Siit stopped processing most of our background tasks between about **07:45** and **09:40** CEST. During that time, notifications, emails, Slack and Teams messages, workflows and integration syncs were delayed. Some interactive actions, such as clicking a button or talking to the AI Agent in a Slack or Teams message, couldn't be completed and may have needed a retry. While we caught up on the delayed work that had piled up, the app was slower than usual until about **11:00 CEST**. All delayed work was queued safely and processed once service was restored. **No data was lost.** **What happened** A routine configuration cleanup after a recent release left our background processing unable to restart during scaling phases. The effect built up gradually overnight and became visible in the morning when activity picked up. **How we resolved it** We found the cause a few minutes after the investigation started and restored processing, handling the most time-sensitive tasks first. Catching up on the backlog put **heavy load** on our database, so we added database capacity and spread out the remaining work. Everything was back to normal by **11:00 CEST**. **Where we failed** We had no alert for this specific failure, and the alerts we did have relied on the affected system itself As a result, the problem went unnoticed until it impacted you. **What we're changing** • We are revamping our Alerting, so such cases are caught right away. • Stricter safeguards and automated checks on production configuration changes. • We’ve upgraded our database capacity • Improving out-of-hours alerting and escalation We're sorry for the disruption to your teams. If an action from that morning still looks incomplete, retrying it should work. If not, our support team is happy to help. --- [2026-10-08T09:01:17Z] (monitoring) All our monitoring data now appears back to normal, we will continue monitoring and provide a post mortem shortly. --- [2026-10-08T08:50:41Z] (monitoring) We are still having impactful latency until we reach full recovery --- [2026-10-08T07:54:07Z] (monitoring) Our fix appears to be working - our process are still catching their queue so you could still expect some latency in the delivery of pilled up notifications. We’ll come up with a detailed reports once we confirm that things settled down for good --- [2026-10-08T07:42:10Z] (investigating) We have pushed a fix and expect recovery to happen during the next 10-40 minutes --- [2026-10-08T07:31:20Z] (investigating) We are receiving multiple reports and alerts of issues affecting our app

**Summary** Between the evening of 16 September and the morning of 17 September, Siit was intermittently slow and some requests failed. The admin dashboard was most affected. Two short interruptions of a few minutes each occurred on 17 September while our primary database failed over and recovered. The platform has been stable since _10:50 CEST on 17 September_. **Impact** • _16 September, 20:20 to 00:15 CEST (18:20 to 22:15 UTC)_: intermittent slowness in the admin dashboard, with occasional requests timing out. • _17 September, 08:30 to 10:50 CEST (06:30 to 08:50 UTC)_: sustained slowness. Requests from the admin dashboard, the employee portal and the Slack and Teams integrations could take many seconds or fail with an error. • _17 September, 09:19 to 09:31 CEST and 10:36 to 10:47 CEST_: two brief interruptions of all write operations while the primary database failed over and returned. • _17 September, 10:18 to 10:35 CEST_: some Slack/Teams interactions and integration webhooks were not processed on first attempt. They were retried automatically. **No data** was lost or corrupted. Requests that failed returned an error to the user and were not partially applied. Automations and integrations that could not run were queued and completed once the platform recovered. **What happened** A recent change introduced real-time refresh of the admin dashboard, so that changes made by other admins or by automations appeared without a manual reload. Under real-world conditions, especially during high traffic / compute periods, this feature generated far more database read traffic than our testing had shown. **How we resolved it** - We rolled back the real-time refresh feature. The admin dashboard has returned to refreshing on navigation and auto/manual pulling for now. - Restored the primary database and increased read replicas capacity. - Reduced the number of open database connections from our application servers. **What we are doing next** - Redesigning the real-time refresh feature to batch updates on the server before it is re-enabled - Tightening the safeguards on our database autoscaling and improving our overall architecture to ensure more resilience in the future We are sorry for the disruption. If you noticed an impact we have not described here, please contact [support@siit.io](mailto:support@siit.io). --- [2026-09-17T08:51:33Z] (monitoring) Our infra seems to be back on tracks, we are monitoring for results and will provide more information once we confirm after a while. --- [2026-09-17T08:39:22Z] (monitoring) Our jobs are pilling up, we are still applying some changes to fix the underlying issue and we expect to come back to normal in a few minutes. You might notice errors with our agents in the meantime until our queues has fully catched up. --- [2026-09-17T07:42:16Z] (monitoring) We applied some configuration changes on our database cluster and are monitoring for recovery --- [2026-09-17T07:25:50Z] (identified) The load and response time have been steadily increasing this morning and we're on it --- [2026-09-16T23:20:06Z] (monitoring) We have identified the issue, applied a fix and we are seeing a noticeable decrease in response time. We'll keep monitoring for now. --- [2026-09-16T22:13:41Z] (investigating) We are investigating slower than usual response time and are investigating the reason

This incident is now resolved. It was caused by an infrastructure configuration change that had unintended side effects. The change has been rolled back and we've added safeguards so it can't happen again. Apologies for the disruption and thank you for your patience. --- [2026-09-10T14:43:49Z] (monitoring) We have deployed a fix and are monitoring --- [2026-09-10T14:32:14Z] (investigating) We are experiencing an issue with our infra and our team is on it