Is Captivated Down?

## Summary On Friday, October 23, 2026, real-time chat and in-app updates in Captivated were unavailable from about 10:58 AM to 12:15 PM ET. The outage was caused by our own infrastructure, not AWS. A supporting system the app relies on ran out of capacity after an expired credential caused a buildup of failed background jobs. We restored service in about 75 minutes by increasing that system's capacity. This outage was preventable, and we own it. Our customers trust Captivated to be available when their businesses need it, and on Friday we didn't meet that standard. Below is what happened, what we've already fixed, and what we're changing so this type of failure can't happen the same way again. ## Customer impact * **Real-time chat and in-app updates:** unavailable Friday, October 23, from about 10:58 AM to 12:15 PM ET. * **iOS push notifications:** not delivered from Thursday, October 22 at about 11:00 AM to Friday, October 23 at about 10:00 PM ET \(about 35 hours\). * **Scheduled messages:** 145 scheduled messages failed to send during the outage. We reviewed every one. 113 were resent or replaced by a similar message the sender sent afterward, and in the remaining 32 conversations the sender followed up with at least one more message, so no conversation was left without a follow-up. * **Sign-in and reconnection:** from about 11:20 AM to 12:15 PM ET, some customers on shared office networks were temporarily blocked by our security protections, which reacted to the surge of reconnection attempts. ## What happened On Thursday morning, a credential we use to send iOS push notifications expired. Every iOS notification after that failed and was automatically retried. The data from those retries built up in a supporting system that was already running close to its capacity, and we did not have an alert on its usage. By Friday morning that system was full and began rejecting new data. The service that powers real-time chat didn't handle those errors gracefully and restarted repeatedly. The surge of devices trying to reconnect added further load until we increased capacity. ## Timeline \(ET\) * **Thu 10/22, ~11:00 AM:** The iOS push credential expires and iOS notifications stop delivering. Usage in the supporting system begins to rise. * **Fri 10/23, ~10:58 AM:** The system reaches capacity. Real-time chat and in-app updates become unavailable. * **Fri, ~11:20 AM:** Some traffic from shared office networks is temporarily blocked during the reconnection surge. * **Fri, 11:41 AM:** We begin a capacity upgrade. The system stays online during the change. * **Fri, ~12:15 PM:** The upgrade completes and service is restored. * **Fri, ~10:00 PM:** The push credential is renewed and iOS notifications resume. * **Mon 10/26:** The system has run stable at under half of its capacity since Friday, with no recurrence. ## What we've already done * Increased capacity on the affected system, which now runs at under 50% usage. * Renewed the iOS push credential. * Added an alert that notifies our team well before this system nears capacity. * Reviewed and accounted for all 145 affected scheduled messages. ## What we're doing next * **Preventing the trigger:** moving iOS push notifications to an authentication method that doesn't expire, and limiting how many times a failed notification is retried. * **Keeping capacity healthy:** setting limits on how long temporary data is kept, and keeping usage well below capacity. * **Failing gracefully:** updating our real-time chat service so it keeps running through this kind of error instead of crashing. * **Protecting scheduled messages:** changing scheduled sends so a failure is retried or raises an alert instead of being dropped silently. * **Better monitoring:** expanding monitoring across our infrastructure, reviewing our alerts so the important ones get immediate attention, and adding automated uptime measurement of our core functions \(signing in and using the app, real-time chat, messaging, and API access\). * **Faster response:** writing an incident runbook and training more of our team to respond to production issues. We're sorry for the disruption this caused you and your customers. If you have any questions, please reach out to our support team. --- [2026-10-28T13:52:45.068-04:00] (resolved) On Friday, October 23, from about 10:58 AM to 12:15 PM ET, the Captivated app was unavailable or degraded for customers. The cause was our own infrastructure, not AWS. A supporting system ran out of capacity after a credential expired. We restored service in about 75 minutes and increased capacity on the affected system. We're also putting changes in place so this type of failure can't happen the same way again, and adding automated uptime measurement. We apologize for the disruption and thank you for your patience.