Is Firstup Down?

From 1:30 to 3:30 PM PT on Friday, October 23rd some users may have experienced intermittent technical difficulties errors. This was determined to be due to a database caching server in use by a subset of Dynamic communities, and the problem was mitigated through a restart of connected web servers to restore steady state.

**Summary:** On October 6th, 2026, at 1:00 PM PT, our monitoring alerted us to a high number of connection errors in the production environments shortly after a hotfix was deployed. Preliminary engineering investigations coupled with internal and external reports of platform loading errors confirmed platform reachability issues. A platform incident was declared at 1:11 PM PT, and the hotfix was rolled back, along with a targeted code fix deployment to address the underlying code error. All services were fully restored by 2:09 PM PT the same day. **Severity:** Sev-1 **Scope:** All Firstup customers were affected, with customers in US2 and EU platforms experiencing a more prolonged service disruption due to circular restart dependencies that had to be addressed separately from the hot fix rollback. Customers in the US1 platform experienced a brief disruption of service primarily accompanied by degraded application responsiveness. **Impact:** * **US1:** From 1:00 PM PT to 1:19 PM PT, service was degraded and the platform processed up to 70% less work than normal. * **US2:** From 1:00 PM PT to 2:09 PM PT \(69 minutes\), users couldn't log in to Studio, Employee Experience or the support portal, and extended message delivery times. * **EU:** From 1:00 PM PT to 1:26 PM PT \(26 minutes\), users had the same problems as in US2. **Root Cause:** The root cause for this incident was identified to be the deployment of a hotfix that was built on an updated shared base software image. That image included newer networking and encryption libraries that don't work with the web server setup of one of our core services, so the service crashed. While this problem had already been identified and protected against in our regular release processes, it had not been protected against in the hotfix deployment process branch. Our systems and services health checks didn't catch the failure until the healthy copies of the impacted service had already been replaced. Additionally, startup dependencies between two backend services increased the time to recovery and was more pronounced in the US2 platform. **Mitigation:** In US1 and EU, the deployed hotfix was rolled back and services were restored by 1:19 PM PT and 1:26 PM PT, respectively. In US2, and in addition to the hotfix rollback, we deployed a code fix that: * removed the startup dependency between the two backend services * locked the base software image to the last working version This fully restored service by 2:09 PM PT. The same fix then went to all regions in a scheduled release at 6:50 PM PT on October 6th, 2026, with no customer impact. **Recurrence Prevention:** To prevent this incident from happening again due to the same root cause, Firstup has done \(or committed to doing\) the following: * Lock and version shared base software images in every build process, including hotfixes, so that what is test is exactly what is released. * Roll out changes in stages, starting with one region or part of capacity, so problems are caught before they reach every region. * Strengthen health checks so they confirm the service is actually working before it takes traffic. * Removed the identified startup dependencies between the two core services in the US1 and EU platforms by deploying the code fix during our maintenance window on October 6th, 2026. * Conduct a review of any remaining dependencies for similar risks. * Tighten the hotfix process. Hotfixes that aren't deployed promptly will need re-approval, and related changes across services will be deployed together. * Update monitoring and recovery runbooks in every production region. --- [2026-10-06T22:47:57.354Z] (resolved) This incident has been resolved. --- [2026-10-06T21:37:30.665Z] (monitoring) Monitoring has confirmed all regions remain health, and the incident has now been fully resolved. --- [2026-10-06T21:15:36.027Z] (monitoring) Impact on US2 has been mitigated, and remaining services are fully operational. Actively monitoring all regions to ensure steady state. --- [2026-10-06T21:00:06.097Z] (identified) New user sessions cannot be created still on US2 region, but other services have recovered. Users with existing active sessions should be able to access and interact with programs hosted on that region. --- [2026-10-06T20:32:21.909Z] (identified) EU region impact mitigated and fully recovered. Continuing to monitor region health. Work continues to mitigate impact on US2. --- [2026-10-06T20:24:57.964Z] (identified) Root cause has been identified. Impact on US1 region should be mitigated, and full recovery in process. US2 and EU regions are being investigated now. --- [2026-10-06T20:20:53.295Z] (investigating) We are currently investigating an elevated error rate when loading employee experience causing slow page loads and intermittent timeouts.

This incident has been resolved, and all impacted components have remained stable and fully available. A Root Cause Analysis for this incident will be provided as soon as the incident postmortem has been concluded. --- [2026-10-01T21:07:32.603Z] (monitoring) A fix for this issue has been released. A refresh of the Employee Experience page may be required to reload the shortcuts. We will be placing the impacted components under monitoring for now. --- [2026-10-01T20:48:18.813Z] (identified) We have identified a potential root cause for this issue and are working to mitigate it. Another update will be provided within 1 hour. --- [2026-10-01T20:40:07.052Z] (investigating) We are currently investigating multiple reports of shortcuts not rendering as expected for programs hosted on US2. No other regions are impacted at this time.