Is Walcu Down?

## Post-Mortem: Northflank Security Incident **Date:** October 30, 2026 **Status:** Resolved ### Summary On October 30, our infrastructure provider, Northflank, experienced a security incident involving potential exposure of credentials stored on its platform. Upon receiving Northflank's notification, Walcu immediately initiated security measures, including rotating affected passwords, API keys, and access tokens. These actions required restarting and reconfiguring some of our services, resulting in temporary service disruptions. ### Incident Timeline All times are CET \(Madrid time\). * **October 30, 14:40:** Northflank notified Walcu of the security incident and instructed us to begin rotating credentials. * **October 30, 14:45:** Walcu began rotating affected credentials and securing its services. * **October 30, 18:00:** Most Walcu services were successfully restored. * **October 31 \(Saturday\):** Remaining email synchronization issues affecting some accounts were fully resolved. * **November 2 \(Monday morning\):** The remaining affected OEM integrations were restored. ### Security Impact Following our investigation and review of system access logs, **we found no evidence of unauthorized access to customer data. No personal data breach was identified.** ### Resolution All affected credentials have been rotated, and all Walcu services have been fully restored. We continue to monitor the situation and follow Northflank's security recommendations. For technical details about the underlying incident, please refer to [Northflank's official incident report](https://status.northflank.com/cmvv0rn5a092f14rutwccodpp). We apologize for the disruption and appreciate your understanding. **The Walcu Team** --- [2026-10-30T17:01:52.968+01:00] (resolved) This incident has been resolved. --- [2026-10-30T14:58:21.297+01:00] (investigating) Walcu CRM is currently unavailable. We are working to get it back up and running as soon as possible.

# Incident Report: Authentication Service Outage — 26 August 2026 _Published: 28 August 2026 · Status: Resolved_ ## Summary On Wednesday 26 August 2026, between 07:50 and 09:16 UTC, the authentication service that all Walcu applications depend on became unresponsive. Because every Walcu application relies on this service to sign users in and validate sessions, the platform was intermittently inaccessible for the duration of the incident. The service recovered at 09:16 UTC after we increased its capacity. The underlying cause — a configuration bug that was routing authentication traffic to a legacy, under-provisioned environment — was fixed permanently at 09:30 UTC, shortly after recovery. We are sorry for the disruption. This was a significant outage during working hours and it affected every one of our customers. ## Impact * **Who:** All users, across all Walcu applications. * **What:** Sign-in and application access failed intermittently. Because authentication is required on every request, users experienced this as the platform being unavailable or unreliable rather than as a login-only problem. * **Duration:** Approximately 86 minutes \(07:50–09:16 UTC\), with intermittent availability throughout. * **Data:** No data was lost or corrupted. This was an availability incident only. ## Timeline \(all times UTC\) | Time | Event | | --- | --- | | 07:50 | The authentication service becomes unresponsive. Walcu applications begin failing intermittently for all users. | | 07:55 | Incident detected. Automated alerts fired, and we received reports from customers and from internal users at close to the same time. | | 07:55 – 09:00 | Investigation. The service remains intermittently available while engineers work to identify the cause. | | 09:00 | First mitigation: capacity increased on the authentication service running on our current platform. No improvement — that instance was not the one actually receiving the traffic. | | 09:15 | Second mitigation: capacity increased on the authentication service still running on our legacy platform, which was serving the requests. | | 09:16 | Significant recovery observed. Applications return to normal operation. | | 09:30 | Permanent fix deployed: the configuration bug was corrected, moving all authentication traffic to the current infrastructure. | ## Root cause Earlier this year we began migrating our infrastructure to a new hosting platform. During a migration of this kind both environments run in parallel, with the legacy environment kept alive at reduced capacity to serve a small number of remaining services. On 31 July, a deploy introduced a bug in how our applications resolve the address of the authentication service. From that point on, our main application servers were sending authentication requests to the **legacy** authentication service on the old platform, rather than the one on the new infrastructure. The legacy instance had been deliberately scaled down, because it was only expected to receive residual traffic from a handful of small services. This went unnoticed for almost four weeks for two reasons: 1. **August is our lowest-traffic month of the year.** The reduced instance was able to absorb the misrouted load without visible degradation. 2. **Nothing alerted on the misrouting itself.** Requests were being served successfully, just by the wrong environment. From a functional perspective the system looked healthy. On 26 August, two things changed at once. Traffic was climbing back toward normal levels as the month ended, and a deploy caused a large number of users to reload the application simultaneously, producing a burst of authentication requests. The under-provisioned instance could not absorb this. It exhausted its available memory and entered a garbage-collection-heavy state in which it spent most of its time reclaiming memory rather than serving requests, and became unresponsive. Because the service was memory-starved rather than simply busy, restoring it required more memory, not more time — which is why the outage persisted until capacity was increased. ## What went well Detection was fast and came through several independent channels, which gave us early confidence that this was a real, platform-wide outage. Once the failure mode was understood, recovery followed within a minute of applying the fix. ## Action items We are in the process of decommissioning the legacy infrastructure entirely, so that no request can be served by it and any code still pointing at it fails loudly and immediately. This work is already underway and we expect the migration to be fully complete by the end of September 2026. It removes the class of problem that caused this incident. _If you have questions about this incident, please contact_ [_support@walcu.com_](mailto:support@walcu.com)_._ --- [2026-08-26T11:45:57.552+02:00] (resolved) This incident has been resolved. --- [2026-08-26T11:18:15.128+02:00] (monitoring) We have identified the root cause of the issue and applied a mitigation. We're seeing improvements in all afected systems and seeing signs of recovery. --- [2026-08-26T10:58:03.764+02:00] (investigating) We are currently investigating a performance issue affecting the platform, which may cause slow load times, interface errors, or unexpected user session terminations. Our engineering team is actively working to identify the cause and restore full service as quickly as possible.