Is Lever Down?

At about 8:30am UTC on 8/25/2026 a scheduled server rotation at https://hire.eulever.co/ caused increased latency and some failed page loads until 9:15am UTC. Impacted users would notice a "504 error" during the affected page loads. Service was restored to all users as of 9:15am UTC on 8/25/2026. An full RCA will be posted to this issue within 7 to 10 business days.

Status: Resolved Our monitoring has confirmed that there are no further issues related to this incident. Service has remained stable, and the incident is now considered resolved. A Root Cause Analysis (RCA) will be provided within 7-10 business days. --- [2026-08-21T13:29:10.456-07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-08-21T13:28:59.403-07:00] (investigating) We are currently investigating this issue.

**Incident Summary** On August 12, 2026, between approximately 10:31 AM and 10:45 AM PDT, the Lever platform experienced partial unavailability. During this 14-minute window, some customers were unable to access the platform, and others experienced elevated error rates for API requests.  **Detection** The issue was identified through internal monitoring. A Datadog alert for high database page faults triggered at 10:37 AM PDT, notifying the engineering team. This allowed responders to isolate the specific database nodes experiencing high load and identify the responsible operations.  **Root Cause** The incident was triggered by a safety mechanism within an internal administrative tool designed to prevent expensive database queries. When an operator searched for unindexed data, this safety check requested a "plan" from the database to see if the query was safe to run. However, because the data was unindexed, the database attempted to evaluate every possible retrieval method, resulting in the same large account being scanned 11 times for a single request. These diagnostic checks bypassed standard time limits, consuming the database read capacity required for normal customer traffic.  **Resolution** Engineering teams identified and terminated the long-running diagnostic operations, which immediately restored database performance and platform stability. Following the cleanup, a database-level safeguard was applied to force these specific queries into a single, efficient path, preventing the database from attempting multiple expensive scans in the future.  **Preventative Measures** * **Redesigning Safety Checks:** We are replacing the current diagnostic method with a new system that verifies query safety by inspecting database definitions directly, eliminating the need to run expensive tests on live data. * **Advanced Alerting**: We are implementing new monitoring for database read churn and long-running operations to detect and resolve similar issues before they affect customers. * **Enhanced Logging:** We are improving internal audit logs to ensure faster identification and attribution of administrative database activity. * Database Optimization: We are adding new indexes to properly support these search patterns, ensuring they are inherently efficient. --- [2026-08-12T11:13:14.334-07:00] (resolved) At about 10:31am PST on 8/12/2026 Lever Support initiated a backend data query that caused Lever hire services to encounter an out of memory error. The result of this caused site degradation and failed page loads for hire.lever.co between 10:31am - 10:46am PST. Impacted users would notice a "500 error" during the affected page loads. Service was restored to all users as of 10:46am PST. At this time we are working on continued enhancements to mitigate the risk of Lever Support initiated queries triggering outages in the future. An full RCA will be posted to this issue within 7 to 10 business days.

We will be undergoing scheduled maintenance to our US database during this time. We do not anticipate any customer impact during this maintenance window.