Is Mixpanel Down?

This incident has been resolved. The upstream issue with our cloud storage provider has been fixed, and our ingestion pipeline has been operating normally with no active delays. No events were lost throughout this incident, all affected data was fully reprocessed. Thank you for your patience throughout this incident. --- [2026-10-08T16:30:13.748-07:00] (monitoring) Systems have remained stable with no recurrence of the ingestion delay. We’re keeping a close eye on things while we wait for final confirmation from our upstream provider. --- [2026-10-08T15:30:30.626-07:00] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-08T14:30:17.391-07:00] (monitoring) No change since our last update, we continue to observe no active ingestion delay on our end, and all systems remain healthy. We’re continuing to monitor closely. --- [2026-10-08T13:30:56.788-07:00] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-08T12:30:32.487-07:00] (monitoring) Our metrics show ingestion is caught up and no longer experiencing delay. Google has not yet confirmed resolution of the underlying upstream issue on their end, so we’re continuing to monitor closely in case of recurrence. --- [2026-10-08T11:31:13.215-07:00] (identified) We've identified the cause of the ingestion delay as an upstream issue with our cloud storage provider, Google Cloud, impacting a subset of US projects. Google has confirmed this as a high-priority issue on their end but hasn't provided an ETA yet. We've deployed mitigations to ensure a smooth recovery once it clears. No events are being lost. This is strictly an ingestion delay, all affected data will continue to be reprocessed once the issue is resolved. --- [2026-10-08T10:32:24.365-07:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-10-08T09:32:32.028-07:00] (investigating) We are currently experiencing delays in our data ingestion pipeline, which is affecting real-time data for projects enrolled in the US projects, resulting in delays for a subset of projects. While no data is being lost, our engineering team is actively investigating the matter and working to restore real-time functionality as quickly as possible. We appreciate your patience during this time. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support.</a>

# Mixpanel RCA: Temporary Query Service Disruption for US Projects August 26, 2026 # Summary Between approximately **2:02 PM and 3:19 PM PT on August 26, 2026**, projects in the US region experienced failures loading reports and running queries through the Mixpanel UI and Query API. During this window, some queries in the region failed or returned errors. **No customer data was lost, and data ingestion was not affected.** All events continued to be collected and stored normally throughout the incident; once query service was restored, all reports reflected complete, accurate data with no customer action required. The cause was identified as a recently deployed internal tool for diagnostic query replay, a capability our engineers use to re-run copies of past queries to debug performance, which unexpectedly wrote large amounts of data to the disks of our query servers, consuming storage capacity the servers need to operate. When those disks filled, the affected servers took themselves out of service. Service was restored, the impacted servers were brought back online, and the internal tooling was disabled. The remediations below add storage safeguards to remove internal tooling's ability to consume resources on query servers, and are designed to prevent this class of failure from happening again in the future. # What happened Mixpanel's query engine runs on a fleet of servers that each use a set of local storage volumes to cache the data needed to answer queries quickly. Separately, our engineers use diagnostic query replay tooling, a capability our engineers use to reproduce and debug query performance. While applying the diagnostic query replay tooling to a large, complex query, a software bug caused the query replay captures to fan out across the entire fleet instead of remaining confined to a single server. Additionally, it caused far more data than intended to be saved without timely eviction on a single server. Two factors then widened the impact of the issue: * A single full storage volume took a server fully out of service. Each server treats its cache as unhealthy if any one of its storage volumes crosses a usage threshold, even when all other volumes are healthy. The replay data was written to one specific volume on every server, so servers across the region failed their health checks nearly simultaneously. * Cleanup limits did not account for data size. The safeguard limiting replay data on disk counted items at the application level rather than bytes at the filesystem level, so a small number of unexpectedly large captures passed the check while consuming most of the volume’s capacity. Together, these allowed a single debugging workflow that normally has a negligible footprint to interrupt production query serving across the US region. # Timeline \(Pacific Time, August 26, 2026\) * **2:00 PM** — First oversized diagnostic replay capture was written; storage volumes began to reach their capacity and query success rate began to drop shortly after. * **2:17 PM** — Automated alerting paged the on-call engineer; investigation began immediately and additional engineers were engaged. * **2:39 PM** — Status page incident posted; in-app banner displayed at 2:40 PM. * **2:53 PM** — Root cause identified; recovery efforts began on the first affected server group. * **3:19 PM** — Query service restored for the vast majority of traffic; the final server group fully recovered at 3:27 PM. * **3:45 PM** — Incident resolved after a stable observation period. The diagnostic replay tooling that triggered the issue was fully disabled the same evening. # Root cause 1. **A software bug in our query replay tooling caused unbounded writes to production storage.** A recently deployed capability for reproducing queries mishandled a particular class of complex query, causing captures to spread to every query server in the region and to write far more data to the affected volumes than the design assumed. 2. **Replay files consumed disk capacity that query serving depends on.** The replay tooling wrote its files to the query servers' local disks, so runaway replay data exhausted storage capacity the servers need to answer queries. 3. **Safeguards only partially accounted for the behavior.** The cleanup policy for diagnostic replay data limited the number of items on disk, but not their total size, so it did not engage. Alerting on storage volumes flagged the growth but was not escalated as critical on a per-server basis, which delayed detection until query failures began. # What we're changing The end state we are building toward: internal diagnostic query replay data stored in dedicated object storage, placing no load on production query servers. Already deployed: * **Disabled the internal tooling** that caused the incident, and remediated the underlying issue so diagnostic captures are confined to a single server and the specific query class is handled correctly. * **Documented the targeted recovery procedure** used during the incident, i.e., clearing only the affected storage volume rather than restarting full server groups, in our operational runbooks, shortening recovery time if any volume’s capacity runs out in the future. In progress: * **Filesystem-level, size-based limits on diagnostic replay data**, capping total bytes on disk rather than item counts, so oversized captures are rejected or evicted before they can affect the volume’s capacity. * **Stricter storage alerting**, escalating per-server volume saturation as critical before it can affect query health checks. * **Moving query replay data to dedicated object storage,** so it consumes no resources on production query servers. # Common questions * **Was any data lost?** No. Data ingestion was unaffected throughout the incident: events continued to be collected, queued, and stored normally. Only the ability to query was interrupted. Once service was restored, all reports reflected complete data. * **Were saved reports, dashboards, or project settings affected?** No. The incident affected query execution only. Nothing stored in your project changed. * **Why did it affect multiple US projects at once?** The oversized diagnostic replay data written to every query server's disks at nearly the same time, and each server removes itself from service when any one volume fills. Preventing internal tooling from consuming query-server storage is a core part of our remediation work. * **How is this prevented going forward?** The tooling issue is resolved and the tooling remains disabled until size-based limits are in place. Storage alerting is being tightened so saturation is caught before it affects query serving. Structurally, we are moving diagnostic data off production query servers entirely, so internal debugging data will consume no storage on the servers responsible for queries processing. We apologize for the disruption and for the time reports were unavailable. Please reach out through your account team or support with any questions. --- [2026-08-26T15:45:49.876-07:00] (resolved) This incident has been resolved. --- [2026-08-26T15:28:00.739-07:00] (monitoring) A fix has been implemented, and we're seeing improved success rates for the Query API. We will continue monitoring to ensure stability. --- [2026-08-26T15:11:48.765-07:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-08-26T14:39:13.540-07:00] (investigating) Mixpanel is experiencing degraded Query API performance, resulting in HTTP 500 responses when querying reports for projects with US Data Residency. Projects with EU and IN Data Residency remain unaffected. We appreciate your patience while our engineers work to restore normal functionality. We will post progress updates on our status page. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support.</a>

This incident has been resolved. --- [2026-08-19T19:52:31.247-07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-08-19T18:04:53.316-07:00] (identified) We are continuing to work on a fix for this issue. --- [2026-08-19T16:23:08.693-07:00] (identified) We are continuing to work on a fix for this issue. --- [2026-08-19T16:20:44.754-07:00] (identified) We are currently experiencing issues with populating event properties in dropdown menus. We appreciate your patience while our engineers work to restore functionality. If you have any questions, please contact support@mixpanel.com