Is ShipHawk Down?

Resolved on FedEx side. --- [2026-09-02T09:24:11.429-07:00] (monitoring) FedEx web services are currently experiencing degraded performance, and some customers have reported issues booking FedEx shipments. We will continue to monitor the situation while FedEx works to resolve the issue. Current FedEx response times are available here: https://www.shippingapimonitor.com/history.html?api=fedex

## Post-Incident Report: WMS Slowness and Errors for Warehouses on One Database Instance - August 25, 2026 **Status:** Resolved **Incident window:** August 25, 2026, ~04:30 - 08:45 PDT \(07:30 - 11:45 ET\) **Affected:** Customers whose databases are hosted on one WMS database instance in our US-East warehouse group: page load times of up to 20x normal, and intermittent errors on scanner and admin screens during the worst period. **Not affected:** All other WMS database instances and warehouse groups, the TMS platform, data integrity \(no transaction was lost, duplicated, or partially applied\), and all completed work - every transaction that was accepted was processed correctly. ### Were you affected? Impact was limited to customers whose WMS databases are hosted on one specific database instance in our US-East warehouse group, and only during the morning of August 25 \(approximately 04:30 - 08:45 PDT / 07:30 - 11:45 ET\). If you did not see slow page loads in the WMS during that window, your environment was not involved. All other WMS database instances and warehouse groups, and the entire TMS platform, operated normally throughout. ### Summary On the evening of August 24, a third-party ETL service that copies ShipHawk data into our data warehouse restarted a large batch of its sync jobs at once against one of our production databases. Each restarted job began re-reading a backlog of historical change data at full speed, in parallel and without any rate limiting. Read volume climbed to roughly five times the expected peak and consumed most of the disk bandwidth available to that database instance. This began overnight, when warehouse activity is light, so it had no effect on operations at the time. When US-East morning shifts started on August 25, normal WMS activity was added on top of the already-saturated channel and the instance reached its bandwidth limit. From the application's point of view this appeared as slow database responses: queries that normally return in milliseconds took far longer, work queued up behind them, and pages loaded slowly. Where a database response exceeded the application's timeout, the page returned an error instead of loading. The service remained operational throughout. Across the affected instance, customers processed approximately 70% of the volume normally handled in that window \(picks, packs and inventory moves\), though the experience was slow and at times difficult to work with, and the degree of impact varied between customers. The situation resolved by revoking the ETL service database credentials entirely at 08:37 PDT. The database recovered within eight minutes. Delayed work flushed through over the following two hours, and all affected warehouses were back to normal pace. No customer action was or is required, and no data was affected. No transaction was lost, duplicated, or partially applied - we verified this against integration logs covering the full incident window. Work that was submitted either completed correctly or failed cleanly before making any change. This is covered in more detail below. ### What was affected Impact was limited to customers whose databases are hosted on the affected WMS database instance in the US-East warehouse group. Throughout the incident window the system was slow across the board - scanner and admin pages that normally load in well under a second took many times longer - and where a database response exceeded the application timeout, the page returned an error rather than loading. What this looked like in practice: **Warehouse floor:** scanner pages \(picking, moving, adjusting inventory\) loaded very slowly; a worker who retried while a page was stuck could receive an error page and have to go back and repeat the action. **Errors were concentrated in a single burst rather than spread across the incident.** Most of them fell within one 15-minute window at the peak of the congestion \(06:15 - 06:30 PDT\). Counting confirmed error pages in our web-server and application logs, the most affected warehouse saw 71 in that window. **The system remained up throughout.** Every submitted transaction either completed correctly or failed cleanly before making any change. During the deepest slowdown window we can show hundreds of transactions completing successfully for the users who continued working. ### What was NOT affected **Data integrity.** The errors occurred at the very start of request processing, before any change was made. No transaction was lost, duplicated, or partially applied. Every fulfillment, inventory move, and shipment posting that completed did so correctly - we verified the integration logs for the incident window. **Order and shipment synchronization to ERPs and marketplaces** completed correctly throughout; postings that queued up during the slowdown were delivered in full during the catch-up \(verified in integration logs - no failed postings\). **All other environments.** Warehouses on our other database instances, and the entire TMS platform, operated normally. **Security and tenancy.** No security boundary was involved at any point. The third-party service in question is a data-integration vendor operating under credentials we issued; the issue was the volume of its reads, not any unauthorized access. ### Timeline \(all times PDT; add 3 hours for ET\) | Time | Event | | --- | --- | | Aug 24, 22:54 - 22:57 | The ETL service restarts ~22 sync jobs against the database within a three-minute window. Each begins re-reading historical change data at full speed. | | Aug 24, 22:54 - 23:50 | Read volume climbs to roughly five times the expected peak, consuming most of the disk bandwidth available to the instance. Overnight warehouse traffic is light, so there is no customer-visible effect yet. | | Aug 25, ~04:30 | US-East warehouse morning shifts begin. Combined demand exceeds the capped network speed; queues start building and the first pages begin rendering slower than normal. | | 05:30 | Automated response-time monitoring alerts as warehouse activity ramps up; customer reports of slowness arrive in the same period. Investigation begins. | | 06:00 - 07:00 | Peak congestion: database connections spike to ~15x normal as requests pile up; the wave of scanner-screen errors occurs \(06:15-06:30\). | | 06:50 | Root cause identified: disk bandwidth over the instance limit; the vendor's replication streams identified as the driver. | | 07:00 | Heaviest internal report queries disabled to free capacity - partial relief. | | 07:20 - 08:30 | ETL service sync jobs are paused in waves in its console and its database sessions terminated; the service automatically reconnects within seconds each time and continues reading. During this period it starts additional jobs. | | 08:35 | The ETL service database credentials are locked and its sessions terminated a final time. | | 08:38 - 08:45 | Database queues drain; page response times return to normal. Customer impact ends. | ### Why resolution took ~3 hours from first reports Three factors extended the timeline. First, the trigger occurred seven hours before symptoms. The ETL service's re-read ran overnight and had already consumed the available bandwidth, but with warehouse activity light at that hour the constraint produced only a slight change in system response times - below our alerting thresholds - so it went undetected. Our automated monitoring did alert once warehouse activity ramped up in the morning, but by then the underlying change was seven hours old and there was no recent deployment or configuration change to point to. Second, the ETL service's replication reads are invisible to standard database query logs - they use a replication protocol rather than queries - so identifying them as the consumer required correlating disk, network, and connection-level evidence. Third, the ETL service is built to survive interruptions: pausing its jobs and terminating its connections both failed as mitigations because it reconnects automatically within seconds, and it restarted additional jobs while we were pausing others. Only revoking its credentials stopped it. ### What we are changing **Tuning WMS response-time alert thresholds.** The condition behind this incident was present for seven hours overnight, but under light load it moved response times too little to cross our alert thresholds - so the first alert came only once warehouse activity ramped up and customers were already affected. We are tuning those thresholds to be sensitive to smaller shifts in WMS response time, including at low load, so events like this are caught and acted on before they reach customers. This includes alerting on the specific leading indicators of this incident - disk bandwidth consumption and disk queue depth. **The database has been migrated to an instance type with substantially more disk bandwidth**, giving significant headroom above peak demand to absorb spikes of this kind. **We are continuing our investigation with the ETL vendor.** We have an open case with them seeking an explanation for the simultaneous job restart, and requiring rate limiting and concurrency caps for re-reads against customer sources. That work is ongoing. --- [2026-08-25T10:51:08.980-07:00] (resolved) This incident has been resolved. We'll share more details as soon as they are available. --- [2026-08-25T09:01:37.063-07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-08-25T07:10:02.867-07:00] (investigating) Some customers are experiencing slowness with the WMS access. Our team is actively investigating the issue.

# Post-Incident Report: Elevated API and Login Errors - August 20, 2026 **Status:** Resolved **Incident window:** August 20, 2026, 06:31 - 08:11 PDT \(13:31 - 15:11 UTC\) **Affected:** ShipHawk API and dashboard requests in shared production environments, plus the login service. Impact was partial rather than a complete outage: approximately 25% of overall API traffic on [shiphawk.com](http://shiphawk.com) failed during its affected window; failure rates within affected environments ranged from approximately 37% to 48%, and approximately 41% of login-service requests failed. **Not affected:** In-Cart rating \(and all /api/v4/rates requests\), background processing \(all scheduled jobs, write backs, webhooks, async label generation, tracking and carrier communications ran normally\), data integrity. ## Summary On the morning of August 20, an operating-system critical security update published by Ubuntu - and applied automatically by our standard patching process - contained a defect in the web server component \(nginx\) that sits in front of the ShipHawk application. The affected package was published on August 19 as [USN-8563-3](https://ubuntu.com/security/notices/USN-8563-3). Ubuntu confirmed that this update introduced a regression and published [USN-8563-4](https://ubuntu.com/security/notices/USN-8563-4) the same day, reverting the problematic change pending further investigation. While the faulty version was running, the proxy layer corrupted the URL of many incoming requests before handing them to the application. The application could not match the corrupted URLs to any known endpoint and answered **404 Not Found**. The failures were immediate, clean rejections: no request was partially processed, routed to the wrong account, or lost after acceptance. Our servers do not all download and install operating-system security updates at the same moment; update checks and installation windows are staggered across hosts. As a result, some servers downloaded the faulty nginx build before Ubuntu published the corrected package, while others checked later and downloaded the corrected build directly. Only the servers that had already downloaded the faulty package became affected when their scheduled installation ran. This is why the issue appeared intermittent: otherwise-identical requests could fail or succeed depending on which server handled them. Even on servers running the faulty nginx package, only a subset of requests failed. The regression affected specific nginx routing rules rather than the entire proxy configuration, so many URL patterns continued to work normally on an affected server. The incident was fully resolved by 08:11 PDT after every affected server was upgraded to the corrected package and verified healthy. No customer action was or is required. ## What was affected The numbers below count **only failures caused by this incident**. Ordinary 404 responses \(lookups of records that genuinely don't exist, invalid URLs, bot traffic\) were identified by their distinct response signature and excluded. | Environment | Scope | Impacted window \(PDT\) | Failed requests | | --- | --- | --- | --- | | sh-p-1 environment | 2 of 3 web servers | 06:34 - 08:08 | ≈37% of requests on affected servers; ≈25% of overall API traffic | | Login service | Both servers | 06:33 - 08:08 | ≈41% of login-service requests | | sh-p-2 environment | 2 of 3 web servers | 06:31 - 08:08 | ≈47% of requests on affected servers | | sh-p-3 environment | 2 of 3 web servers | 06:31 - 08:11 | ≈48% of requests on affected servers | ‌ What this looked like in practice: * **API integrations** received HTTP 404 responses for valid requests. Because failures were immediate and stateless, client retries could succeed when they landed on an unaffected server. * **Dashboard and login** pages failed to load or sign in intermittently. * Failures depended on the exact URL: some request types passed through unaffected even on faulty servers, adding to the intermittent appearance. ## What was NOT affected * **In-cart rating requests.** All rating requests from the web portal, e-commerce platforms, ERP platforms and regular API requests to `/api/v4/rates` were working as usual. * **Background jobs were not affected at all.** All asynchronous processing - scheduled jobs, write backs, inventory sync, webhook deliveries, document and label generation, carrier and ERP communications - runs behind the proxy layer and continued normally throughout the incident. No queued work was lost or delayed. * **Data integrity.** No data was lost, altered, or corrupted. Requests either completed normally or were rejected outright. * **Security and tenancy.** No request was routed to another account, and no security boundary was crossed. The corruption occurred after all access controls were applied. The underlying Ubuntu update was a preventive security patch; the vulnerability it addressed was not exploited on our systems. ## Timeline \(all times PDT, August 20, 2026\) * **Aug 19 \(daytime\)** - Ubuntu publishes a security update for nginx; a defect is reported, and Ubuntu publishes a corrected package the same day. The corrected version propagates to public update mirrors overnight. * **Aug 19, 18:24 - 23:09** - The nightly update checks on the later-affected servers download the day's nginx update. At these moments, the faulty build is still the newest available on the mirrors. This step only downloads the package; installation happens during the next morning's patch window. * **Aug 20, 04:12 - 05:05** - Another group of servers runs its nightly update check after the corrected build has reached the mirrors. These servers download the fixed version and remain healthy throughout the incident. * **~06:00** - A routine, unrelated application configuration update is applied for upcoming releases. It has no effect on currently released functionality and **plays no role in the incident**, but because it is the only known change that morning, it becomes the first suspect once errors appear. * **06:00** - The nightly automated patching window begins rolling nginx updates across environments. Some servers already have the corrected package downloaded, while others have the faulty package. * **06:31 - 06:34 - INCIDENT START.** As the rolling automated patch window progresses, the previously downloaded faulty nginx package is installed on multiple web and login servers across shared production environments. Because patch schedules are staggered, not all servers update at once, and some servers remain healthy. The first customer-facing failed requests begin at **06:31**. * **06:35** - Automated external monitoring alerts on elevated errors. **Investigation begins immediately.** * **06:36 - 07:15** - Engineers first investigate the ~06:00 configuration update, the only known application-level change with closely matching timing. It is ruled out, and attention turns to the web/proxy layer. * **06:52** - The rolling patch window continues and the faulty package is activated on additional servers. Impact increases as more affected servers restart onto the faulty nginx version, while servers that downloaded Ubuntu's corrected package remain healthy. * **06:55** - A remaining web server updates using Ubuntu's corrected package and stays healthy throughout, continuing to serve its share of traffic correctly. * **07:18 - 07:19** - Affected web servers are restarted as a mitigation attempt. This has no effect because the faulty nginx package remains installed. * **07:20 - 07:55** - Suspect servers are removed from load-balancer rotation. Symptoms persist because the login service and application environments are independently affected, which materially widens the search. * **07:41** - The URL-corruption pattern is identified in application logs. * **07:45 - 08:00** - Per-server testing isolates the faulty servers. The only difference from healthy servers is the nginx package version. The faulty build is matched to Ubuntu's published regression notice and corrected package. * **08:02 - 08:11** - The corrected package is installed across all affected servers. Error rates return to normal immediately on each server as it restarts onto the fixed version. The final affected environment returns to normal at **08:11 - INCIDENT FULLY RESOLVED**. * **08:11\+** - Full verification is completed: every server is individually tested, and API, dashboard, login, and production environments are confirmed healthy. ## Why resolution took ~95 minutes from alert Detection was fast, but three factors slowed diagnosis. First, a routine configuration change earlier that morning was the only known change in the environment and had to be ruled out - automated OS patching does not appear in any application-level change log. Second, the failure was intermittent by nature: unaffected servers continued serving normally, and even affected servers successfully handled request types whose routing rules were not impacted. Third, removing the suspect servers from rotation did not stop the errors - because other tiers were independently affected - which initially pointed the investigation away from those servers. ## What we are changing **1. Stage operating-system security patches before production.** Automated OS- and nginx-level security updates, including critical patches, will first be installed on non-production servers. Automated application-level validation will exercise representative API, dashboard, and login paths against the updated servers before the same package versions are allowed to roll into production. Production rollout will begin only after those checks pass. **2. Faster version-level diagnosis.** Our incident runbooks now include immediate comparison of package versions and restart history across servers whenever identically-configured servers behave differently. --- [2026-08-20T11:06:16.942-07:00] (resolved) This incident has been resolved. --- [2026-08-20T08:16:39.207-07:00] (monitoring) A fix has been implemented and we are monitoring the results. --- [2026-08-20T08:02:42.101-07:00] (identified) The issue has been identified and a fix is being implemented. --- [2026-08-20T07:06:22.827-07:00] (investigating) We've received reports that TMS WebPortal is returning errors or failing to load for some customers. We're actively investigating the issue and will provide updates as more information becomes available.