Is Flxpoint Down?

Impact: From around 11:00 PM EST on Nov 10 until 12:33 AM EST on Nov 12, transfers on ftp.flxpoint.com were slow, with downloads as low as ~50 KB/s, and some uploads failed. Every customer using the hosted FTP service could be affected; customers transferring large files were hit hardest. Root cause: The FTP server streams every file to and from cloud storage, so each transfer passes through the server twice. Overnight on Nov 10–11, a larger-than-usual volume of large files arrived, and at the same time several clients held 100–200 connections open each, about 630 sessions in total. Together this pushed the server past its network capacity. The cloud provider then throttled its network traffic, slowing every connection on the server and causing some long transfers to drop. Resolution: At 12:33 AM EST on Nov 12, we upgraded the server to an instance type with about 12× the guaranteed network capacity and twice the CPU. Read speed from storage went from ~156 KB/s to ~190 MB/s, and network throttling dropped to zero. The upgrade took about 5 minutes of downtime. Prevention: * Alerts to our on-call channel when the FTP server approaches network, CPU or connection limits * Per-user connection limits and idle-session timeouts on the FTP service, so one client can't slow down others * Reviewing server capacity against FTP traffic growth regularly --- [2026-11-12T05:38:52.909Z] (resolved) This incident has been resolved. Between around 11:00 PM EST on Nov 10 and 12:33 AM EST on Nov 12, the FTP server behind ftp.flxpoint.com was reaching its network capacity under load, which caused slow transfers and intermittent upload failures. We upgraded the server at 12:33 AM EST on Nov 12, increasing its network capacity about 12x, and transfer speeds are back to normal. If a transfer was interrupted during this period, please retry it. We're also adding monitoring and per-user connection limits to prevent a recurrence. --- [2026-11-11T16:11:59.735Z] (identified) Since around 11:00 PM EST on Nov 10, some customers have seen slow transfers and intermittent upload failures on ftp.flxpoint.com. We've identified the cause: the FTP server is reaching its network capacity under current load. We'll upgrade the server on Nov 12, 1:00–1:30 AM EST. FTP will be unavailable for about 5 minutes during that window. Connections will drop and should be retried afterwards. We'll post an update once the upgrade is complete.

# Integration Disruption Post-Mortem - October 13 & 15, 2026 Oct 15, 2026 On October 13 and October 15, several supplier, marketplace and EDI integrations failed to run for part of the day after two updates to our integration platform. Both problems are fixed and all integrations are running normally. EDI documents received during a two-hour window on October 15 \(11:50–13:49 UTC\) are being reprocessed. ## What happened As part of routine maintenance, we updated a shared component that many integrations use to exchange data with suppliers and marketplaces. Two of those updates accidentally removed pieces that some integrations still needed. * **October 13.** The first update affected integrations that connect through older web-service connections, including eBay and several suppliers. Their scheduled jobs failed until a fix went out at 17:33 UTC. * **October 15, from 08:52 UTC.** A follow-up update affected EDI integrations and a further set of supplier and marketplace connections. These jobs failed until a fix went out at 11:44 UTC. * **October 15, from 11:50 UTC.** That fix exposed a second problem. EDI integrations ran and showed as successful, but could not read the documents they received, such as invoices, shipment notices and order acknowledgements. A second fix went out at 13:48 UTC. Every EDI file received during the disruption was kept, so suppliers do not need to resend anything. On both days, integrations that had failed returned to their regular schedule at about 20:00 UTC. ## Timeline \(UTC\) Affected integrations were disrupted from 08:52 to about 20:00 UTC on October 13, and from 08:52 to about 20:15 UTC on October 15. **October 13** * 08:52 — First update released; integrations listed under October 13 below begin failing * 17:33 — Fix released; failures stop * ~20:00 — Affected integrations back on their regular schedule **October 15** * 08:52 — Follow-up update released; integrations listed under "October 15, 08:52–11:59" below begin failing * 11:44 — Fix released; last failure at 11:59, from a job that started before the fix * 11:50 — EDI integrations begin failing to read incoming documents \(jobs still show as successful\) * 13:48 — Second fix released; last unreadable document at 13:49 * 20:02–20:18 — Integrations that failed earlier that day back on their regular schedule * In progress — Reprocessing EDI documents received 11:50–13:49 ## Affected integrations 21 integration types were affected. Find yours below to see which processes were down and when. A dash means it was not affected in that window, and integration types not listed were not affected at all. All times are UTC. **October 13, 08:52–17:33 UTC** * Crow Shooting: Shipments, Send POs/FRs, PO/FR Acknowledgements * eBay: Get Orders, Sync Orders, Sync Listings * Keystone: Shipments, Send POs/FRs, Inventory * Kroll: Shipments, Invoices * PoolCorp: Inventory, Send POs/FRs, Shipments * Promostandards: Inventory * Seawide: Send POs/FRs, Shipments, Inventory **October 15, 08:52–11:59 UTC** * Camuto: Send POs/FRs, Shipments, Invoices * Crow Shooting: Shipments, PO/FR Acknowledgements * Essendant: Inventory * Fanatics EDI: Orders * Flxpoint Standard EDI: Invoices, Shipments, Inventory * Flxpoint Standard EDI V2: Inventory, Shipments, Invoices, Send POs/FRs, PO/FR Acknowledgements * Keystone: Shipments, Send POs/FRs, Inventory * Land N Sea: Inventory, Invoices * McKesson: Shipments, Invoices, PO/FR Acknowledgements, Send POs/FRs * OJG: Shipments, Inventory, Invoices * Orgill: Send POs/FRs * Rothco: Inventory * RSR EDI: Invoices, Shipments, PO/FR Acknowledgements, Send POs/FRs * Seawide: Shipments, Inventory * Sports South EDI: Invoices, Shipments, PO/FR Acknowledgements, Send POs/FRs * TDSynnex: Shipments * Walmart: Sync Listings **October 15, 11:50–13:49 UTC — EDI documents could not be read** * Camuto: PO/FR Acknowledgements, Inventory * Fanatics EDI: Orders * Flxpoint Standard EDI: Inventory * Flxpoint Standard EDI V2: Inventory, Shipments, Invoices, Send POs/FRs, PO/FR Acknowledgements * Land N Sea: Inventory, Invoices * McKesson: Shipments * RSR EDI: Invoices, Shipments, PO/FR Acknowledgements, Send POs/FRs * Sports South EDI: Invoices, Shipments, PO/FR Acknowledgements, Send POs/FRs Integrations that failed in the first two windows did not run again until about 20:00 UTC the same day. In the third window, EDI jobs kept running but could not read the documents they received. ## What this means for your data Most integrations caught up on their own when they resumed. EDI documents from the third window are the one thing still being restored. * **October 13 and the October 15 08:52–11:59 window.** Affected integrations picked up where they left off when they resumed at about 20:00 UTC. Orders, shipments, invoices and purchase orders that were due during the outage were handled on that run, and inventory refreshed with current data. * **EDI documents received October 15, 11:50–13:49 UTC.** Invoices, shipment notices \(with tracking\) and order acknowledgements from this window have not yet appeared in Flxpoint. Your suppliers also have not yet received our receipt confirmations for them. We are reprocessing these documents now; the confirmations will be sent as part of that, and we will update this post when it is complete. EDI inventory files from this window will not be reprocessed, because newer inventory has already been applied. If you still see a missing order, shipment, invoice or acknowledgement from these windows after we confirm reprocessing is complete, contact support with the order or PO number. ## What we're doing to prevent this * Adding automated checks that exercise each type of supplier, marketplace and EDI connection before any update to the shared component is released. * Alerting when an integration cannot read the documents it receives, even if the job otherwise completes. * Changing how EDI documents are retried, so files that fail because of a problem on our side are processed automatically once it is fixed. * Making sure affected integrations return to their regular schedule as soon as a fix is released, not hours later. We're sorry for the disruption to your operations, and thank you for your patience. --- [2026-10-15T20:54:14.983Z] (resolved) On October 13 and October 15, several supplier, marketplace and EDI integrations failed to run for part of the day after two updates to our integration platform. Both problems are fixed and all integrations are running normally. EDI documents received during a two-hour window on October 15 (11:50–13:49 UTC) are being reprocessed.

**What happened** From Oct 15, 00:10 UTC to Oct 15, 15:49 UTC, Order Imported and Fulfillment Request Processing webhooks were delayed. **Impact** These webhooks weren't sent during that window. Orders kept importing and processing normally, and no data was lost. Once the issue was fixed, all delayed webhooks were delivered, so you may have received a burst of older events. **Cause** Our cloud provider replaced the server that sends webhooks, and the new server didn't start correctly. Our monitoring didn't catch it. **Resolution** We restored the webhook service, and all delayed notifications were delivered by 15:49 UTC. **Next steps** We're adding alerts for delayed webhook delivery and fixing the deployment issue that kept the replacement server from starting. If your system was unavailable while we caught up and you're missing events, contact support and we'll resend them. --- [2026-10-15T15:54:28.019Z] (resolved) All delayed webhooks have been processed and the incident is fully resolved. --- [2026-10-15T15:41:40.730Z] (identified) Webhooks stopped processing at 2026-10-14 15:00 UTC Root cause has been identified and is being fixed.