Is Auto Integrate Down?

# RCA - Auto Integrate Incident – 10/26/2026 ## Issue On October 26, 2026, at 4:30 am CT, Auto Integrate changed its DNS configuration as part of a new routing strategy. The change caused intermittent connectivity failures. Some customers reported being unable to access the service or log in. We received the first customer report at 7:32 am CT, applied a corrective DNS change at 8:11 am CT, and marked the incident resolved at 10:28 am CT. ## Timeline * 4:30 am CT - DNS changes deployed * 7:32 am CT - First customer report received * 8:00 am CT - Status page updated to communicate the incident * 8:11 am CT - Missing DNS records added to the old provider * 9:10 am CT - Throughput monitoring showed normal traffic across affected customers * 10:28 am CT - Incident marked resolved ## Root Cause We changed the DNS record structure while moving the domain between providers. The new structure introduced aliases: records that point to other DNS names. Some records needed to resolve those names existed only at the new provider. During the switch to the new provider's nameservers, DNS resolvers - the systems that look up addresses for customers - could combine cached information from the new configuration with answers from the old provider. For example, a resolver could cache an alias from the new configuration, then later ask a nameserver still serving the old configuration for its target. Because that target record was missing there, the lookup would fail. Normal DNS caching exposed the inconsistency between the providers. The underlying cause was the missing records, not caching itself. ## Contributing factors * **Inconsistent configurations:** The required records were not available with consistent destinations at both providers throughout the transition. * **One-way validation:** We checked that existing records and the lookups that used them worked the same way at the new provider. We did not check that records introduced in the new configuration were also available and returned the same results at the old provider. * **Monitoring gaps:** Existing synthetic checks, including DNS resolution checks, remained successful during the incident and did not detect the failures experienced by affected customers. We first learned of the issue through a customer report. ## Resolution At 8:11 am CT, we added the missing records to the old provider so it could answer the affected DNS lookups. The correction did not take effect for everyone immediately: DNS resolvers could retain earlier responses until their negative cached entries expired. We monitored throughput—the volume of traffic—for customers identified as affected. Their traffic increased from near zero, providing evidence of recovery at 9:10 am CT. We marked the incident resolved at 10:28 am CT based on that observed recovery and confirmation from affected customers that service was resumed. ## Mitigation The following steps will be taken to mitigate this type of incident in future: * Compare DNS records in both directions between providers. * Identify records missing from either configuration. * Complete and test record-structure changes at the existing provider before switching nameservers. * Separate structural changes from the provider migration. * Test lookups that combine cached records from one configuration with answers from the other provider. * Catch failures that testing either provider separately may miss. * Enhance throughput alerts for drops across key APIs and customer access paths. * Detect partial connectivity failures earlier and reduce reliance on customer reports. ## Lessons Learned 1. **Treat the transition as a distinct operating state.** Two configurations that work separately may still fail when customers use information from both. 2. **Change one thing at a time.** Separating record-structure changes from the nameserver switch makes each step easier to validate. 3. **Measure recovery through customer activity.** A corrected configuration is not, by itself, evidence that affected customers have recovered. --- [2026-10-26T10:28:30.362-05:00] (resolved) This incident has been resolved. --- [2026-10-26T09:03:21.507-05:00] (monitoring) If you are still experiencing any issues, please ask your IT team to clear your DNS cache. --- [2026-10-26T08:38:15.583-05:00] (monitoring) A fix has been implemented and we are monitoring to ensure connectivity --- [2026-10-26T08:00:13.026-05:00] (investigating) We are currently investigating an issue relating to users connecting to Auto Integrate, we will update with more information as we investigate

We will be undergoing scheduled infrastructure maintenance during this time. There should be minimal impact on the applications during this window.

Auto Integrate will be renewing SSL certificates across our infrastructure at the scheduled time. This will apply to the web portal, FTPs server and APIs. The certificate has already been renewed in UAT environment so please verify connectivity prior to the production renewal. There should be no downtime during this maintenance.