Is Lucidworks Platform Down?

## Summary On October 27, 2026, between 02:51 UTC and 17:29 UTC, Connected Search and Agent Studio agents in the Lucidworks production environment began returning intermittent errors. Any query request could potentially return a HTTP 500 error after waiting up to two minutes for a response. Other Lucidworks Platform services, including embedding models, LLM access, Commerce Studio, and the Lucidworks Platform UI, were not affected. Lucidworks was alerted at 03:12 UTC and began remediation efforts. The incident was resolved at 21:30 UTC. ## Root Cause The first window of disruption began at 02:51 UTC. The successful-response count from our search backend, hosted by a third-party provider, dropped to zero, and the search service stopped answering queries. Lucidworks restarted the services that were reporting unsuccessful search connections, which restored normal operation by 03:33 UTC. The second window began at 13:50 UTC, with smaller bursts of errors through 17:30 UTC. Lucidworks restarted services again and re-engaged our incident response procedures. During this window, we turned off extraneous search query services across the Lucidworks Platform, updated the third-party HTTP client settings to the provider's current recommendations, and added explicit timeouts and a larger connection pool. No errors were observed after these changes were put into effect at 17:29 UTC, and the incident was resolved at 21:30 UTC. The query service connection to the third-party search backend used default connect, read, and connection-wait timeouts, and the connection pool was small. When a connection stalled or was closed without a response, the request waited indefinitely, holding a pool slot and a request thread until the caller timed out after two minutes. With enough stalled connections, the service stopped serving search requests until it was restarted. The default timeouts turned connection problems into a service outage instead of a few quickly failing requests. In the second outage window, the third-party provider confirmed that the disruption coincided with a platform upgrade that restarted two of its container nodes. The provider recommended the changes to connection timeouts that we were already planning to add to our service. However, the behaviour was not limited to the confirmed upgrade window. In the first window, the third party reported no restarts, errors, or changes in request rate on its side, and the trigger for the stalled connections had not yet been identified. After the fixes were deployed, three connection attempts \(at 21:05 UTC and twice at 21:42 UTC\) received no answer to the TLS handshake. In each case, the new connection timeout ended the attempt after 2 seconds, and the retry on a new connection succeeded. Over the same period, all search requests returned successfully, with a median \(p50\) latency of 44 ms, validating our connection timeout fixes.The cause of the remaining intermittent issues on our search backend continues to be investigated by our provider. ## Lucidworks Actions Lucidworks will take the following actions as a result of this incident: * Add a scheduled check to our third-party provider's current recommendations for HTTP client settings to their service to proactively capture new suggestions for our query services * Update our tracked metrics for our services to monitor for recurrences of this particular issue * Ensure that adding additional infrastructure does not increase the connection load to our regionalized search services * Work with our third-party provider to identify what stalled the connections in the first window, and why some TLS handshakes still go unanswered outside the confirmed upgrade window ## Recommended Client Actions There are no recommended client actions as a result of this incident. --- [2026-10-27T21:29:38.937Z] (resolved) We have fully resolved the incident and validated the fix. All previously affected Lucidworks services are operating normally, restoring full functionality across the platform and eliminating impact to all users. An incident report will be published by end of day Friday. --- [2026-10-27T19:05:58.132Z] (monitoring) Lucidworks services continue to operate normally with no platform disruption or customer impact. We are continuing to monitor for any further issues and will provide an update once full resolution is confirmed. --- [2026-10-27T17:00:29.949Z] (monitoring) We are continuing to monitor for any further issues. --- [2026-10-27T16:11:53.173Z] (monitoring) We have deployed an infrastructure change to mitigate the issue. Connected Search and AI agent services are currently stable. Lucidworks services and software are operating normally. Platform users are no longer experiencing service disruptions. There is no direct customer or end-user impact observed. We will continue to actively monitor Connected Search and AI agents and will provide a final update once full resolution is confirmed. --- [2026-10-27T15:28:21.539Z] (investigating) We are continuing to investigate the source of Connected Search and AI agent errors on our platform.  This is a recurrence of the event from October 27, 2026, but the same mitigation steps taken then are not resolving the issue this time.  Lucidworks engineers are actively working to identify a root cause, and we will post further updates here as soon as we have more information available to share. --- [2026-10-27T14:55:51.030Z] (investigating) Lucidworks Connected Search and AI agent services are experiencing a service disruption resulting in failed requests. We are actively investigating the issue to restore normal operations. We will provide further updates as more information becomes available.

This incident is now fully resolved and all systems have returned to normal operations. Lucidworks services and software are operating normally with no remaining disruption, and there is no remaining direct customer impact to end users. --- [2026-10-27T04:43:23.788Z] (monitoring) We have restarted the affected backend services, and have seen the errors subside. We can confirm that stability has returned; Connected Search queries and Agent Studio agents are once again fully functional.  We are continuing to monitor the services to watch for a recurrence of errors and will provide a final resolution update once full stability is verified. --- [2026-10-27T04:40:21.750Z] (investigating) We are continuing to investigate this issue. --- [2026-10-27T04:04:08.186Z] (investigating) Lucidworks Connected Search is currently experiencing a service disruption resulting in query request failures. Customers using the platform to execute search queries are receiving failed requests. In addition, interactions with Agent Studio agents are unable to complete. We are actively investigating the issue to restore normal operations. We will provide further updates as more information becomes available.