Is Honeybadger.io Down?

Between 13:38 and 13:45 UTC on November 7, during planned maintenance on one of our Redis servers, part of our error-processing pipeline failed after the server switched to its upgraded node. Processing of incoming errors in the US region was delayed by up to about six minutes. No error data was lost: every error report we received was stored. For roughly 1,900 error reports received between 13:38 and 13:41 UTC, some follow-up steps did not run. As a result, you may not have received an alert for a new error first seen in that window, and a resolved error that recurred in that window may not have been automatically reopened. Some of these occurrences may also appear more than once in search results and occurrence counts. The issue has been resolved and processing has been normal since 13:45 UTC. We are working on fixing the alerts and duplicate occurrences that were affected, and we have identified the code change that will prevent this from happening during future maintenance.

Between 19:54 and 20:07 UTC on August 15, requests to our US error reporting API (api.honeybadger.io) experienced an elevated failure rate. During this window, a majority of requests to submit errors and events timed out or received 5xx responses. Browser (JavaScript) error reporting, source map uploads, the Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 19:54 UTC, a single source began submitting error reports at roughly 150 times its normal rate, with payloads several times larger than average. This more than doubled the total volume of data arriving at our ingestion tier, which saturated within a minute. Because the saturated service kept accepting connections rather than returning errors outright, our autoscaling and alerting — which watch for error responses — did not register the problem quickly, and capacity was added later than it should have been. Service recovered at 20:07 UTC once additional capacity came online. Most affected reports were retried successfully by our client libraries once the API recovered, so the large majority of data submitted during the window was ultimately received. Reports sent by libraries without retry logic were not received and cannot be recovered. What we've done: our ingestion tier now scales on response latency in addition to error rates, so this kind of saturation is detected within a minute or two rather than only once errors appear. We've also raised the tier's baseline capacity and widened our paging alerts to cover failures that don't surface as server errors. We're additionally adding limits on how much data a single source can submit. We're sorry for the disruption, and for any gaps this caused in your error data during the window.