Is Honeybadger.io Down?

Between 19:54 and 20:07 UTC on August 15, requests to our US error reporting API (api.honeybadger.io) experienced an elevated failure rate. During this window, a majority of requests to submit errors and events timed out or received 5xx responses. Browser (JavaScript) error reporting, source map uploads, the Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 19:54 UTC, a single source began submitting error reports at roughly 150 times its normal rate, with payloads several times larger than average. This more than doubled the total volume of data arriving at our ingestion tier, which saturated within a minute. Because the saturated service kept accepting connections rather than returning errors outright, our autoscaling and alerting — which watch for error responses — did not register the problem quickly, and capacity was added later than it should have been. Service recovered at 20:07 UTC once additional capacity came online. Most affected reports were retried successfully by our client libraries once the API recovered, so the large majority of data submitted during the window was ultimately received. Reports sent by libraries without retry logic were not received and cannot be recovered. What we've done: our ingestion tier now scales on response latency in addition to error rates, so this kind of saturation is detected within a minute or two rather than only once errors appear. We've also raised the tier's baseline capacity and widened our paging alerts to cover failures that don't surface as server errors. We're additionally adding limits on how much data a single source can submit. We're sorry for the disruption, and for any gaps this caused in your error data during the window.

Between 06:22 and 08:23 UTC on July 8, requests to our US error reporting API (api.honeybadger.io) experienced a highly elevated failure rate. During this window, a majority of requests to submit errors, events, and check-ins received 5xx responses or timed out. Error reports sent by client libraries without retry logic during this period were not received and cannot be recovered. The Honeybadger web application, dashboards, uptime monitoring, and our EU region were not affected. What happened: Beginning at 06:18 UTC, we started receiving error reports at roughly triple our normal total API volume. The service tier that ingests this traffic was at its overnight capacity level and became saturated. Our autoscaling responded, but the metrics it relied on significantly under-measured the failed traffic, so capacity was added far too slowly — full recovery came at 08:23 UTC when scaling caught up and absorbed the load. Compounding the problem, PagerDuty alerts were not fired to wake up a human due to a misconfiguration. What we've done: We've fixed the underlying scaling weakness: API capacity now also scales directly on load-balancer error rates (previously we scaled only on request rates, and that scaling doesn't respond quickly enough), and we've doubled the maximum capacity of the ingestion tier. These changes are already deployed to both US and EU regions. We're also evaluating adding adaptive per-key rate limiting to the ingestion service. We're sorry for the disruption, and for any gaps this caused in your error data during the window.