How PulsAPI Tracks 247+ Cloud Services in Real-Time
A look under the hood at how PulsAPI's crawler aggregates status data from hundreds of cloud providers every 60 seconds.
James is co-founder and CTO of PulsAPI. Before PulsAPI he was a staff engineer at a Series C infrastructure company where third-party outages were a constant operational pain. He started PulsAPI to solve the problem once and for all.
The Crawler Architecture
At the core of PulsAPI is a distributed crawler system that polls the public status pages of 247+ cloud services every 60 seconds. Each provider has a dedicated adapter that understands the specific format of their status page — whether it's a Statuspage.io-hosted page, a custom-built status dashboard, or an API endpoint.
The crawler is built with resilience as a first principle. When a fetch fails due to a network timeout or rate limiting, we don't mark the service as unknown. The existing status is preserved until we can successfully refresh. This prevents false alerts during transient connectivity issues.
Each crawl cycle collects the current operational status, active incidents with their timelines, individual component statuses, and scheduled maintenance windows. This data is normalized into a consistent schema regardless of the source format.
Component-Level Granularity
Most status pages report service health at a high level: "AWS is operational" or "GitHub is experiencing degraded performance." But engineering teams need more precision. When AWS has an issue, you need to know if it's S3, Lambda, RDS, or CloudFront — and in which region.
PulsAPI's crawlers extract component-level data from every provider that exposes it. For AWS, that means individual service components across all regions. For Stripe, that's API, Dashboard, and individual product lines. For GitHub, it's Git Operations, API Requests, Actions, and more.
This granularity is what makes PulsAPI actionable. Instead of seeing a generic "partial outage" banner, you can immediately determine whether the affected component is one your application depends on.
Real-Time Processing Pipeline
When a status change is detected, it triggers PulsAPI's real-time processing pipeline. The new status is compared against the previous state to determine if a transition occurred (e.g., Operational to Degraded). If a transition is detected, the system creates an incident record and fires alert evaluations.
Alert routing rules are evaluated per-user, per-service, and per-severity. A team might route AWS Critical alerts to their #infra Slack channel, Stripe alerts to the payments team's email, and everything else to a daily digest. The routing engine processes these rules in under 4 seconds from detection to delivery.
Historical data is retained for 30 days on Free plans and 90 days on Pro plans. This data feeds the SLA metrics engine, which continuously calculates rolling uptime percentages, MTTR, and incident frequency for every monitored service.
Scaling to Hundreds of Providers
Adding a new provider to PulsAPI's monitoring network takes minutes, not days. Our adapter framework provides base implementations for common status page platforms (Statuspage.io, Cachet, custom RSS/Atom feeds) that handle 80% of providers out of the box.
For providers with custom status page implementations, we write specialized adapters that parse their specific HTML structure or API responses. These adapters are tested against historical snapshots to ensure accuracy before going live.
The system is designed to scale horizontally. As we add more providers and more users, the crawler distributes work across multiple nodes, the processing pipeline handles increasing alert volume, and the dashboard serves real-time updates efficiently through optimized database queries and caching.
Start monitoring your stack
Free for up to 10 services. No credit card required.