# How we track our providers' outages, without the noise

By Lucas Espindola · June 23, 2026 · https://lucasespindola.com/notes/tracking-upstream-outages

> A small Cloudflare Worker that polls the status pages of the clouds and model providers we depend on every ten minutes, keeps only the incidents that touch services we use, and posts each one once in Slack, with its updates threaded underneath.

Engineering · internal tools

When something breaks, the first question is whether it's us or one of our providers, and the answer is often on a status page nobody has open. This is the small Cloudflare Worker we built at Pinata to watch those pages for us: it checks every provider we depend on every ten minutes, keeps only what touches the services we use, and posts each incident once, with its updates threaded underneath.

## Why not just subscribe to the status pages?

Every provider offers email, RSS or Slack subscriptions to its status page. On their own they fall short in two ways:

- **Too much.** A subscription sends everything: every region, every product, every minor degradation. Most of it has nothing to do with what we run, so people mute it, and then they miss the incident that matters.
- **Too scattered.** Each provider formats and threads its updates differently, or not at all. An incident that goes from "investigating" to "resolved" over three hours becomes a dozen unrelated messages.

What we wanted was one place, only the incidents that affect us, and one conversation per incident.

## What it watches

It started with Cloudflare, added GitHub, AWS and DigitalOcean a week later, and since June it also watches Claude and OpenAI: we now depend on model providers as much as on clouds. Most of these publish their status through Atlassian Statuspage, which serves the whole page as one JSON document at `/api/v2/summary.json`: overall status, components, unresolved incidents with their impact and history, and scheduled maintenance.

AWS is the exception. Its public health feed is a list of events in a different format, and it arrives as UTF-16 encoded JSON. The Worker converts it into the same summary shape as the others, keeping only open issues on the services we use, so everything after the fetch is one code path for every provider.

## How it works

![A cron trigger runs the Worker every 10 minutes. For each provider on its own, the Worker fetches the status page (Statuspage summary.json for Cloudflare, GitHub, DigitalOcean, Claude and OpenAI; the AWS Health feed converted to the same shape), diffs it against the last state stored in Workers KV (one key per provider, holding the overall status, each open incident and the Slack thread it was posted in), filters by impact and components, then posts to Slack and saves the new state. Major and critical incidents go to the main channel; minor ones on components we use go to a separate minor channel. Each incident gets one message, later updates go in its thread, and the resolution is also posted back to the channel.](https://lucasespindola.com/posts/tracking-upstream-outages/architecture.svg)

A cron trigger runs the Worker every ten minutes. For each provider it fetches the current summary, reads the previous one from Workers KV, works out what changed, posts the changes to Slack and saves the new state. Each provider runs on its own: if one status page times out or changes its format, the others are still checked and that provider is retried on the next run.

The state is small: the overall status, and for each open incident its status, impact, affected components, last update time and the Slack message it was first posted as. Polling plus a stored diff is less elegant than webhooks, but it is the same for every provider, survives missed deliveries, and needs no inbound endpoint. Our own monitoring catches the symptoms; this Worker brings in the provider’s account of what happened. Status pages often lag and can understate an incident, so ten-minute polling is enough for that job. An incident opened and resolved between checks will be missed.

## Deciding what is worth a message

Most of the value is in what the Worker leaves out. The rules:

| Incident | Posted? |
| --- | --- |
| Major or critical | Yes, unless it only touches components we explicitly ignore |
| Minor, on a component we use | Yes, in the minor channel |
| Minor, on anything else | No |
| Overall status rising to major or critical | Yes |
| Overall status recovering | No: each incident's own resolution already says so |

"Components we use" is a short list per provider, matched case-insensitively against the component names on the status page, for example Actions and git operations on GitHub. The ignore list is the opposite: components that are too noisy or irrelevant even when an incident is major. For GitHub it defaults to Pages, Codespaces and Webhooks. An incident is skipped only if *every* component it touches is ignored; if it also touches something we use, it gets through. Because matching uses names, a provider renaming a component can make incidents slip past the filter until we update the list.

Once an incident passes the filter, the diff turns it into one of four kinds of change:

- **New:** an incident ID we haven't seen.
- **Update:** its status moved, say from investigating to identified or monitoring.
- **Body update:** the provider posted new text without changing the status. We compare the incident's last-updated time, because providers can post "we are still investigating" several times before anything else changes.
- **Resolved:** its status became resolved, or it disappeared from the list. The summary only lists unresolved incidents, so a fix often shows up as an incident that is simply gone between two checks; the Worker treats it as resolved, using what it stored about it.

## One thread per incident

![A Slack message from the status bot: GitHub, new incident, Incident with Actions, 'We are investigating degraded performance for GitHub Actions', affected component Actions, impact major, status investigating. Its thread has two replies: an update from investigating to identified, and the resolution, which is also sent to the channel.](https://lucasespindola.com/posts/tracking-upstream-outages/thread.svg)

The first time an incident shows up, it is posted as a new message in the channel, and the message's ID is saved with the incident's state. Every later change to that incident is posted as a reply in that thread, so the channel shows one line per incident instead of one per update. The resolution is the exception: it is posted in the thread *and* sent to the channel, so people who only skim the channel see that it is over.

Severity decides the channel. Major and critical incidents go to the main channel the whole team watches; minor ones on components we use go to a quieter channel, where they are on record without interrupting anyone. Without a bot token, the Worker can fall back to a plain Slack webhook, which works but can't thread.

## Small on purpose

Each provider is a few lines of configuration: a name, a status URL, the components to watch and ignore, and an optional custom fetch for providers that don't use Statuspage. Notification channels sit behind a small interface too, so Slack is one implementation rather than an assumption baked into the detection logic.

The detection rules have tests built on mock status-page payloads, because that is where regressions hide: a recovery that suddenly posts, a minor incident that leaks into the main channel. Pull requests run the tests; pushes to main run them again, then deploy. A few HTTP endpoints make it easy to poke at: run a check now, show the stored state, show the configuration.

Adding Claude and OpenAI took a ten-line configuration file each, plus registering them. Their incidents now follow the same rules as the cloud providers: one message when something relevant breaks, updates underneath, and a resolution when it’s over. That was the whole job we wanted this tool to do.