Changelog ·

Confirmation rounds no longer arrive all at once

A confirmation round, a Check now and an alert-mode round now reach a site spread over about six seconds from the voting probes only, and server-wide re-checks are rate-limited; a burst of our own requests could push a small site over its timeout and confirm an outage that was not there.

monitorsprobes

Since 11:04 UTC the probes in a confirmation round start one after another, about 400 milliseconds apart, instead of at the same instant, and only the fifteen voting probes take part. A site under confirmation therefore sees about three of our requests a second, spread over six seconds, where it used to see eighteen in one second. The same applies to alert-mode rounds and to «Check now», and a Check now on several monitors runs one round every six seconds. Nothing changes in how an outage is decided: the quorum rule, the two agreeing rounds and the four attempts are as before.

We changed this after a false alarm today on a customer's site. Two sites on one shared server answered 500 and 503 on their very first check, and our server-wide re-check sent every probe at the other sites of that server at once, about 180 requests within a second. A third site, which answers a single visitor in under a second, needed more than ten seconds under that burst, timed out on every probe, and was confirmed down although visitors could open it; every recovery check was the same burst. Server-wide re-checks now run at most once per server every ten minutes and never because of a brand-new monitor's first result. We are sorry for the alarm. We are also building recognition of this pattern, a site that passes scheduled rounds and fails only under our own bursts, so that it is caught on its own.

← All updates