Resources / Watchdog

Watchdog

The watch over a component that runs the release controller itself (spec 0047 section 3 'Not being able to roll itself back'): a bad build of it has nothing behind it to roll it back. The component's own Worker writes a heartbeat on each pass; a pass in another Worker reads it, and when a new version took all the traffic and the heartbeat stays away for the grace time it moves the traffic back to the previous version, once, asks the active release to record it and opens an Incident.

Overview

APIplatform.etorm.com/v1alpha1
Resourcewatchdogs (cluster-scoped inside its cluster)
Short nameswd
Available inthe platform only
Statuswritten by a controller through /status

Minimal example

apiVersion: platform.etorm.com/v1alpha1
kind: Watchdog
metadata:
  name: example
spec:
  component: "<component>"
  script: "<script>"

Spec

FieldTypeRequiredDefaultAllowedDescription
componentstringyeslength 1 to 63The component of a release this watches (for example operator).
scriptstringyeslength 1 to 63The name of its Worker script at the provider.
graceSecondsinteger600120 to 3600How long a new version has to report after it took all the traffic.

Status

FieldTypeRequiredDefaultAllowedDescription
heartbeatobjectThe component's own sign of life (written by the component's Worker, not by the watchdog).
heartbeat.atintegerUnix time of the last heartbeat.
heartbeat.bystringWho wrote it.
observedobjectThe version that has all the traffic, the one before it and when it took it.
observed.currentstringThe version id with all the traffic.
observed.previousstringThe version id it replaced.
observed.sinceintegerUnix time the watchdog first saw it with all the traffic.
observed.commitstringThe commit the deploy recorded on this version (its tag); empty when the deploy recorded none. The invariant build-matches-deploy compares it with the build the host reports.
rolledBackFromstringThe version the watchdog moved the traffic away from (it never does it twice to the same one).
phasestring (one of)Healthy, Waiting, RolledBack, NeedsHuman, UnknownHealthy when the component reports after its latest version; Waiting during the grace time; RolledBack after the one rollback; NeedsHuman when there is nothing safe to do; Unknown when the provider cannot be read or the traffic is split.
messagestringWhat the watchdog sees.
lastCheckedAtintegerUnix time of the last pass.

Columns in listings

ColumnFrom
Component.spec.component
Phase.status.phase
Version.status.observed.current

All resources