Skip to content

ADR 0026: Operational health probes and application-owned backup

Status

Accepted

Context

Duckbill™ operators need per-account sync lag, queue/DLQ depth, and reconciliation drift without reading mail. NFR §10 and Sync §15 require those views. Deployment §§80–81 require liveness versus readiness, and a provider outage must not mark the whole application unhealthy.

Application-owned state (rules, corrections, classifications, attention, audit, sync metadata) cannot be rebuilt from the provider. NFR §§3, 8–9, and 16 plus Deployment §§108–115 require a backed-up inventory, documented RPO ≤ 24h and RTO ≤ 4h, recovery mode that pauses mutations, and a recorded restore rehearsal.

Decision

  1. Expose GET /api/health/live, GET /api/health/ready, GET /api/health, and GET /api/providers/health. Liveness is process-alive. Readiness is database plus required configuration. Provider degradation is per-account and never a 5xx for the application process.
  2. Assemble health in @inbox-agent/application and emit metrics from @inbox-agent/observability. Every health document is redacted with the existing privacy rules. Missing recent sync is stale, not empty success. Cursor tokens are not returned; only cursorPresent and checkpoint status.
  3. Persist recovery mode, API latency samples, durable rules/corrections, and backup-run records in migration 0018_ops_health.sql.
  4. Backup application-owned tables only. Skip provider tokens, OAuth state, and regenerable caches. After restore, enter recovery mode (global automation pause), reconcile, then keep mutations paused until an operator clears recovery.
  5. Record restore rehearsals against disposable PGlite with synthetic fixtures. No production mailbox content.

Alternatives

Consequences

/ops shows the same redacted health document as the API. RPO 24h / RTO 4h are release targets, not provider SLAs. Triage UI and write-back commands stay on other streams. Epic MAIL-11 remains open until remaining observability work lands.