Execution primitives
Packages: @magpie/idempotency, @magpie/jobs, @magpie/scheduler,
@magpie/webhooks, @magpie/reconciliation, @magpie/blob,
@magpie/control-plane.
In plain language
Real systems retry, time out, get duplicate deliveries and drift. These packages make that safe: an operation that is retried produces its effect exactly once; long work runs as durable jobs with explicit retry and dead-letter rules; timers fire through the same job path; incoming webhooks are verified before anything happens; drift between what should be true and what is stored is detected and repaired with evidence; files are stored under a tenant prefix with encryption mandatory; and runtime switches (including kill switches) are versioned configuration, not code changes.
The mechanism
One idempotency primitive, scope-tagged (ADR-0015). A single
IdempotencyStore contract serves six scopes — API, EVENT, LEDGER,
JOB, WEBHOOK, RECONCILIATION. A key is (scope, tenantId, key, requestHash); tenant is part of the partition key. The request hash is
always framework-computed by canonical SHA-256 over RFC 8785
canonicalised JSON — there is no caller-supplied hash and no escape hatch —
and the algorithm is versioned on the row. The state machine is
in-flight → completed | failed-permanent | released; completed returns
the cached response, failed-permanent returns the same failure without
re-executing, and a stale in-flight lock is taken over after a bounded TTL
so a handler that died mid-flight does not wedge the key. Events, ledger,
jobs, webhooks, scheduler and reconciliation all consume it.
Jobs. A frozen JobEnvelope v1 contract and a wrapJobHandler
orchestrator with idempotency, a kill switch, blob-payload hydration and
classified retry / dead-letter semantics. Every job definition declares its
retry policy, max attempts, timeout and dead-letter target explicitly — no
job inherits implicit defaults.
Scheduler. Owner-scoped one-shot timers backed by EventBridge Scheduler
with deterministic diff-replace; fire targets are JobEnvelope deliveries
to SQS, never direct Lambda invocation, so a timer is just a job that
starts later. Scheduled rules are defined in infrastructure code with drift
detection; console-created rules are prohibited.
Webhooks. wrapWebhookIngress implements verify → persist receipt →
enqueue job, with an HMAC-SHA256 reference verifier. Unverified webhooks are
rejected with no side effects; ingestion is idempotent on the provider's
event id; processing is asynchronous on the job runtime.
Reconciliation. A generic detect → apply pipeline. Reconcilers compare
expected against stored state, emit a RecoveryEvidence ledger row per
finding and per run, and are detect-only by default with repair as an
operator opt-in. Knowledge drift reconciliation is built on it.
Blob storage. One tenant-prefix-enforced object store with KMS-mandatory put. All file storage is tenant-scoped at the path level; uploads from untrusted callers go directly to storage via signed URLs, never proxied through compute.
Control plane. Runtime configuration with a warm cache, version-gated optimistic concurrency and per-control scope (platform or tenant). Every control-plane change is a ledger event; dangerous controls (kill switches, incident toggles, provider failover) require elevated permission and are rate-limited; values are consulted at invocation time with a bounded cache TTL, never cached indefinitely. Cost-ceiling and rate-quota policies live here as typed, schema-validated resources.
Design notes
- Why one idempotency primitive (ADR-0015). Four new primitives all needed deduplication; letting each grow its own store would have shipped five subtly different semantics. The reference case that shaped the design had done exactly that and paid for it operationally. Caller-supplied hashes were "the single largest source of same-key, different-payload, silent-dedup bugs", so the door was closed on day one.
- Why timers fire as jobs. One execution path for deferred work means one set of idempotency, kill-switch and dead-letter semantics.
- Why detect-only reconciliation by default. Evidence first; repair is a decision an operator opts into, and both are on the record.
Sources
magpie/docs/adr/0015-idempotency-primitive.md(with §7.2, §7.3 amendments).magpie/docs/architecture-constitution.md— INV-JOB-01…02, INV-WH-01…03, INV-SCH-01…02, INV-RCP-01…03, INV-FIL-01…03.magpie/README.md§"What Magpie enables" (Wave 9).magpie/docs/runbooks/— idempotency, control-plane, jobs, webhooks, scheduler and reconciliation runbooks.