Event spine and audit ledger
Packages: @magpie/events, @magpie/ledger, @magpie/observability,
@magpie/core.
In plain language
Everything that happens on the platform is either an event (something occurred, and other parts of the system may react) or a ledger entry (something meaningful changed, and the record of it can never be altered). Parts of the system talk to each other only through events; they never call each other directly. The ledger is the platform's memory of what happened — append-only, indefinitely retained, and the source every report and replay reads from.
The mechanism
Every event carries the same envelope. eventId, a dotted versioned
eventType, eventVersion, occurredAt, recordedAt, tenantId,
actorId, correlationId, causationId and a typed payload. The
publisher validates every event against this envelope and refuses events
that fail; a compliance test exercises every registered event type.
Event handlers are idempotent. A handler invoked twice with the same
eventId produces the same outcome as once. Event schemas are versioned and
immutable: a semantic change ships as a new version, and the prior version's
schema is preserved while anything references it. Every subscription has a
dead-letter queue and an operator alert.
Cross-module communication is event-only (ADR-0007). Modules must not
invoke each other — no function calls across package boundaries, no
Lambda-to-Lambda invokes, no reads of another module's tables, no parsing of
another module's identifiers. Every event has exactly one owning module,
which declares it with a schema and lists it in its manifest. Consumers
subscribe by (name, version); if a consumer needs a different shape, it
publishes its own event under a name it owns.
The ledger is append-only at the storage layer. A fixed, ADR-governed list of entity types is trust-bearing; changes to them are expressed as new entries that reference the prior entry. Every write carries actor, timestamp, causation and correlation identifiers — not optional. Ledger storage has its own backups with longer retention, is recoverable to any point in time, and is never purged.
Erasure is tombstoning, not deletion. Compliance-driven erasure is satisfied by a new ledger entry that redacts subject-identifying fields in the current-state projection while preserving the immutable history. The platform satisfies erasure through projection control; it does not falsify history.
Workflow execution is recorded row-per-transition (ADR-0019). Every
executor emits a WorkflowExecution ledger row at every transition — start,
node entered, node exited, wait armed and fired, resume armed and fired,
condition branched, terminal — with a state-hash chain: each row's
previousStateHash equals the prior row's nextStateHash, ordinals are
contiguous, and the chain anchors on the hash of an empty state. Hashing uses
the same canonical-JSON serialiser that hashes workflow definitions, so replay
is byte-deterministic. Rows are written once; a retried write produces one
row.
Trust-bearing entity types today include: inquiries, alert lifecycle
events, identity and membership changes, role grants, control-plane changes,
AiExecutionRun (every model invocation), WorkflowExecution (every
transition), HumanReview, ToolInvocation, PromptPointerFlip and
WorkflowPointerFlip (every promotion or rollback), AlarmFire (every
CloudWatch alarm), RecoveryEvidence (every recovery drill), ReplayEvent
and RecordMigration. Extending the list requires an ADR.
Observability is structured JSON logging with a fixed base schema
(timestamp, level, service, environment, requestId,
correlationId, tenantId, actorId, message), and a correlation id that
is generated at the request entry point or event publisher if absent and
propagated to logs, traces and downstream calls. The logger wrapper redacts
credential-shaped values before emission.
Design notes
- Why event-only between modules (ADR-0007). Direct invocation couples modules to each other's deployment shape and identifiers; the codebase contained exactly such a coupling (a handler invoking another module's Lambda by name and recovering an entity id by string-trimming a run id). The rule is absolute — "there is no simple-case exception, no internal-use exception, no for-now exception" — because a foundation with first-class modules needs one rule for how they behave at runtime.
- Why row-per-transition with a hash chain (ADR-0019). Before it, execution truth was inferable only indirectly from model-call, tool and review rows; the executor's own transitions were absent, so run summaries could never say a run was complete. Emitting every transition with a verifiable chain makes replay a mechanical guarantee rather than an observation.
- Why tombstones (constitution INV-LED-05). Erasure is legally consequential and projection control is genuinely tricky; the platform satisfies the obligation without ever deleting or rewriting a historical entry.
Sources
magpie/docs/architecture-constitution.md— INV-EVT-01…04, INV-LED-01…06, INV-OBS-01…04.magpie/docs/adr/0007-cross-module-event-invariant.md.magpie/docs/adr/0019-workflow-execution-per-transition-rows.md— INV-WFE-01…03.magpie/packages/module-ai-workflow/src/replay/validate-transition.ts— the chain validators.