MagWorksManaged Intelligence

Magpie

The AI runtime

Packages: @magpie/module-prompt-engine, @magpie/module-model-gateway, @magpie/module-ai-workflow, @magpie/module-ai-evals, @magpie/module-ai-observability.

In plain language

This is the part of the platform that talks to AI models — and the part built most deliberately for the models to change. Prompts are versioned artifacts, never edited in place. Models and providers are adapters behind one gateway, chosen per step with a fallback chain. Every model call is recorded with what was attempted, in what order, at what cost, and why. A change to what runs in production is a promotion with an audit row and, optionally, evaluation evidence attached — and it can be rolled back the same way. Spend ceilings and rate quotas are refusals decided before a provider is ever called.

The mechanism

Prompts are versioned content behind movable pointers

A PromptVersion is append-only, content-hashed (SHA-256 of canonical JSON) at creation, and carries a status (draft → active → deprecated, terminal). A PromptPointer names the current version for a scope. The prompt engine composes a prompt from a version, a variable bag, optional conversation memory and the user turn — interpolating, validating, scrubbing per policy and truncating to a token budget — and returns provider-agnostic segments. It knows nothing about any provider, reads no environment and has no wall-clock without an injected clock.

Resolution is identity-first (ADR-0036). A prompt node carries a load-bearing promptPointerId and the engine resolves it by id — no scope tuple, no naming convention, no inference. Tenant and scope checks run after resolution and before content is loaded, with typed refusals (PROMPT_POINTER_NOT_FOUND, PROMPT_UNAUTHORIZED) that never fall back to tuple lookup. The platform tenant is an ordinary owning tenant; a platform-authored capability's use of its prompts is an authorisation outcome, not ambient magic.

Promotion and rollback are ledgered flips (ADR-0011)

promote, rollback and setStatus are the only paths that move traffic between versions. Each flip writes a self-contained PromptPointerFlip or WorkflowPointerFlip ledger entry carrying both the old and new version ids and content hashes, the pointer version before and after, the reason, the actor, and optionally an EvalRunRef. Promotion is guarded by status, hash match and compare-and-swap on the pointer version. A run is bound to a definition at start, so a flip never affects an in-flight run.

Eval-justified promotion. A promote can require an EvalRunRef proving an evaluation cleared on the exact content hash being shipped; verifyEvalRunRef checks the archive and returns typed errors (no eval cleared, hash mismatch, missing archive).

The model gateway: adapters, capabilities, routing, refusals

Two provider adapters behind one interface. @magpie/module-model-gateway/server is the Amazon Bedrock adapter (Converse API); /server-openai is the OpenAI adapter (chat completions). Each implements the same supports() / capabilities() / invoke() / invokeStream() contract. The two subpaths never share an import graph — the Bedrock adapter imports only @aws-sdk/*, the OpenAI adapter only openai — so a deployment bundles only the providers it declares (INV-PRV-01). Provider is a free-form tag used for audit attribution only; the gateway never branches on it.

Capabilities are declared by the adapter, consumed by the gateway (ADR-0020). Every supported model declares a frozen nine-field ModelCapabilities record — streaming, toolUse, responseFormatJson, structuredOutputJsonSchema, vision, maxContextTokens, maxOutputTokens, and identity. Declarations describe this adapter's current behaviour, not the model's theoretical capacity; flipping a bit and implementing the mechanism land in the same change. Routing chains are validated against declarations at workflow-definition validation time; an incompatible chain is refused with CAPABILITY_MISMATCH before anything runs. Nothing queries capabilities at runtime.

Routing is per step (ADR-0012). Each prompt node declares its primary model, an optional emergency-fallback chain (with which error codes trigger it) and a retry policy. Because routing lives in the workflow definition, a routing change flips the definition's content hash and produces a new version and an audit entry for free. Fallback is a failure-recovery move ("ship something"), not steady-state routing: the primary's prompt content is re-invoked on the fallback model.

Provider-level fallback with attribution (ADR-0021, ADR-0022). The gateway walks [primary, …fallbacks]; each candidate is served by the first adapter that supports it. Every response and every thrown error carries attemptedModels: one entry per chain position with its chainOrdinal, provider, and a status — attempted, failed, not_attempted_due_to_refusal or skipped (with a closed set of skip reasons). Replay reconstructs the routing decision tree from that record alone. The gateway is the single retry authority; adapter-level SDK retries are disabled.

Cost ceilings and rate quotas are deterministic refusals (ADR-0022). COST_CEILING_REFUSED and RATE_QUOTA_REFUSED are typed, non-retryable gateway errors with fixed-shape details — scope, ceiling or quota id, a policySnapshotHash pinning the exact policy at refusal time, actual versus limit, and the window. A refused candidate never reaches adapter.invoke(); the refusal is decided pre-flight and lands on the AiExecutionRun ledger row. Quota accounting is idempotency-aware: a redriven run cannot burn through a quota the original already consumed. Policies (global / tenant / workflow scope; per run, day or month; rolling windows) are operator-authored in the control plane.

Structured output is enforced at the boundary (ADR-0039). When a step asks for responseFormat: "json" and the resolved model's capability declaration supports it, the adapter engages the provider's constrained mechanism — forced tool-use on Bedrock Converse, native response format on OpenAI — so a well-formed object is produced by construction. Onboarding a model is a declarative registry change (toolChoiceMode, capability bits, a test row, a live smoke), never adapter code. When constrained output is unavailable, a layered fallback applies and every step is observable — degradation is measured, never silent.

Streaming (ADR-0024). invokeStream yields typed chunks plus an authoritative finalize(); chunks are never authoritative, retry and fallback apply only before the first chunk, cancellation is a typed finish reason with usage attributed to observed spend, and exactly one ledger row is written per streaming invocation at finalisation. A streamed run is replay-equivalent to its unary terminal truth. Implemented for both adapters.

The workflow kernel

The kernel owns workflow semantics and run truth; executors (in-memory, DynamoDB, Step Functions) are interchangeable implementations of one contract. Definitions are typed, validated before a run starts (dangling references, unreachable nodes, cycles, invalid delays, partial retry configs, incompatible routing), and content-hashed. Node kinds: prompt, condition (a deterministic, total, JSON-serialisable predicate language), tool, human_review, wait.timer, terminal. The transition engine is pure — no side effects, no clock, no I/O — over (definition, state).

Tools have real-world side effects and are treated accordingly (ADR-0013). Eight typed error codes classify a failure as retryable or not; a per-tool retry policy re-invokes the same tool with the same idempotency token; there is no tool fallback chain, because falling over from one tool to another is a semantic change, not a routing decision. Every invocation writes a ToolInvocation ledger row with its error class.

Human review is an action with edit and resume (ADR-0014). A reviewer approves or rejects, optionally with an edited output that downstream nodes consume from run state. Resolution is append-once; a duplicate resume with a different edit is a typed conflict, not silent corruption. A no-op edit is detected by canonical comparison and recorded as no edit. The review row carries the full edited body; the ledger row carries hashes only.

Output projection (ADR-0037). A terminal's output is projected from run state by literal, path or first_present — an ordered candidate list where the first defined value wins (null, false, 0 and "" are present values). A routed capability thereby exposes one canonical output, and the winning source path is recorded on the terminal node run.

Replay and recovery tooling. inspectRun, compareRuns, replayRun and redrive, with the magpie-workflow CLI, validate and operationalise substrate truth; a ReplayEvent is itself a trust-bearing ledger entity.

Evaluations and observability

@magpie/module-ai-evals defines EvalCase (input plus assertions — containsText, matchesRegex, refusedWith, costAtMost, tokensAtMost, finishedWith, scoreBand and others), a pure runner with injected invoker, per-case and per-run cost ceilings, a failFast mode, and a KMS-encrypted S3 archive keyed by date and run id. Every result is bound to a specific promptVersionId + contentHash (or workflow definition + hash); an eval run aborts if any successful invocation lacks version identity. Snapshot reporter and comparator give regression baselines; the magpie-eval CLI integrates with CI.

@magpie/module-ai-observability aggregates the ledger into typed RunSummary and TenantCostSummary — cost, tokens, latency, routing, retries, human edits, outcome — in one call.

Design notes

  • Why identity-first prompts (ADR-0036). Resolution by shared attributes collided in production the day a second capability bound the same model at the shared scope — it silently resolved the other capability's prompt and produced fluent, plausible, wrong output. Any scheme keyed on shared attributes collides as the prompt population grows, and the platform is designed for tens of thousands of prompts.
  • Why self-contained flip rows (ADR-0011). Investigating an incident months later must not require joining back to a version row that may have been tombstoned, renamed or changed status. The flip entry alone tells the full story.
  • Why validation-time capability checks (ADR-0020). Runtime guessing is non-deterministic and contaminates the audit trail. A workflow either validates against the adapters' declared capabilities or it does not run.
  • Why refusals rather than throttling (ADR-0022). Economic governance is substrate safety, not a billing feature — the same typed-refusal infrastructure as an invalid request, ledgered on the same row, so a runaway workflow cannot exhaust spend silently or starve other tenants.
  • Why constrained decoding (ADR-0039). Structured emission that is prompt-instructed, caller-parsed and prose-degraded is a platform-wide posture, not a bug in one consumer. Moving the guarantee to the gateway makes the instruction load-bearing rather than aspirational — and makes model onboarding data, not code.

Sources

  • magpie/docs/adr/0011, 0012, 0013, 0014, 0020, 0021, 0022, 0024, 0036, 0037, 0039.
  • magpie/packages/module-model-gateway/src/server/bedrock-adapter.ts, src/server-openai/openai-adapter.ts, src/structured.ts, src/structured-stream.ts.
  • magpie/packages/module-ai-workflow/README.md (promotion runbook), src/core/types.ts, src/replay/.
  • magpie/packages/module-ai-evals/README.md.