The knowledge substrate
Packages: @magpie/knowledge, @magpie/text-extraction, @magpie/chunking,
@magpie/embeddings, @magpie/vector-store, @magpie/retrieval,
@magpie/knowledge-reconciliation.
In plain language
A Solution that answers from a client's own documents needs those documents stored, broken into retrievable pieces, turned into vectors a model can search, and served back with a citation. The substrate does all of that with one discipline throughout: the document is the truth, everything derived from it carries the document's fingerprint, and the search index is a cache that can be thrown away and rebuilt without touching the truth. That is what lets the platform change embedding models, detect when a source has drifted, and erase a client's corpus provably.
The mechanism
Documents are records; versions are append-only (ADR-0023).
KnowledgeSource, Document, DocumentVersion and Attachment are typed
records on the records primitive, written only through KnowledgeService
inside a unit of work (direct record writes are refused by a source-grep
test). There is no update or delete method for a version — new content is
a new DocumentVersion with a fresh id and the parent's currentVersionId
pointer moves. Tenant scope is inherited from the transaction; callers
cannot supply a tenant that differs from it.
The pipeline is a chain of contracts. Bytes → normalised text
(TextExtractor: frozen segment kinds, hard size caps, ordinal contiguity)
→ retrieval units (Chunker: derived chunkId, segment provenance
preserved) → fixed-dimensional vectors (Embedder: frozen dtype union,
dimension cap, range validation) → tenant-isolated storage
(VectorStore) → ranked, citation-bearing chunks (Retriever +
ContentResolver). Each stage is a port with a shipped implementation and
room for others behind the same contract.
The hash-binding chain. sourceContentHash from the DocumentVersion
is carried verbatim through every intermediate artifact — extracted text,
chunks, embeddings, stored vector metadata, retrieved chunks — together with
extractorId, chunkerId and embedderId. Verbatim-preservation tests in
every package pin the convention. This is what makes drift detectable end to
end without re-fetching bytes.
Vectors are cache, not truth (ADR-0031). A vector's existence does not
imply a version exists and vice versa; the lifecycle authority is
KnowledgeService. Every stored vector retains enough lineage to identify
its source — sourceId, documentId, versionId, chunkerId,
embedderId, modelId, dimensions, chunkOrdinal, sourceContentHash —
plus the framework-injected tenant in the storage key and as a query filter
on every search. Deleting or rebuilding vectors is a reconciliation
operation, never a knowledge mutation.
Tenant isolation at the vector boundary. Storage keys are
<tenantTag>||<chunkId>; the tenant filter is injected by the framework on
every query; similarity scores have one "higher is better" semantics across
cosine, euclidean and dot product. The shipped adapter is native Amazon S3
Vectors; other stores plug in without contract changes.
Production embedders are provider-supplied (ADR-0018). The shipped
production embedder is Amazon Titan Embed Text v2. The deterministic
test-fixture embedder refuses to run in production unless an explicit,
audit-visible override is set. Given the same (embedder, model, dimensions, content), an embedder must produce a byte-identical vector — the
foundation of the hash chain.
Drift reconciliation. buildKnowledgeDriftReconciler compares stored
sourceContentHash against current source state through a pluggable probe,
emits RecoveryEvidence per finding, and is detect-only unless an operator
opts into delete-stale.
Design notes
- Why append-only versions (ADR-0023). Point-in-time reconstruction and drift detection both depend on the version's hash never changing; a mutable version would break the substrate guarantee reconciliation depends on.
- Why vectors are not canonical (ADR-0031). "Vector systems may cache. They may not become canonical knowledge state." Without the boundary, every store would claim its own truth and reconciliation would become meaningless.
- Why a provider-supplied embedder with a production refusal (ADR-0018). A fixed warning string is documentation, not governance; the house rule is invariant-first — an unenforced invariant does not count.
Sources
magpie/docs/adr/0018,0023,0031.magpie/README.md§"Wave 10" — the package contracts and the hash-binding chain.magpie/packages/vector-store/src/types.ts—VectorRecordMetadata.magpie/tests/unit/vector-store-s3.test.ts— metadata round-trip.