Build, Knit, Dream

Draft. Sirsh — I inferred this one from the specs (world-model-ingestion, the consolidation notes in agentic/brief.md, and the builds_on / dreamed_from edges in the query layer). The shape is here and the facts are checked against the repo, but the argument is mine rather than yours. Tell me where it is wrong and I will rewrite it.

Most systems that claim to have memory have storage. You give them a document, they chop it into chunks, they embed the chunks, and when you ask a question they find the chunks that look most like your question. That is retrieval, it is genuinely useful, and it is roughly where the industry stopped.

It stopped there because the first step is the only one with an obvious API. The other two are harder to name, so we have started calling the whole thing build, knit, dream.

Build

Building is the part everybody has. Something arrives, whether that is a PDF, a spreadsheet, a message in a channel or a page somebody clipped, and it becomes rows. In Percolate a file becomes a resource, the resource becomes chunks, the chunks get vectors and a lexical index, and if the thing was tabular it also becomes a Parquet dataset you can query directly. Nothing has to run after the upload; the pipeline is a workflow and the upload starts it.

The important property of this stage is that it should be lossless and boring. It is not the place to be clever. Whatever interpretation you apply while building is an interpretation you are stuck with, because you will throw the original away and then discover six weeks later that you needed the bit you discarded. Build is a faithful transcription with an index on it.

Knit

Knitting is where the content stops being a pile.

A chunk that mentions "Acme" and a chunk that mentions "Acme Corp" and a row in a CRM export with an acme_corp key are all about the same thing, and until something says so they are three unrelated pieces of text that happen to score well against the same query. Knitting is the work of turning mentions into entities and entities into a graph: extract what a piece of content refers to, resolve those references against what you already know, and write the edges.

In Percolate the graph is not a second database. Nodes, node keys and edges are ordinary Postgres tables with a property-graph view over the top, which is why adopting it costs no migration. A node carries several keys, a canonical form and whatever short forms and aliases turned up, and resolution is a cascade: exact canonical first, then short and alias, and only then, if you asked for it, a trigram match. Fuzzy-matching a correctly spelled name is a way to get a worse answer slowly.

Getting this stage wrong is quiet in a specific and annoying way. A trigram query over the keys table returns one row per matching key, so an entity holding a canonical, a short form and an alias comes back three times for a single misspelling. Nothing fails. A caller that hydrates each row fetches the same company three times and pays for it, and a language model on the other end has no way to notice. We only found it by running the mode against a fixture with more than one tenant in it and reading the output properly, which is a theme.

Extraction belongs at this stage rather than at build time. Content lands in a staging area rather than going straight into the graph, because an extraction is a proposal and proposals should be reviewable before they become facts about your world.

Dream

The third stage is the one we care most about and the one that is hardest to justify to somebody looking at a roadmap.

Dreaming is background consolidation: a process that reads what you have accumulated and produces something that was not in any single piece of it. A summary of a week. A pattern across sessions nobody asked about. A note that two projects you filed separately have been converging for a month. It runs on a schedule rather than in response to a question, and its output is stored alongside the raw material rather than instead of it.

The reason it needs a name is that it is structurally different from retrieval. Retrieval answers a question you asked with material that already existed. Consolidation produces material that did not exist, at a time when nobody is waiting, and the value only shows up later when a retrieval hits it. If you measure your system on query latency you will never build this, because it costs money and improves nothing you are currently measuring.

The mechanism turns out to be cheap once the rest is in place, which is the argument for building the rest in the order described. A consolidation run is an agent run with trigger_kind = 'scheduled' instead of a live session, started by the workflow engine as a task rather than by a user typing. It reads and writes the same runs and messages shape an interactive conversation does. There is no second system, no separate pipeline, no bespoke batch framework. What it produces gets builds_on and dreamed_from edges back to the material it came from, so an insight is always traceable to what informed it.

That traceability is the whole reason we are comfortable letting a model manufacture claims about your data unsupervised. A dreamed node is not asserted as true. It is asserted as having been derived, from these specific things, at this specific time, by this specific agent, and every one of those is queryable. If it turns out to be wrong you can find everything downstream of it.

The order matters

You can build without knitting and you will have search. You can knit without dreaming and you will have a knowledge graph, which is better than search and still fundamentally reactive. You only get something that behaves like memory when all three are running, because human memory is not a lookup either; the consolidation happens while you are not using it.

The thing we would push back on, if we were reading this, is whether the third stage is worth the compute. Our honest answer is that it is worth it exactly when the same corpus gets asked about repeatedly over a long period, and not otherwise. A system somebody queries once has nothing to consolidate. A system that has been accumulating your working life for two years has almost nothing but consolidation worth doing.