Every night, a chain I built collects whatever new documents have landed in a fund’s records, has an agent read them and decide what each one asserts, tests that against everything already on the books, and — when nothing already settled moves — writes it into the register itself. Most nights it finishes without me. Some nights it stops and leaves a note.

It exists because of a moment I reached as a partner in a venture fund, and I doubt I am alone in it. The work gets done — more or less right, often painfully, always slowly — and one day you conclude you need to change the tool. What I wanted was easy to say and hard to buy: a system whose picture of the business is tied to the real documents, without depending on anyone to key them in or point at them. Every tool I looked at asks people to do the work, and then to narrate the work into the tool. That second job is where the fragility lives — when the week is busy, it is the part that does not get done. I wanted something that blends into the job instead of sitting on top of it.

So I went and looked at what is sold. Every quote comes with two costs it does not mention: owning the thing, and running it. Large organisations can afford that; I had the question, not the department. AI turns up in exactly that gap, promising the outcome without the project.

Nothing here replaces anything. Records like these are kept the way most organisations keep them — by people, document by document, correctly, and slowly. I wanted to see what a chain of agents would do with the same paperwork, running beside them.

A few days ago it stopped on a signed custodian statement. It had read the document correctly. It had resolved the holder without ambiguity. It stopped because the register — the one book everything else gets checked against — had no way to carry what the document said, and because I had forbidden it from deciding that on its own.

That block was not a defect. It was the design. I did not have to build it this way: I could have shipped an agent on its own, arbitrating from a prompt. It would have passed the first tests, passed the demo, and broken in production. I know, because I ran the chain without me, in a sandbox, to see what it would do unattended. Same documents, same protocol. Then I gave it the same ten documents twice, twenty minutes apart: four facts the first time, seven the second. I did not need an eleventh document — the variance was already there. The chain I actually run has since put hundreds into the register.

So I shipped the other one. And ever since, I have been the man inside the Mechanical Turk — the 1770 chess automaton with a man folded into the cabinet. It makes the moves. I choose the ones it cannot.

The rule fits in one sentence

As long as it does not call the past into question, it validates without me.

That sentence draws a line through the chain. Everything upstream of it is agentic: the agent reads a document, infers what happened, and produces a structured list of assertions. Everything downstream is not. The chain takes that list and confronts it, deterministically, with the register as it stands — and with a thousand-odd checks I wrote by hand as I built it: sums that must tie, figures that cannot contradict what is already on the books. It runs that confrontation on a copy before it writes anything anywhere.

That was my opening design guess, written before the chain had ingested a single document — not something an accident taught me later. It has held through every document since, and it turned out to be the cheapest protection in the system: nothing that would disturb a settled figure gets written while I am not looking.

That is the whole point. The split between what the machine settles and what a human settles is not handed to you by the state of the art. It is a decision, and it is yours.

One document in twelve still needs a person

Every night the chain sweeps close to three thousand documents across the firm’s records. Over seven weeks, a couple of hundred of them changed something in the register. About twenty stopped and asked for a person.

That is roughly one document in twelve that the machine would not close on its own. I did not see that coming when I started — it landed on me halfway through the build. And not one of those decisions has yet turned into a rule that closes the next one, which means the whole thing runs at the speed of my attention. That is a throughput problem, not a design achievement.

Almost none of the stops were reading errors. The agent knows the trade — what a contract says, what a signature commits. That part is in the model, and it covers most of what lands.

It stops when the answer stops being generic. Is this document still the one in force here? Does this one replace that one? Is this an error in the paperwork, or a piece of context nobody ever wrote down? None of those are questions about documents. They are questions about one organisation, and the model has never been near it.

The custodian statement stopped there. Nothing in the file said what to do, and nothing in the weights ever could. It did not guess, because it is not allowed to. Compliance did what a machine cannot — checked, arbitrated, had the document re-issued carrying what it should have carried in the first place. The chain took it without a word from me and carried on.

One of those questions is harder than the rest. The agent knows what an amendment is, can read it, can derive what it changes. It has no rule for putting it into force. That rule exists in no form at all — not code, not prompt, not paper. Nobody ever wrote it.

There are two ways an answer becomes binding: a document settles it, or someone does. The second is not reading and it is not reasoning. It is standing — a position, not a skill.

That is what a person does when they key data into a system. Most of it is transcription; some of it is a call, made at the speed of the typing and recorded nowhere. Every set of records is already full of them. The agent’s only difference is that it cannot make one silently — not out of scruple, it has none, but because the rule sits in the code. In a prompt, it would have taken the leap.

That is not a gap in the tool. It is a gap in the governance — and the tool is what made it visible. Which is also why the line does not move when the models get better.

But on average, humans and AI together do worse than the better of the two alone

The question is not whether to put a human in the loop. It is whether, at the exact point where you put one, the human knows something the machine does not.

Others have measured that better than I could, so I looked — starting with work that could prove me wrong.

A 2024 review pooled 106 experiments, published between 2020 and 2023, that measured people alone, AI alone, and the two together. You would expect the pair to beat its best member. On average, it did slightly worse — especially when the task was a decision.

The average hides a split. When the machine was already better at the task, adding a person made the result worse. When the person was better, the pair beat both. The authors’ best guess: a person who is better than the machine is also better at telling when to trust it. One who is not leans on it when it is wrong and ignores it when it is right.

In my register, there is no doubt which one to trust. On can this document supersede that one, in this house, the machine has no rule at all. I am not marginally better there. I am the only one holding the answer.

Almost every one of those experiments had a person make the final call, case by case, after seeing the machine’s answer. My chain is not built that way. The split is set in advance: the machine settles what passes its checks, and only what does not comes to me. That is the arrangement the authors point to as the next thing to test. Only three of their experiments tried it — too few to conclude anything.

And the same question tells me where my design would be wrong: anywhere the model already does it better than I do.

Curriculum, manufactured in advance

Both roads answer the same question: how does expert judgment get into a system? Mine collects it, one arbitration at a time, as the work throws them up. There is another, and it is not hand-waving: you can manufacture it in advance.

In early September, a16z argued that the durable advantage in vertical AI is not owning the record but learning the job — and that memory is not learning. Keeping context tells you what was decided last week. It does not tell you whether the work was good, why an expert changed it, or what to do differently next time.

Their example is Harvey, and Harvey documented it themselves. They post-trained an open-weight model via asynchronous reinforcement learning across roughly 1,750 simulated legal task environments, each one a matter handed down by a partner, graded against an expert rubric averaging around fifty criteria. And, in their own words:

We did not use any customer data in any of our post-training efforts.

— Harvey, post-training update

They did not wait for production to hand them arbitrations. They manufactured them, in bulk, before shipping.

It works, and I think it settles one half of the problem. The half it cannot settle is named in the a16z piece itself: there are two curricula, not one. There is the profession — how a good practitioner does the work — and there is the institution: this house’s templates, precedents, risk thresholds, escalation rules. Lessons about the profession improve the product for everyone. Lessons about one firm improve it for that firm only.

A simulation can manufacture the first. It cannot manufacture the second. The institution’s curriculum exists nowhere except in the arbitrations its own authorities render, on its own documents, in production. That second curriculum is the context gap I described in the first piece — this time at the level of a house, not a person — and this is the only place it gets produced.

It does not move onto the machine

Everything above is a description of where I chose to put that authority, in a house I built the chain for. Whether it can be moved onto the machine at all is not for whoever builds the thing to decide. It is decided by whoever has the standing — a committee, a regulator, a court — and one small case has already made the answer public.

In 2024, a Canadian tribunal heard a passenger who had relied on an airline’s chatbot for a fare policy the chatbot had invented. The airline argued that the chatbot was a separate entity, responsible for its own actions.

The tribunal refused. The chatbot was part of the airline’s website; the airline answered for what was said on it.

What matters is the attempt: a company tried to move authority onto the machine, and a court declined to let it. You are only ever choosing where it sits — or discovering where it sat, in a dispute.

One number will not tell you how you are doing

You need two, and neither is about a person — both are about a position.

The first is human frontier — how often it needs a human authority. Of the cases that carry a stake, how many come back to someone with the standing to settle them. Here, about twenty out of a couple of hundred — one in twelve.

The second is learning impact — what a human arbitration buys, counted in future cases one decision closes on its own. Here, zero. Nothing I have arbitrated has yet become a rule that covers the next case. The chain records my answer, validates it, keeps it. It is not equipped to generalise from it, because I forbade it from judging.

The couple is what speaks, never either alone. Low on both is a sealed system — a pacemaker, an avionics stack, a satellite: no authority can be there while it runs, so every judgment is frozen before it leaves. Low frontier with high impact is new software — the arbitrations have been rendered and have become rules the thing applies on its own. High on both would be a copilot, asking often and keeping what it is told. That may be a form factor still arriving; I have not seen one in wide use.

Which leaves the corner where it asks constantly and nothing stays: the forever assistant. That is where my chain sits, measured. The other marker on that map is the same work once my decisions have become rules, and the arrow between the two is the build ahead.

The frontier is the point

The machine does what it does better and far faster than I do: find a document, read it, extract what it asserts, confront it with a rule, and record it. It stops where we know a human holds the decision. The work ahead is to move that frontier — carefully, and in a way that survives inspection.

Which is to say the generalisation has to come, but its application has to stay determined, traced, logged and auditable. Not for the sake of compliance theatre. Because the day I stop being present at every step, the only thing that still gives me authority is the ability to come back on a decision I was not there to make.

So here is the falsifiable version. In six months, if the chain has no auditable learning of its own, and the share of cases that come back to me has not fallen, then I was wrong — not about the design, but about my ability to build past it. And I will have built a very expensive Mechanical Turk.

Sources

1. Vaccaro, Almaatouq & Malone, When combinations of humans and AI are useful: a systematic review and meta-analysis, Nature Human Behaviour, 2024 — 106 experiments from 74 papers, 370 effect sizes.

2. Harvey, post-training update — asynchronous reinforcement learning, ~1,750 agentic legal task environments, ~50 rubric criteria per task, no customer data.

3. a16z, The Incumbents Are Coming, September 2026 — memory is not learning; the profession’s curriculum and the institution’s are not the same curriculum.

4. Moffatt v. Air Canada, 2024 BCCRT 149 — the tribunal rejected the argument that the chatbot was a separate entity responsible for its own actions.

There is a version of the AI augmentation story that has become the default: everyone gets access to the same tools, and the people who use them best win. The playing field flattens. Expertise becomes less of a moat.

I ran a version of this experiment, unintentionally, over the past year. The conclusion is different.

Past the ceiling

I started where everyone starts: new chats, general prompts. Then structured prompts, system instructions. Then Claude projects — the kind of discipline that already puts you ahead of most users. Then CoWork, Claude’s AI-native workspace for handling documents and complex files. Each step was genuinely good. Each step had a ceiling.

My technical background would not let me stop there. Not by ambition — by dissatisfaction. The tools plateaued too early. The context window filled and compressed. I could feel the ceiling. So I kept going, past what I would recommend to anyone without a technical baseline: a proprietary stack, a universal brain, a memory framework I designed and told the system how to maintain. Not because I wanted to overkill. Because overkill was the only way to get what I needed.

The result, twenty-four months in: a system that could cross-check a financial report against eighteen months of portfolio context in the time it takes to write an email. One instruction. One output. No iteration.

Then I started working alongside people who were using the same tools.

Three types of AI users

The gap was visible immediately. Not in the tools — everyone had access to ChatGPT, Claude, the usual suspects. Not in intelligence or work ethic. In one thing: context.

Context, in this piece, means two things at once: what you know, and what the system has access to. The gap I am describing is the distance between the two.

There are three types of AI users, and they live on different planets.

The first opens a new chat each time. General prompt. The output is a function of what they asked, which is a function of what they already knew. The tool is as blind as the user. This is where most knowledge workers are today.

The second is the power user of native tools — Claude projects, AI-integrated spreadsheets and documents, co-working features pushed actively by Anthropic and OpenAI over the past year. A growing population, actively cultivated. Better output. But they remain prisoners of the vendor roadmap: they use what is put at their disposal, nothing more. And the context window still saturates — send a thirty-page document, run three or four iterations, and the system starts compressing. What looked like memory is really just a long conversation with a ceiling.

The third is model-agnostic. They use the raw model and build the context themselves, independent of any product team’s priorities. Obsidian, custom memory frameworks, proprietary stacks. Same drive as the level-two user — just pushed past the point where off-the-shelf tools satisfied them.

Fifteen to twenty flags

As a partner at a VC firm, when my analysts — especially those coming from a pure business background rather than a technical one — received a data room, they were operating at level one or two. New chat. General prompt. When I worked a data room, my agents already knew the sector thesis, the portfolio comparables, the specific flags we look for at this stage, the terms we had seen and rejected. The question I asked was simple. The answer was specific, cross-referenced, and dense.

The difference in output was not subtle. On financial documents, my agents routinely surface 15 to 20 flags per review — not errors in isolation, but contradictions between the document and prior reporting, inconsistencies between what the team said last quarter and what they are presenting now. The same documents, reviewed without that accumulated context, produced a fraction of that.

The gradient

The impact is not uniform. The context gap is most visible in domains where things repeat, recur, and must remain coherent: finance, compliance, accounting. Cross-check this quarter’s numbers against last year’s. Flag what changed. Check the document against the term sheet. The agent’s value compounds with every prior document it has processed.

In deal analysis, the gap is real but smaller. The rules are more subjective, the thesis evolves, judgment is harder to encode. Context helps — knowing what we have passed on and why, what patterns tend to break down post-investment in this geography — but it does not replace judgment the way it replaces manual cross-checking.

Understanding this gradient matters. The technology does not work the same way everywhere. Deploying it as if it does is how you get disappointed teams and dismissed tools.

The opposite of what vendors sell

I looked at the vendor market for a solution. I wanted to avoid building everything myself.

What I found was a consistent mismatch. Most AI coworker tools are built on active context construction: they ask you to point at documents, feed a knowledge base, stop what you are doing and write things down. Some connect to your CRM, assuming your data is already structured and clean. Others position themselves as replacements for headcount — a thesis I do not share. These are tools. They augment people; they do not replace them.

The model that actually worked for me is the opposite of what vendors sell. My context grew passively — captured through every interaction, every document reviewed, every correction made. I did not stop to document. I worked, and the system learned from the work.

This works because I am the practitioner with authority in the loop. Every decision I feed into the system carries weight. The context it accumulates is not neutral — it reflects fifteen years of pattern recognition, a specific investment thesis, rules about what we look for and what we pass on. Someone feeding the same system without that grounding would accumulate a different kind of context: shallow rather than authoritative, potentially reinforcing errors rather than correcting them.

What makes context robust is auditability. Not all inputs carry the same weight. An authoritative judgment overrides a shallow one. A signed document overrides an email. An external auditor’s finding overrides an internal estimate. Building a reliable context means encoding not just information, but its source — and the hierarchy of those sources. Without that, the system amplifies noise with the same confidence it amplifies signal.

And here is the gap the vendor market has not addressed: there is no tool today that makes this kind of context shareable, diffusable, and auditable across a team. The level-three user can build it for themselves. Building it for others is still a custom job.

The model that transfers is not “give everyone the tool and let them build their own context.” It is: the expert builds the context, the system holds it, the team uses it. The output becomes a function of the system, not the user.

This model also assumes the expert controls the system. In corporate environments, data sovereignty, compliance, and confidentiality introduce constraints that can change that assumption entirely. I have been operating without them, for now — and I remain aware that the model may look different under them.

What would make me wrong

I have not fully deployed this yet. It is the bet I am making.

What would make me wrong in twelve months? Two things.

First, if the team does not adopt it — not because the tool fails but because the behavior change does not take. Augmentation requires a feedback loop: the agent surfaces something, the human acts on it, the system updates. If that loop does not close — through lack of rigor, lack of trust in the output, insufficient adoption — the context decays and the system becomes noise.

Second, if the context stops accumulating. What I have today is a function of twenty-four months of continuous input. A system that stops learning — because it hits a limit, because the maintenance breaks, because the inputs slow down — will slowly become a static prompt. The gap it closes will reopen.

The variable that matters in AI-augmented teams is not the tool. It is not even the person. It is the context — how much of it exists, how well it is structured, how it was built, and who controls it.

For knowledge workers who are not developers, context does not build itself. Someone has to design the system that captures it. And that person does not have to be technical — but they have to be experienced enough that their judgment is worth encoding, and honest enough to admit that the tool amplifies whoever feeds it.

That is not the story most AI vendors are telling. But it is the one I am watching play out.