On this page
In August I described MBLegalOS, the persistent AI workspace my practice runs on. One part of that post was a promise: research goes into a knowledge base, standing positions live in a wiki, and every real matter that tests a position makes it sharper. Research in, position out, matter feeds back into the position. I called that loop the difference between a system that improves with use and one that just accumulates files.
Two months later, one Sunday showed me where the loop breaks. The system was very good at writing down what I learned. It was much worse at putting that knowledge in front of the next piece of work. This post is about that gap and what I built to close it.
One Sunday, four misses
I spent most of that Sunday on a long revision of a youth-sports platform client’s consumer terms and its customer terms. Over the course of the day, the same thing happened four times.
The wiki page on consumer arbitration already listed one open question for this client: whether to register the revised arbitration clause with the arbitration provider. The question came up anyway, from a different direction, and nothing pulled the page into the conversation. The client’s contract playbook (the document that holds its negotiating positions, fallbacks and walk-away points) was keyed to an older version of the terms, and it still described a liability-cap decision as “pending approval” after the terms carrying that cap had been published. A set of template enforcement letters still cited damages provisions the terms had deleted that morning. And my own grounding rule, the one every drafting session runs before it produces a review or a memo, said to “consult matching wiki pages.” Which pages match was left to the session’s judgment, and nothing checked the choice.
None of that knowledge was lost. Every piece of it was written down somewhere in the workspace. The problem was that finding it depended on someone (me, or the AI) remembering it existed at the right moment, and positions that changed in one place kept living on in copies somewhere else.
Capture was never the hard part
AI makes capture easy. At the end of every session, my system proposes what should be added to the wiki. A scheduled job files new articles and alerts into the knowledge base three times a day. A weekly review checks whether the capture steps fired. By that measure the system was healthy.
The measure was wrong. The knowledge that pays off is the knowledge that changes the next answer. Lawyers know this problem from the paper world: the precedent sits in the forms bank, and the person drafting late at night never opens it.
So the first thing I did was measure use. The weekly review now reads the log of which internal sources each deliverable relied on. In the week before I built anything, 9 of 17 deliverables cited a wiki page, a playbook or a knowledge-base source. That’s 53 percent. When the review sampled ten substantive sessions and asked what they should have read, it found four where existing knowledge went unused, eleven gaps in what I’d expect a lookup to return, and three sessions that never looked at all.
The design choices
I wrote a build spec before writing any code, and four choices in it do most of the work.
Look it up, don’t remember it. Deciding which pages apply to a task is now a lookup in a maintained table instead of a judgment call made fresh each session. The table (I call it the task map) lists each type of work I do, thirteen of them so far, from consumer terms revisions and liability caps to NDA review, vendor reviews and LLC governance. For each one it says which wiki pages, playbook sections, approved clauses, checklists and client-specific notes must be read before drafting, and which headings in each.
Deterministic before clever. The lookup is a small script that reads the table. You give it the request in plain words, it works out the type of work from trigger phrases, and it returns at most eight sources in priority order. There is no vector search or embedding model in it. I can read exactly why a source came back, and when the answer is wrong I fix a row in a table I can see. If the measured miss rate ever says a table can’t keep up, I’ll add semantic search then, against a baseline.
Measure use, not capture. Every piece of the build has a check that shows whether it changed an answer.
One index. The clause library, the dependency register and the tests all key off the same work-type IDs in the task map, so they can’t drift into separate lists that disagree.
What I built
The lookup. Before the lookup went into any workflow, I tested it on ten real requests from the previous month. For all ten, the source that actually controlled the answer came back in the top five. Three cases where I had missed a source in real life came back too.
A clause library. Sixteen entries, each copied verbatim from the published or executed document, with a note on when to use it, when not to, and the analysis behind it. Fourteen are approved language. Two are rejected, and I’ve found those as useful as the approved ones: one records a fixed per-incident damages clause we removed because that structure reads as a penalty under Florida law, and the other a one-way shortened limitations period, which Florida law voids for contract claims. A rejected entry stops a future draft from re-proposing something we already decided against.
Checklists. One for consumer arbitration clauses, 39 checks built from the AAA’s Consumer Due Process Protocol and its consumer rules, and one for publishing a revised legal document on a client’s website.
A dependency register. This one answers the stale-copy problem. Each settled position (a cap amount, a section number, a deleted remedy) is listed with every document that quotes it and the strings that would mean a document still carries the old value. A script greps the dependents and prints STALE or OK for each. It never edits anything; the output is a list for me. Its first run found ten stale copies across eight documents.
Verify-by dates. Some facts go stale on a date rather than on an event: a fee schedule, a vendor’s terms, a pending certiorari petition, a version number. Those facts now carry an inline marker with a re-check date and a source, and a script lists the overdue ones every week. Forty-three facts carry one so far.
Building the lookup turned up a bug I would never have gone looking for. My register of verified case citations had 67 verified rows, added since early August, sitting under the heading “Superseded.” The citation-verification instructions said to “append a row,” and appending to a file puts the row under whatever section comes last. To a human reader, every one of those cases looked dead. None was. The fix was easy: move the rows and tell the instructions where a new row goes. Finding it took a parser that cared where a row sat.
The wiring. All of that is inert until the drafting workflows call it. I rewrote the grounding rule every drafting skill already runs through so that it starts with the lookup, and added a lookup step at intake to fifteen drafting and review skills (contract redlines, NDA and indemnity reviews, vendor assessments, quick answers to a client’s business team, among others). The returned list is the floor: a session can read more but has to say why if it reads less. If a returned source is past its verify-by date, the session re-checks that fact at the source before relying on it. Proposed contract language starts from the library text, and a rejected entry is never proposed. And the deliverable names the internal sources it relied on, which is what lets the weekly review compare what the lookup returned against what the work used.
Testing whether any of it works
Two tests run on a schedule.
The weekly review now classifies every source the lookup returned but the work didn’t use. Each one is a gap in the map, a gap in the page, a lookup that never ran, or noise. Each class has a different fix, so the classification tells me what to change.
Once a month, a retrieval test asks twenty questions whose answers live inside the workspace: firm-specific positions, settled decisions, numbers you couldn’t guess from general legal knowledge. Each question goes to a fresh AI session with no context beyond the workspace. To pass, the answer has to cite the right internal file and state the key facts.
I ran the first set before the new workflow steps were installed. It scored 19 out of 20, and I don’t trust that number. The questions named the client, and fresh sessions found most answers in the matter notes as often as in the wiki. A test that passes before you’ve built the thing it measures isn’t measuring much. On its own, the lookup script missed the right source on 5 of the 20 questions, which tells me where the task map needs better trigger words. The next set will drop the client names, ask questions whose only answer lives in a concept page or an approved clause, and add strings that only a stale answer would contain.
The test paid for itself anyway. Three of the twenty answers flagged real problems in my own files: a contract template whose liability cap disagreed with the playbook, a vendor-scoring rubric whose weights added up to 105 percent in one configuration, and a wiki page still recommending a step I had decided to defer. A retrieval test turns out to double as an audit of the thing it retrieves from.
Why it’s guidance and not a rule yet
My workspace has a short list of hard gates, rules every session follows without exception, each written after a real mistake. The lookup is not one of them yet. It’s a required step in the skills, and the decision on making it a gate waits until two monthly test runs show it changes answers. A firm doesn’t write a procedure into its policy manual before it has watched the procedure work, and I’m holding my own system to the same standard.
What it comes down to
The point of a persistent workspace is that the practice gets smarter with use. That only happens if what the practice learned on Tuesday is in front of the draft on Friday, and if a position that changed on Sunday stops showing up in the old documents on Monday. Capture gets you a record. Use takes a lookup the drafting has to run, a register that knows where every position is quoted, and a test that tells you whether any of it changed an answer.
I’ll know whether this works in about a month, when the weekly numbers and the second test run come in. The baseline is written down, and so is what I expect to see.
MBLegalOS and the linked materials are independent projects of the firm. They are not affiliated with, or endorsed by, Anthropic or any client of the firm.