Contract Intelligence Platform

The brief

“Design a system that reads our contracts. We want to know what’s in them, where the risk is, and how a new one differs from our standard form.”

“In-house legal at a mid-size company. Two hundred incoming vendor agreements a month, four lawyers, everyone is drowning. Build them something.”

The second framing is the honest one, and it is the one to design for, because it names the constraint. The bottleneck is lawyer-hours. Lawyer-hours are expensive and scarce, and they cannot be replaced by a model that is right most of the time.

The product does four things. Extraction pulls the structured facts out into fields you can query, with every field pointing back to the exact text it came from. The facts are parties, term, renewal mechanics, governing law, liability cap, indemnities, termination rights, and payment terms. Risk identification compares what the contract says against what your organisation has decided it will accept, and flags the gaps. Comparison diffs this contract against your standard form or against a prior version, semantically rather than character-by-character. Summarisation produces a page a partner can read in two minutes instead of forty.

The thing that makes this domain different from every other RAG product is the accuracy bar. A support copilot that is wrong 5% of the time is a good support copilot. A contract system that misses one liability cap in twenty is a liability generator, and the person holding the liability is a licensed professional with a malpractice insurer.

What I’d ask first

“Is this pre-signature review or post-signature analysis?”

These are two different products that people describe with the same words.

Pre-signature is a workflow tool. A contract arrives, the system marks it up against the playbook, proposes fallback language, and a lawyer negotiates. Latency matters here, because a reviewer is waiting. The unit of value is time-to-turnaround.

Post-signature is a data product. You have forty thousand executed agreements in a shared drive, and someone in finance needs to know your total termination-for-convenience exposure before a restructuring. Latency does not matter, because you can batch for a week. Coverage and recall matter enormously, because a clause you missed is a clause that does not exist as far as the business is concerned.

I would build the extraction and provenance core once, and put two different surfaces on it. I would say that explicitly, because it is the architectural insight the question is testing.

“Are these your paper or theirs?”

This is enormously consequential, and it is often skipped. If you send your standard form and receive redlines, the problem is nearly tractable. You know the base document, so the task is classifying deviations from a known baseline. If you receive two hundred different third-party forms, every one written by someone else’s counsel with their own defined terms and their own structure, then you are doing open-ended extraction from arbitrary documents. That is an order of magnitude harder.

“Does a lawyer sign off on every output, or are some outputs consumed directly by the business?”

This is the copilot-versus-autopilot question wearing a suit, and it is where the liability lives. If a lawyer reviews everything, the system is a productivity tool, and its failures cost time. If a procurement manager reads the risk summary and signs, then the system is giving legal advice through a plausible-sounding intermediary. That creates a professional-responsibility problem, an unauthorised-practice-of-law question in some jurisdictions, and a contract with an unreviewed indemnity in it.

The answer I want is this. A lawyer signs anything that goes external or creates obligation. The business may consume read-only summaries of executed agreements, with a visible confidence and a link to source.

“Where do the documents live, and what are the confidentiality constraints?”

Contracts are among the most sensitive documents an organisation has. If this is a law firm, they are client confidences. ABA Formal Opinion 512 (July 29, 2024) addresses generative AI directly under the duties of competence, confidentiality, supervision, communication, and fees. It states that lawyers may need informed client consent before putting client information into a tool that could expose or learn from it (ABA). That immediately constrains vendor choice, data residency, retention, training-on-your-data terms, and cross-matter isolation. There is one more constraint. Some contracts are under NDA restricting who inside the company may read them, so the permission model is per-document and sometimes per-clause.

“What does the playbook look like today, and does it exist in writing?”

Usually it exists in one senior lawyer’s head and a Word document from 2019. If the risk rules are not written down, extracting them is the first project rather than a preliminary. It is also a genuinely valuable project, independent of any AI.

“What languages and jurisdictions?”

A liability cap in a German-law contract and a New York-law contract are not the same object. Neither are the enforceability assumptions your playbook encodes.

The answers I’ll design against. In-house legal, mostly third-party paper, with both pre-signature review and a back-catalogue of about 40,000 executed agreements. A lawyer stays in the loop for everything that creates obligation. Read-only summaries go to the business. Documents stay in the company’s cloud tenancy, with no training on their data and strict per-matter access control. The playbook exists partially, and it will be formalised as part of the project. English plus German and French, under US, UK, and EU governing law.

The design

  INTAKE                          PROCESSING                       KNOWLEDGE
  ------                          ----------                       ---------
  Email inbox   ─┐
  DMS / iManage ─┤   ┌────────────┐   ┌──────────────┐
  SharePoint    ─┼──►│ Ingest     │──►│ Layout-aware │
  Upload UI     ─┤   │ dedupe     │   │ parse: OCR,  │
  E-sign system ─┘   │ virus scan │   │ headings,    │
                     │ classify   │   │ numbering,   │
                     │ doc type   │   │ tables, defs │
                     └────────────┘   └──────┬───────┘
                                             │  every token keeps
                                             │  (page, bbox, char range)
                                             ▼
                                  ┌────────────────────────┐      ┌──────────────┐
                                  │ Structure model        │      │ Clause       │
                                  │ split into clauses on  │─────►│ library +    │
                                  │ the doc's own numbering│      │ embeddings   │
                                  └──────────┬─────────────┘      │ (your own    │
                                             │                    │  past deals) │
                    ┌────────────────────────┼─────────────┐      └──────┬───────┘
                    ▼                        ▼             ▼             │
            ┌───────────────┐      ┌──────────────┐  ┌───────────┐       │
            │ Clause        │      │ Field        │  │ Defined-  │       │
            │ classifier    │      │ extractor    │  │ term      │       │
            │ (which type)  │      │ (LLM, JSON,  │  │ resolver  │       │
            │ cheap encoder │      │  w/ spans)   │  │ (rules)   │       │
            └───────┬───────┘      └──────┬───────┘  └─────┬─────┘       │
                    └──────────┬──────────┴────────────────┘             │
                               ▼                                         │
                      ┌──────────────────┐                               │
                      │ Verifier         │  span must exist verbatim,    │
                      │ span-grounding   │  types must validate,         │
                      │ + type checks    │  cross-field consistency      │
                      └────────┬─────────┘                               │
                               ▼                                         │
                      ┌──────────────────┐   ┌──────────────────┐        │
                      │ Contract Record  │   │ Playbook engine  │◄───────┘
                      │ fields + spans   │──►│ rules + retrieved│
                      │ + confidence     │   │ precedent → risk │
                      └────────┬─────────┘   └────────┬─────────┘
                               │                      │
                               ▼                      ▼
                 ┌──────────────────────────────────────────────┐
                 │  REVIEW WORKBENCH                            │
                 │  document viewer with highlighted spans      │
                 │  risk list, severity, suggested fallback     │
                 │  accept / correct / reject  ── every action  │
                 │  is training data and an audit record        │
                 └──────────────────┬───────────────────────────┘
                                    ▼
                      [Audit log]  [Obligations DB]  [Redline export .docx]

Intake and parsing. More engineering goes here than anyone budgets. Contracts arrive as scanned PDFs of faxes, as Word documents with tracked changes and comments, as PDFs generated from Word with broken text ordering, as email attachments, and as amendments that only make sense alongside the master agreement they amend. You need OCR with a quality gate, layout-aware parsing that preserves the numbering hierarchy, and table extraction for schedules and pricing exhibits. You also need the piece people forget, which is a document family model. A Master Services Agreement, its four Statements of Work, two amendments, and an NDA are one commercial relationship. Answering “what is our liability cap” requires all of them.

Provenance from the first byte. Every extracted character keeps its origin: page number, bounding box, and character offset in the parsed text. This is not a nice-to-have. Provenance is the property that makes the whole product acceptable in a legal setting. It converts “the AI says the cap is 12 months of fees” into “here is the sentence, on page 14, that says so.” A lawyer can verify that in three seconds instead of thirty minutes. Design it in from the parser outward. Retrofitting span provenance onto a system that lost it during chunking is a rewrite.

Chunking, which is the interesting technical problem. Contracts are long. A hundred pages is unremarkable, and a credit agreement can be six hundred. They are also pathologically non-local. Section 11.4 caps liability “subject to Section 11.6,” which carves out breaches of Section 8, which incorporates a schedule by reference. A fixed-window chunker slices through that, and every downstream answer is wrong in a way that looks right.

So chunk on the document’s own structure, not on token counts. The numbering hierarchy is a gift, so use it. Each chunk carries its full section path, its heading, the defined terms it uses, and the cross-references it contains, resolved to targets where possible. Defined-term resolution is largely deterministic. Capitalised terms are defined in a definitions section or inline via quotation marks, and rules resolve them. Then inject them into the chunk’s context, so the extractor is not guessing what “Losses” means in this particular agreement.

Long-context models genuinely help here, and the practical shape has shifted. You can now put a whole hundred-page agreement into a single call. However, do not treat that as a licence to stop retrieving. Cost scales with input. Recall degrades in the middle of very long inputs for many models. And, decisively for this domain, you still need to know which span an answer came from, which a whole-document call does not give you for free. Here is the pattern that works. Retrieve the candidate sections for a given field. Then pass those sections plus their resolved cross-references in a focused call, and require the model to quote the exact supporting text.

Extraction as structured output. Every field is defined in a schema with a type, an enum where applicable, and a required evidence object containing the verbatim quote and its span. The model fills the schema. Then a verifier runs in code. Does the quoted text appear verbatim in the source document? Does the date parse? Is the currency one we recognise? Is the cap amount consistent with the cap type? Does the term end date follow the start date? A field whose evidence span does not exist in the document is discarded, not surfaced. That single check kills the most dangerous failure mode in the product, which is a fabricated quotation, for the price of a string search.

The playbook engine. The playbook is a configurable rule set, versioned, and owned by legal. Each rule says that for clause type X, the acceptable position is Y, the fallback is Z, and anything else escalates to the GC. Some rules are deterministic once extraction is done, such as liability_cap_multiple < 1.0 → flag. Some need judgement, such as whether this indemnity, read with the carve-outs, actually leaves us exposed. So route the deterministic ones through code. Route the judgement ones through a model, and give that model the extracted clause, the playbook position, and the two or three most similar clauses from your own past deals with how they were resolved. That retrieval over your own precedent is the most under-appreciated component in the system. It turns institutional memory into a feature, and it is the thing an off-the-shelf vendor cannot give you.

Comparison and redlining. Use semantic clause alignment, not character diff. Align clauses between the two documents by type and content. Then classify each aligned pair as identical, cosmetic, or substantive. Then characterise the substantive ones by direction: more favourable to us, less favourable, or ambiguous. Direction is what a reviewer actually wants, and a plain diff never gives them that. Output must round-trip to a .docx with real tracked changes, because that is the artifact the other side’s counsel expects. A system that cannot produce it will not be used, regardless of quality.

The workbench. The document sits on the left, and the extracted fields and risks sit on the right. Clicking a field scrolls to and highlights the source span. Every item has accept, correct, or reject, and corrections are captured as structured data. This is the review UI, and it is most of the product’s perceived quality.

Where the AI actually is

Genuinely needs a model: classifying a clause into a type when it has an unusual heading and unusual wording; extracting a field whose expression varies infinitely; judging whether a deviation from the playbook is material; drafting proposed fallback language; summarising; and semantic alignment for comparison. On the extraction point, “shall not exceed the aggregate fees paid in the twelve months preceding the event giving rise to the claim” is a liability cap, and no regex finds it.

Ordinary engineering, and it is most of it: intake connectors, OCR and its quality gate, layout parsing and the span-preservation plumbing, the document family model, and defined-term resolution. Then the schema registry and validators, the verifier, and the playbook rules engine with its versioning. Then the access-control model, which in a law firm means matter-level walls and ethical screens, plus encryption, retention, and deletion. Then the .docx round-trip, which is genuinely fiddly, the audit log, the workbench UI, and the obligations database with its calendar reminders.

The ratio holds here as firmly as anywhere. The model does the reading. Everything that makes the reading trustworthy, permitted, auditable, and usable is software.

What I would deliberately not use an LLM for:

Verifying that a quote exists. Use string matching. It is deterministic and instant, and the model cannot talk itself out of it.

Defined-term resolution. This is overwhelmingly a parsing problem with a rules solution, and rules do not invent a definition that is not there.

Date and money arithmetic. Extract the components with the model, and compute with code. Models do arithmetic adequately and unpredictably, and “adequately” is not a word you want near a payment schedule.

Deciding who may see a document. Access control is a database query executed before any model sees anything.

Detecting exact-duplicate or near-duplicate documents. Use hashing and shingling.

Producing the final legal conclusion. The system flags, characterises, and evidences. A lawyer decides. That is not a limitation of current models. It is the professional-responsibility structure of the domain, and designing as though it will change is how you build something no legal department will buy.

Key decisions and tradeoffs

The fork worth arguing is recall versus precision on risk flags, because it determines whether the product is used.

The case for recall: a missed uncapped indemnity is the failure that ends careers, and a false positive costs a lawyer fifteen seconds to dismiss. The case for precision: a review that surfaces sixty flags on a routine NDA trains the reviewer to dismiss without reading. At that point your recall is theoretically high and practically zero. That second dynamic is real, and it is how these products die.

The resolution is not a single threshold. The resolution is tiering. Have a small number of high-severity flags, tuned for recall and never suppressed. Have a larger set of medium flags, tuned for precision, and collapsible. Have informational extractions that populate fields without demanding attention. Then measure dismissal rate per flag type and retire the ones nobody ever acts on. A flag that is dismissed 98% of the time is training your users to ignore the system.

What breaks

Cross-references and carve-outs, which is the domain-specific failure a generalist misses. “Notwithstanding Section 11.4, the limitations in this Article shall not apply to breaches of Section 8 (Confidentiality) or to a party’s indemnification obligations under Section 9.” The liability cap you extracted from 11.4 is real. It is also substantially hollowed out by a sentence forty pages away. A system that reports “cap: 12 months fees” without the carve-outs has told a lawyer something false while quoting accurately. There are three mitigations. Model carve-outs as a first-class part of the extracted object, rather than as prose. Follow cross-references during chunk assembly. And treat any cap-type field extracted without a carve-out search as low confidence by construction.

Amendments and the document family. The master agreement says the term is three years. Amendment No. 2, executed eighteen months later, extended it and changed the notice period. Analysing the master alone gives a confidently wrong answer. The mitigation is family assembly at ingestion. Link amendments, SOWs, order forms, and side letters to their parent. Then compute an effective contract state as of a date, with each effective field pointing at whichever document actually governs it. This is a hard data problem, and it is not an AI problem.

Definitions that redefine ordinary words. A contract can define “Affiliate,” “Confidential Information,” or even “Material” in ways that invert the plain meaning. A model reasoning from general knowledge rather than the document’s own definitions gets this wrong, and it is very hard to spot, because the output reads perfectly. The mitigation is to resolve defined terms explicitly and inject them, then flag any clause whose interpretation depends on a defined term that could not be resolved.

Scanned documents and OCR. A 1998 agreement scanned crooked from a fax will produce garbage, and the pipeline will process the garbage without complaint. An OCR confidence gate that routes bad scans to human transcription is cheap, and it prevents an entire class of silent failure. Signature pages, handwritten marginalia, and initialled changes are a special case. A handwritten “20” struck through and replaced with “30” in the margin is legally operative, and OCR will miss it entirely.

Fabricated quotations. The nightmare output is a plausible clause quotation that is not in the document. The legal profession has already lived through the general version of this. In Mata v. Avianca (S.D.N.Y. 2023) the court sanctioned counsel who filed a brief containing citations to cases that did not exist. A rigorous study of purpose-built legal research tools also found meaningful hallucination rates even with retrieval, at 17% and 33% for two leading commercial products, under a definition covering incorrect or misgrounded responses (Magesh et al., JELS 2025). Retrieval reduces this. It does not eliminate it. The mitigation is the verifier: no span, no output. It is the single most important hundred lines of code in the system.

Confidentiality bleed across matters. Retrieval over “your own past deals” is a superb feature and a compliance hazard. If a clause from Client A’s confidential agreement is retrieved as precedent while working on Client B’s matter, you have a problem that ends engagements. The mitigation is retrieval scoped by matter and by an ethical-wall model, enforced in the query. Add a separate, deliberately curated, de-identified precedent bank for cross-matter use where the confidentiality position permits it. Never use a single undifferentiated index.

The reviewer stops reviewing. This is the same automation-complacency failure as the support copilot, with much worse consequences. The mitigation is to measure time-on-document and per-flag dwell time, keep flag volume low enough that reading them is realistic, and randomly seed a small number of known-issue documents to check that reviewers are catching them.

Silent degradation on a new counterparty template. A large vendor changes its form, and suddenly a whole cohort of contracts extracts badly. The system reports nothing, because it does not know it is wrong. The mitigation is to monitor extraction coverage, meaning the rate at which each expected field is found, segmented by counterparty and document template. A field-found rate that drops from 94% to 60% for one counterparty is an alarm, even when you have no ground truth.

How you’d evaluate it

Offline. Build a gold set of a few hundred contracts annotated by lawyers, at the span level. Stratify by document type, counterparty, jurisdiction, length, and scan quality. This is expensive, and it is the single most valuable asset the project will produce. Budget real money and real lawyer time for it, and treat it as infrastructure.

Score extraction at the field level with precision and recall. Because provenance is the point, also use a span-overlap criterion, rather than just string equality of the value. A right answer with a wrong citation is a failure. A reviewer who clicks through and lands in the wrong place stops trusting every citation. Report per field type. Aggregate F1 hides that you are excellent at governing law and mediocre at indemnity scope, and indemnity scope is what matters.

Risk flagging gets precision and recall against lawyer judgement, evaluated per severity tier, with recall on high-severity flags treated as a gate rather than a metric. Summarisation gets a rubric judge for factual support against the source, plus periodic human review. The rubric’s most important criterion is not fluency. It is whether every assertion is traceable.

Keep an adversarial slice. It should contain contracts with unusual structures, carve-outs that reverse a cap, amendments that change a field, defined terms that invert meaning, and documents where the correct answer to a field is “not present.” A system that never says “not present” is a system that hallucinates on absence.

Online. Time-per-review is the headline productivity metric, measured against a matched baseline. It must be paired with a quality measure, or it is meaningless. Correction rate per field type, from the workbench, is the continuously available quality signal. It needs no annotation, because reviewers generate it as a byproduct of working. Flag dismissal rate per rule tells you which rules are noise. Escalation-to-GC rate tells you whether the playbook thresholds are set sensibly.

The metric that actually matters to the business depends on which product you built. For pre-signature, it is contract cycle time, meaning days from receipt to signature, at constant or improved risk outcomes. Cycle time is revenue for the sales organisation, and legal is measured on it. For post-signature, it is the ability to answer a portfolio question that was previously unanswerable at any price, such as total exposure under a clause type before an M&A event. Never lead with F1 in front of a general counsel.

Catching regressions. Run the full gold set in CI on every model, prompt, parser, or chunker change. The parser is the sneaky one, because an upgraded PDF library can shift spans and silently break provenance everywhere. Use per-field gates rather than just an aggregate, with high-severity risk recall as a hard blocker. Shadow-run the new version against the old on live traffic and diff the outputs. Disagreements are the cheapest possible source of new gold-set candidates. Keep permanent human spot-review of a weekly sample.

The sibling agentic-ai-evaluation-guide covers rubric construction, judge calibration, and building annotation sets with real inter-annotator agreement, in the depth this deserves. Inter-annotator agreement is unusually important here, because two experienced lawyers will genuinely disagree about whether a clause is risky. A gold set that pretends otherwise will cap your measurable accuracy below your actual accuracy.

Follow-ups they will ask

“A hundred-page contract does not fit your prompt. What do you actually do?” I do not treat it as one blob, because the answer to any given question lives in one or two sections plus whatever they cross-reference. Parse into the document’s own hierarchy and build a section index with headings and defined terms. Then, for each field in the schema, retrieve the candidate sections. Retrieve by clause classifier, by heading match, and by embedding similarity to a canonical description of that clause type. Then assemble a focused context of those sections, plus their resolved cross-references, plus the definitions they depend on. That context is a few thousand tokens for most fields, which makes it cheap, fast, verifiable, and parallelisable across fields. Long-context models make the fallback path easy for the awkward cases, and I would use them for global questions like “summarise the commercial deal.” However, I would not let a long context window become an excuse to skip structure, because structure is what gives me the span provenance the product is built on.

“How do you guarantee the system doesn’t invent a clause?” I do not guarantee it at the model layer, because I cannot. I guarantee it at the verification layer. Every extracted value must arrive with a verbatim quotation, and code checks that the quotation appears in the parsed document. That is an exact match after whitespace normalisation, with the character offsets recorded. If it does not appear, the extraction is dropped, and the field is reported as not found rather than surfaced with a warning. A warning is something a busy person clicks past. That converts an unbounded hallucination risk into a bounded recall cost, which is the right trade in a legal setting. On top of that, the UI never displays a claim without its clickable span, so the reviewer’s default action is verification rather than trust.

“Where exactly is the lawyer in the loop, and why can’t you remove them?” The lawyer decides anything that creates or waives obligation. The system extracts, evidences, compares, and proposes. The lawyer accepts. There are two reasons I would not design them out, and only one of them is about model quality. The professional reason is that legal advice carries accountability, and accountability requires a person and a licence. ABA Formal Opinion 512 puts generative AI squarely inside the existing duties of competence, confidentiality, and supervision, rather than creating an exemption from them. A lawyer remains responsible for work product regardless of what produced it. The commercial reason is that the customer is buying risk transfer as much as speed. A system that removes the accountable human removes the thing they are actually paying for. What I would remove the lawyer from is the mechanical part: finding the clause, retyping it into a spreadsheet, and checking the term dates. That is where the hours are.

“How do you compare two contracts properly?” Align, then classify, then characterise. Alignment is clause-to-clause by type and content similarity, not a text diff. The same obligation may sit in Section 7 of one document and Section 11.2 of the other, with entirely different wording. Then classify each aligned pair as identical, cosmetic, or substantive. Cosmetic changes are the majority, and hiding them is most of the value. Then characterise the substantive ones by direction and magnitude relative to your position: more favourable, less favourable, or ambiguous. Name the specific delta, such as “cap drops from 12 months’ fees to 3 months’ fees.” Unaligned clauses on either side are their own category, and they are often the most important thing in the review. A clause that was in your form and is absent from theirs is easy to miss, and it can be the entire risk. Output round-trips to tracked changes in Word.

“How do you handle a playbook that differs by business unit, region, and deal size?” By making the playbook data, not prompt. Rules are versioned objects with a scope predicate covering jurisdiction, entity, contract value band, counterparty tier, and product line. They have an evaluation order with explicit precedence, so a rule can be overridden rather than duplicated. Legal edits them in a UI, and sees which contracts a change would have affected before publishing. Every fired rule is recorded by version on the contract record, so an audit two years later can reconstruct exactly what standard was applied. The thing I would resist is letting the playbook live in a prompt. Then nobody but an engineer can change it, nobody can diff it, and you cannot answer “why was this flagged” without re-running a model.

“The model flags sixty issues on a routine NDA and nobody reads them. Fix it.” That is a product failure, not a model failure. I would fix it with tiering and measurement rather than a better prompt. Use severity tiers: a handful of must-read flags with high recall, a collapsible medium tier tuned for precision, and silent field extraction for everything else. Then measure per-rule dismissal rate, and retire or demote any rule dismissed above some threshold. A rule dismissed 98% of the time is worse than no rule, because it costs attention and teaches dismissal. Contract-type-aware playbooks help enormously. An NDA should have eight applicable rules, not a hundred and forty. Applying the master-services playbook to an NDA is the actual bug in most systems that behave this way. And a “known counterparty, standard form, no deviations from last time” fast path, which says exactly that in one line, is often the highest-value output the system produces.

“Client documents are confidential. How does that constrain the architecture?” It constrains the vendor contract before it constrains the code. You need zero data retention, no training on inputs, defined data residency, and a subprocessor list I can show a client. Those are procurement requirements, and I would treat a vendor without them as unusable, regardless of benchmark scores. Then, in the system, you need per-matter access control enforced at the query layer, ethical walls modelled as first-class objects, and encryption at rest with per-tenant keys. You also need retention and deletion policies that can actually execute deletion, including from indexes and caches, plus audit logging of every document access by every user and every automated process. The subtle one is the vector index. Embeddings are derived from confidential text, so they must inherit its access controls. A shared index across matters is a confidentiality breach waiting for a retrieval query to find it. If the client’s position requires it, the fallback is a model deployed inside our own tenancy. That means worse quality and more review time, and it is a trade the client is making knowingly.

“How do you evaluate this when two lawyers disagree about what’s risky?” I measure the disagreement instead of hiding it. Use multiple annotators on an overlapping subset, and report inter-annotator agreement per field type. Accept that the human ceiling on subjective fields is well below 100%. Then split the eval. Objective fields get hard precision and recall against a single correct answer, and those should be near-perfect. Objective fields are governing law, notice period, cap amount, and term dates. Judgement fields get agreement-with-a-panel as the metric. The target is to match the human agreement rate, not to exceed it, because exceeding it is usually a sign your gold set is one person’s opinion. Reporting these separately is also politically useful. It lets you show a general counsel a very high number on the things that are objectively checkable, which is what builds trust in the rest.

“What about contracts in German or French?” There are three separate concerns, and I would not let them blur. Extraction quality needs its own gold set and its own per-language metrics, because a system that quietly performs worse in French will not announce it. Legal substance is different too. A limitation of liability clause under German law is subject to different mandatory rules than under New York law, so the playbook is jurisdiction-specific and the risk logic cannot simply be translated. This is legal work rather than engineering work, and it is the long pole. Provenance is the third. The quoted span must be in the original language even if the summary is in English, because the reviewer verifying it is reading the original document. I would launch one non-English jurisdiction at a time, with local counsel writing the playbook, rather than switching on five languages and discovering the failures in production.

“How would you handle the back catalogue of forty thousand executed agreements?” As a batch data project with completely different economics from the live path. No latency constraint means I can be lavish: multi-pass extraction, cross-checking, higher-capability models, and human review targeted by confidence. I would process in priority order driven by business value, rather than chronologically, so value arrives in week two instead of month six. Priority means highest-spend counterparties, contracts with renewal dates in the next twelve months, and anything the current question is about. Extraction coverage per field is my quality signal in the absence of ground truth. I would also sample-audit a few hundred documents by hand to estimate real accuracy, and publish that estimate with the dataset. Crucially, the output is a database with confidence scores and spans, not a set of assertions. When someone queries total termination-for-convenience exposure, they get a number, a confidence, and a list of the documents where extraction was uncertain and a human should look.

“What’s the first thing you’d ship?” Extraction of ten objective fields, with spans, over one contract type, into a searchable table, with the review workbench. No risk scoring, no redlining, no summaries. It is unambiguously useful, because those ten fields are what people currently retype into spreadsheets. It is verifiable in seconds per field. And it forces you to build the parser, the provenance plumbing, the schema registry, the verifier, and the workbench, which is the whole hard substrate that everything else sits on. If the spans are trustworthy, you have earned the right to ship judgement features. If they are not, no amount of risk-scoring sophistication was ever going to save you.

“How much of this project is AI?” Small, and unusually so even for this genre. The extraction prompts and schemas are a couple of weeks. The rest of the year is the PDF and Word parsing with preserved spans, the OCR gate, the document family model, the access control and ethical walls, the .docx tracked-changes round-trip, the playbook engine, the audit log, and the workbench. The .docx round-trip alone routinely surprises teams. Producing tracked changes that Word opens cleanly, and that opposing counsel can accept individually, is a genuinely fiddly piece of software with no AI in it whatsoever. If you cannot do it, lawyers will not use your product.

Say it in one breath

Contract intelligence is span-grounded structured extraction over long, cross-referential documents. You parse to the contract’s own numbering hierarchy, retrieve the candidate sections plus their cross-references and defined terms, and extract into a strict schema where every field carries a verbatim quotation. Then you verify in code that the quotation actually exists, because the rule is no span, no output. Risk lives in a versioned playbook that legal owns as data, evaluated against your own precedent, and tiered so that reviewers still read the flags that matter. The lawyer stays in the loop because accountability requires a licence, not because the model is not good enough. And the year of work is parsers, permissions, tracked changes, and the review workbench, rather than prompts.