A US-based mid-market law firm with four practice groups needed an AI legal assistant that could do first-pass MSA review at partner-trust quality: split a contract into clauses, retrieve matching policies and precedents from a reconciled clause library, flag clause risk against the firm's playbook with cited policy IDs, and refuse out loud on novel patterns. We built the AI contract review software on Claude Sonnet 4.6, LangChain 0.3, LangGraph 0.2, a hybrid pgvector and BM25 index, and a forced-JSON clause-risk schema. AI-powered contract review the general counsel could defend on an ABA Op. 512 audit. Nine weeks, partner-shadow-first, with a clause-library drift kill point at week 4 that we paused the build for.
A US mid-market law firm shipped a first-pass MSA review agent on Claude Sonnet 4.6 plus a LangChain orchestrator over a reconciled clause library of 1,420 master-service-agreement clauses. Across n=180 MSAs (95% CI), partner time per first-pass review dropped 71% with partner sign-off on every output. Stack: Claude Sonnet 4.6, LangChain 0.3, LangGraph 0.2, pgvector 0.7, BAAI bge-reranker-large, Langfuse, iManage Work. Compliance: attorney-client privilege, ABA Model Rules of Professional Conduct, ABA Opinion 512 on UPL, FRE 502. Multi-month ongoing engagement. The clause-extraction layer underneath is the same engagement shape we ship as a standalone document processing pipeline when the extraction itself is the primary surface.
A US-based mid-market law firm — ~70 attorneys, 4 practice groups (M&A, employment, real estate, IP), ~180 MSAs a year. Too small to fund Ironclad/Spellbook seats firm-wide; too large for the managing partner to hand-review every contract. Partners spent 6–9 hours per MSA on first-pass review. The pre-build process was partner-by-partner, playbook-by-playbook, and the playbooks had quietly drifted apart.
Two failure modes. Wall-clock: partners burning 6–9 hours per MSA on first-pass review, cross-checking each clause against their practice-group playbook by hand. Quality cost of the drift: a general counsel audit traced 11% of post-execution disputes back to inconsistent first-pass clause calls — same fact pattern, opposite playbook calls between practice groups. Ironclad, Spellbook, and three smaller LegalTech vendors had all been evaluated and turned down. The conversation we walked into was not "should we ship LangChain" — it was "show us how a clause-risk agent could miss a hard-no, and tell us how you'd catch it before a partner sends the redline to opposing counsel."
Vendor turn-downs detailed below. The four objections decided the engagement shape.
iManage matter-scoped pull, semantic clause splitter with §N.M boundaries, two parallel retrieval lanes (clause-RAG over the reconciled 1,420-clause library + practice-group-aware policy lookup with regex-validated IDs), bge-reranker-large (A/B winner against Cohere Rerank v3), Claude Sonnet 4.6 with forced-JSON ClauseRisk schema. Zero write tools — the agent produces redlines; partners send them. Diagram below.
When something regresses, the per-component metric tells us which stage broke. No single end-to-end number that hides which subsystem moved.
Labelled risk-band correctness + policy-citation accuracy on the frozen 740-item eval. Forced-JSON schema enforces a regex-validated policy_id on every rationale claim — the model cannot suggest a redline without naming the policy it traces to.
Recall@5 on the reconciled clause library. RRF fusion + bge-reranker-large tuned against this number, not end-to-end accuracy.
Citation-accuracy against senior-counsel ground truth. Practice-group-aware: IP clauses retrieve IP policies first.
bge-reranker-large self-host vs Cohere Rerank v3.
Refusal rate by design, not a failure mode.
Every disagreement opened in the weekly override-review meeting.
Every clause runs the same six-stage chain. Two retrieval lanes fan out at stage 4 — house policy and the reconciled clause library — then converge on a forced-JSON clause-risk model. The four risk bands at the bottom are the only legal outputs. Hover any stage for its tool inventory.
stage latencies above are per-clause p50 / p95 · a typical MSA runs ≈ 80–140 clauses in parallel batches of 12 · end-to-end MSA wall-clock 38–62s
Everything in the build is a thing your IT director can write a question about. Nothing in the build is `our proprietary AI`. Vendor swap-out cost is bounded because the eval set, prompts, schemas, and policies are all checked into the firm's repo. Cohere Rerank stays wired as a fallback so the rerank stage has a documented swap-out path.
Latency is measured at the agent boundary; cost math uses Anthropic's published Sonnet 4.6 pricing as of May 2026; eval composition is the frozen 740-item clause-eval set the CI gates on.
| stage | p50 | p95 | tooling |
|---|---|---|---|
| MSA intake + iManage pull | 280 | 720 | iManage Work · matter-scoped OAuth · document hashed + version-pinned |
| Semantic clause split | 410 | 880 | Heading-aware splitter · §N.M boundary + sub-clause merge |
| LangChain orchestrator step | 36 | 88 | LangGraph 0.2 chain · shared scratchpad · clause-by-clause |
| Clause-RAG retrieval + rerank | 372 | 620 | pgvector + tsvector RRF k=60 → bge-reranker-large top-6 |
| Policy lookup (parallel) | 22 | 58 | Practice-group-aware regex-validated policy index |
| Sonnet 4.6 clause-risk decision | 1600 | 2400 | Anthropic API · response_format json_schema · ~2,800 in / ~520 out tokens |
| Validator + audit log | 18 | 32 | Zod schema · policy_id regex check · Langfuse trace write |
| Total (per-clause end-to-end) | 2738 | 4798 | agent boundary; MSAs batch 12 clauses in parallel for ≈ 62s wall-clock |
p50/p95 from a 6-month rolling window over n ≈ 18,400 per-clause decisions (180 MSAs × ≈ 102 clauses average). SLO is p95 ≤ 90s wall-clock per MSA; current burn ≈ 69%.
| model | recall@5 | price tier | pick reason |
|---|---|---|---|
| voyage-3-large (1,024d) | 0.92 | $$ | Pareto-best on recall, matched price tier — SHIPPED |
| OpenAI text-embedding-3-large | 0.89 | $$ | Lost 3pts recall@5 on the legal corpus |
| Cohere embed-multilingual-v3 | 0.87 | $$ | Lost on legal-domain recall, tied on price |
| bge-large (self-hosted, fine-tuned) | 0.90 | $ (infra only) | Close on recall, lost on 24/7 GPU operational cost |
Bake-off ran on a 60-clause held-out slice from the 740-item eval set. Same retrieval pipeline (pgvector + BM25 + RRF k=60) for every variant. We chose voyage-3-large after the bake-off; Cohere stayed wired as a runtime fallback.
// contracts/schema/clause-risk.ts
// Forced-JSON clause-risk schema. Validated on every model output;
// if Sonnet produces something that doesn't parse, we retry once
// with a stricter system prompt, then fail closed (manual-review).
import { z } from "zod";
export const ClauseRisk = z.object({
clause_id: z.string(),
risk: z.enum(["acceptable", "negotiate", "block", "manual-review"]),
rationale: z.array(z.object({
claim: z.string().min(20).max(420),
policy_citation: z.string().regex(/^policy_[A-Z]{2,4}-\d{3,5}$/),
precedent_ids: z.array(z.string()).min(0).max(8),
})).min(1),
suggested_redline: z.string().optional(),
});
export type ClauseRisk = z.infer<typeof ClauseRisk>;
Six clauses from a sample MSA, rendered with the same risk-band tinting and expandable rationale that partners see in the firm's tooling. Click any clause to expand the policy citation, precedent ids, and (where present) the agent's suggested redline.
Auto-renewal with 60-day opt-out notice matches the firm's standard MSA playbook for the vendor practice. No deviation flagged.
Counterparty's draft caps IP-infringement indemnity at 1× annual fees and excludes injunctive relief. House playbook requires uncapped IP-infringement indemnity plus injunctive coverage for the licensed product.
Mutual consequential-damages waiver is acceptable as a category; however, this draft also waives consequential damages for breach of confidentiality and data-protection obligations. Both of these the firm's policy carves out from any waiver. Partner notification required before send.
Delaware governing law + ICC international arbitration seat in Singapore matches the firm's cross-border SaaS playbook. No deviation flagged.
Clause contains a sub-paragraph permitting vendor to use customer data for `model improvement` outside the named regions. This is a novel clause shape not present in any of the 1,420 reconciled-library reference clauses; agent refuses to issue a band and routes to manual review.
Draft permits assignment to any affiliate without consent. House playbook requires written consent for any assignment outside the same parent group, plus a 30-day cure period for change-of-control events.
sample only — anchor_ids, policy_ids, and precedent_ids are illustrative and follow the shape the schema enforces · live surface renders all clauses, not just the 6 shown
| line item | $ / MSA | $ / month (≈ 30 MSAs) | note |
|---|---|---|---|
| Claude Sonnet 4.6 input tokens | $0.857 | $25.71 | 102 clauses × 2,800 tokens × $3.00 / 1M |
| Claude Sonnet 4.6 output tokens | $0.795 | $23.86 | 102 clauses × 520 tokens × $15.00 / 1M |
| voyage-3-large embeddings (per clause) | $0.037 | $1.10 | 102 clauses × ≈ 3,000 tokens × $0.12 / 1M |
| pgvector + RDS db.m6i.large | fixed | $284 | Postgres 16 in firm tenant · clause library + policy index |
| g5.xlarge reranker (24/7) | fixed | $378 | BAAI bge-reranker-large self-host · Cohere fallback wired |
| LangChain · LangGraph runtime | fixed | $94 | Python on Fargate · 2 vCPU · per-clause parallelism = 12 |
| Langfuse self-hosted (t3.medium) | fixed | $67 | trace store · 90-day hot / 7-yr cold |
| iManage Work connector | fixed | $0 | uses firm's existing iManage Cloud seat |
| All-in monthly (≈ 30 MSAs) | ≈ $1.69 | ≈ $874 | vs. ≈ 200 partner hours saved at firm rates |
Token costs use Anthropic's published May-2026 Sonnet 4.6 pricing ($3 / 1M input, $15 / 1M output). Infra costs are AWS US-east-2 list price (firm's tenant). Per-MSA token cost assumes the median 102-clause MSA observed in the eval set; range across the 180 production MSAs in the 6-month sample is $0.94 (62 clauses) to $2.83 (164 clauses). Payback period from go-live, including the 9-week build at $215k, is ≈ 4.4 months at the firm's published blended partner rate against the ≈ 71% time saved on partner-signed-off MSAs.
| category | items | what it checks | ci-gate threshold |
|---|---|---|---|
| M&A clauses (golds) | 320 | labelled risk band + policy_id + suggested redline · senior-counsel signed | ≥ 0.88 band precision · ≥ 0.90 policy accuracy |
| Employment clauses (golds) | 180 | labelled risk band + policy_id · employment senior-counsel signed | ≥ 0.88 band precision |
| Real estate clauses (golds) | 140 | labelled risk band + policy_id · real estate senior-counsel signed | ≥ 0.88 band precision |
| IP clauses (golds) | 100 | labelled risk band + policy_id · IP senior-counsel signed | ≥ 0.88 band precision |
| Block-clause must-catch | see note | subset across all 4 groups · catch every hard-no clause (must) | ≥ 0.95 block recall |
| Manual-review (novel patterns) | see note | deliberately novel clauses · agent must refuse, not guess | 100% refusal on listed must-refuse |
Eval set is frozen (items only added, never edited). Senior counsel from the relevant practice group signs off any addition. CI fails the release if any category drops more than 1 point from the prior cut; release engineer can over-ride with a signed CHANGELOG entry. Block-catch and manual-review subsets are sub-categories overlapping the 740-item count.
| cadence | what runs | who owns | compliance fit |
|---|---|---|---|
| Weekly | Override-review meeting — every clause the partner overrode the agent on | Senior counsel (one per practice group) + on-call engineer | Patterns showing 3× become eval-set additions |
| Monthly | Audit-log sample — model version, retrieved candidates from both lanes, reranker scores, policy_id, partner override | Firm's IT director | ABA Op. 512 reasonableness · FRE 502(b) inadvertent-disclosure |
| Per release | Full 740-item clause-eval re-run against the frozen set | Engineering on-call | CI fails the release if any metric drops > 1pt |
| Retention | Langfuse traces — 90-day hot in firm tenant + 7-year cold in tenant-scoped S3 | Firm IT (we do not hold the keys) | Matches the firm's privilege retention policy |
None of this is published anywhere else by anyone shipping legal agents. That's the bar.
Five stages, milestone-billed. The week-4 build turned up a clause-library drift problem: the M&A and real estate playbooks contradicted each other on the same fact pattern, and an AI contract review software trained on the drift would inherit it. We halted the build for a 2-week reconciliation pass with senior counsel from each practice group, then resumed. The honest version of 9 weeks is 7 weeks of build plus 2 weeks of pause.
Two weeks shadowing partners across the four practice groups. The managing partner of each group sat in the design council. We sampled 60 MSAs from the prior 18 months, anonymized them, and the four senior partners labelled each clause with the correct risk band + policy citation + suggested redline. That sample became the frozen 740-item clause-eval set: 320 from M&A, 180 from employment, 140 from real estate, and 100 from IP.
Ingested each practice group's existing clause library (1,840 reference clauses across the four groups) into pgvector 0.7 with embedding via voyage-3-large at 1,024 dimensions. Built the Postgres tsvector BM25 sidecar over the same corpus. RRF fusion tuned on a held-out eval slice; cross-encoder rerank A/B-tested between bge-reranker-large and Cohere Rerank v3. bge won on the legal corpus by ≈ 3 points top-1 precision; Cohere stayed wired as a fallback.
Building the per-clause review chain in LangGraph turned up a structural problem: M&A's standard indemnification language and real estate's contradicted each other on the same fact pattern (joint-and-several vs several-only for sub-tenancy indemnities). Employment and IP had two more such contradictions. An agent trained on the drift would inherit it. We halted the build, convened a 2-week reconciliation pass with senior counsel from each practice group, and produced a single reconciled clause library: 1,420 clauses after dedupe and reconciliation, down from 1,840. Cost two weeks of wall-clock; bought eighteen months of build-on-firm-ground.
LangChain 0.3 orchestrator wraps the LangGraph clause-by-clause chain. Claude Sonnet 4.6 with `response_format: json_schema` set to the ClauseRisk shape. The schema is the contract: every flagged clause has to name a policy_id matching the regex; precedent_ids are bounded 0–8; suggested_redline is optional and only renders if the partner expands the clause card. Confidence < 0.8 routes to manual-review; the agent never produces an autonomous redline.
Promoted to first-pass review with partners running in parallel for the first 6 weeks (every MSA reviewed both by the agent and by the partner; outputs compared in Langfuse). After week 6 of shadow the metric held: partner-override rate fell to 9.2% from a baseline 14% in week 1, and the firm cut over to agent-first first-pass with partner final-pass. The override-review meeting runs weekly with senior counsel from each group; patterns that show up three times become eval-set additions.
The AI contract review eval set is frozen. Every model change, prompt change, retrieval change, and policy change re-runs the full 740. Nothing ships if any metric red-lights against its target. Numbers below are from the current production cut and the frozen eval slice; live partner-shadow numbers are within ±1.5 points across all rows over the last 30 days.
Sample size for the headline time-saved number (≈ 71% first-pass MSA review time saved) is n=180 partner-signed-off MSAs across a 6-month rolling window; the figure is a 95% confidence interval, not a point estimate. Partner-override rate is the share of clauses where the partner overrode the agent's risk band on the live shadow slice (by design, not by failure). Manual-review refusal rate is the share of clauses the agent legally refuses to band (novel patterns, score-margin failures, off-corpus clauses) and routes straight to a partner — also by design. Latency is end-to-end MSA wall-clock from upload to full clause-risk report, measured at the agent boundary.
An AI contract review agent built on these patterns will mislead partners or counterparties in any of the following situations. We turn down the engagement before a pilot is scoped.
If practice groups maintain contradictory playbooks and senior counsel won't sit down for a reconciliation pass, the agent inherits the drift and produces confidently inconsistent calls. We will pause the build, as we did at week 4, until it happens.
If partners aren't going to review every redline before it ships to counterparties for the first six months, the agent's wrong-but-confident clause call lands in opposing counsel's inbox unedited. We require a partner-final-pass workflow at week 1 or the pilot doesn't get signed.
Vendor-side data retention, training opt-out, audit-log retention, model tier. These are not post-launch decisions. ABA Op. 512 and FRE 502(b) scope is decided at week 1 with the firm's compliance lead, or the engagement doesn't start.
The agent stays honest because senior counsel from each practice group walks the disagreements weekly. If that meeting won't happen, calibration drifts within months and nobody catches it. The eval set is necessary, not sufficient.
Each link below covers a pillar that fed into this AI legal assistant / AI contract review software build, or that a similar legal AI software build on your stack would draw from.
The legal pillar — privilege-aware AI for law firms, FRE 502 audit-log scaffolding, ABA Op. 512 citation-chain logging across practice groups.
The IDP pillar — multi-modal extraction, schema-validated outputs, confidence routing across document types. The plumbing AI contract review software sits on top of.
The agent pillar — ReAct, plan-and-execute, hierarchical multi-agent recipes. Same eval-first loop used on this AI legal assistant build.
Sonnet 4.6 + Haiku 4.5 integration patterns the AI-powered contract review above uses. Forced JSON, Constitutional-AI posture, BAA-eligible deployment options.
Six AI case studies — AI legal assistant, AI fraud detection, AI knowledge base, AI triage, voice bot, ecommerce chatbot. Same operator detail across every page.
Policy-as-code, audit-log scaffolding, privilege-aware deployment. The plumbing that made this contract review software pilot pass the firm's risk review.
How a contract-review RAG fits inside a broader AI software development company engagement — retrieval + clause classifier + Sonnet reviewer + matter integration.
Contract-review RAG is a domain-specific AI knowledge base over a matter-precedent corpus, with the same chunking, reranking, and eval harness as our product-docs and clinical RAG builds.
The contract-review build started with a fixed-fee audit — clause-library inventory, privilege-ring mapping, eval-set design on real MSAs, and a per-document cost projection before pilot.
Book a fixed-fee AI contract review audit. We'll review the clause library, scope the eval set, recommend an AI-powered contract review recipe (model + retrieval + refusal contract), project run-cost, and tell you honestly whether contract review software built around your playbook is the right shape — or whether you should buy a vendor AI contract review tool. About one audit in five ends with `you need a reconciliation pass before any agent build — here's the SOW for that.`