Skip to content
results published · checking…

Clause Risk Review

Does legal pretraining help a model find risky contract clauses?

Multi-label classification of commercial contract clauses into six risk categories on CUAD v1, 510 contracts annotated by lawyers. A TextCNN baseline, BERT and Legal-BERT train on identical word windows and per-contract splits, and are scored per window and per contract, with every threshold tuned on validation data only.

the problem
Contracts are long (97% exceed BERT’s 512-token window) and each label appears in only 1.5–4.2% of windows, so a model that flags nothing would score above 95% accuracy.
what I built
A leakage-safe pipeline: splits per contract, identical windows for every model, per-label thresholds from validation, noisy-OR pooling up to the contract, and 90 tests that pin each of those decisions.
The results dashboard: four findings that shaped the build, such as 97% of contracts exceeding BERT’s 512-token window and 85% of windows carrying no label

The question

BERT and Legal-BERT share an architecture, a parameter count (109,486,854), a context window and a chunk width; they differ only in pretraining corpus. So any gap between them measures domain adaptation. The prediction: the gain should concentrate in labels carried by legal vocabulary.

data

  • CUAD v1 (The Atticus Project): 510 commercial contracts, clause spans annotated by lawyers
  • Median contract 5,039 words, longest 47,733; 85% of windows carry no label
  • Held-out test split: 76 contracts, 3,314 windows, never seen in training

Results

Held-out test split. Document scores pool each contract’s windows with noisy-OR. AP is average precision.

ModelWindow F1Window APDocument F1Document AP
TextCNN0.663—0.8460.904
BERT0.7230.7360.8790.933
Legal-BERT (one seed)0.7180.7690.8750.936
Longformerawaiting a GPU run

what held, and what didn’t

  • not supportedNo measurable advantage from legal pretraining. Legal-BERT is 0.005 below BERT on window F1, while BERT re-run with a second seed moved 0.006 from itself; the two tie at contract level.
  • not supportedThe per-label prediction did not hold: the one clear gain is Termination for Convenience (+0.064 F1), and Cap on Liability, the most jargon-heavy label, gained nothing.
  • supportedBoth transformers clearly beat the CNN, most on Non-Compete, the rarest and hardest label.
  • measuredPooling matters as much as the model: plain averaging scores 0.511 document macro F1 against noisy-OR’s 0.846, because a real clause is diluted across windows that rightly contain nothing.

Limitations

what these results do not show

  • Legal-BERT has one seed so far; run-to-run variance is about ±0.03 macro F1, so smaller differences are not evidence either way.
  • The Longformer comparison (does more context per window help?) has not been run.
  • The CNN’s attributions show it flags Cap on Liability on the word “liability” in insurance clauses, not on capping language.

Reproduce

sources last checked 2026-11-08

pip install -e ".[analysis]"
legal-risk-prepare
legal-risk-train-cnn --epochs 12
legal-risk-train-transformer
legal-risk-evaluate --run_dir outputs/cnn --split test

The pipeline

  • Six labels: Cap on Liability, Non-Compete, License Grant, Audit Rights, Termination for Convenience, Insurance
  • Splits assigned per contract and saved to disk, so overlapping windows never straddle train and test, and every model is scored on the same 76 contracts
  • Windows counted in words, not tokens, so all models see identical boundaries and a score gap is a gap between models
  • Per-label thresholds (0.22 to 0.83) tuned on validation, never on test
  • Noisy-OR pooling from windows to contracts, chosen after measuring seven pooling rules and a learned aggregator
  • A demo that flags passages and, for the CNN, the exact phrases that triggered each flag
  • 90 tests that run as plain scripts, including one showing accuracy would rate an all-negative model above 95%

How it's built

each layer feeds the one below

Data
  • CUAD v1: 510 contracts
  • spans by category
  • splits per contract, saved
Model
  • overlapping word windows
  • TextCNN · BERT · Legal-BERT
  • one shared training loop
Evaluate
  • thresholds from validation
  • noisy-OR pooling to the contract
  • window and document scores
PythonPyTorchTransformersNLP
next projectMed-EEG