Clause Risk Review
Does legal pretraining help a model find risky contract clauses?
Multi-label classification of commercial contract clauses into six risk categories on CUAD v1, 510 contracts annotated by lawyers. A TextCNN baseline, BERT and Legal-BERT train on identical word windows and per-contract splits, and are scored per window and per contract, with every threshold tuned on validation data only.
- the problem
- Contracts are long (97% exceed BERT’s 512-token window) and each label appears in only 1.5–4.2% of windows, so a model that flags nothing would score above 95% accuracy.
- what I built
- A leakage-safe pipeline: splits per contract, identical windows for every model, per-label thresholds from validation, noisy-OR pooling up to the contract, and 90 tests that pin each of those decisions.
The question
BERT and Legal-BERT share an architecture, a parameter count (109,486,854), a context window and a chunk width; they differ only in pretraining corpus. So any gap between them measures domain adaptation. The prediction: the gain should concentrate in labels carried by legal vocabulary.
data
- CUAD v1 (The Atticus Project): 510 commercial contracts, clause spans annotated by lawyers
- Median contract 5,039 words, longest 47,733; 85% of windows carry no label
- Held-out test split: 76 contracts, 3,314 windows, never seen in training
Results
Held-out test split. Document scores pool each contract’s windows with noisy-OR. AP is average precision.
| Model | Window F1 | Window AP | Document F1 | Document AP |
|---|---|---|---|---|
| TextCNN | 0.663 | — | 0.846 | 0.904 |
| BERT | 0.723 | 0.736 | 0.879 | 0.933 |
| Legal-BERT (one seed) | 0.718 | 0.769 | 0.875 | 0.936 |
| Longformer | awaiting a GPU run | |||
what held, and what didn’t
- not supportedNo measurable advantage from legal pretraining. Legal-BERT is 0.005 below BERT on window F1, while BERT re-run with a second seed moved 0.006 from itself; the two tie at contract level.
- not supportedThe per-label prediction did not hold: the one clear gain is Termination for Convenience (+0.064 F1), and Cap on Liability, the most jargon-heavy label, gained nothing.
- supportedBoth transformers clearly beat the CNN, most on Non-Compete, the rarest and hardest label.
- measuredPooling matters as much as the model: plain averaging scores 0.511 document macro F1 against noisy-OR’s 0.846, because a real clause is diluted across windows that rightly contain nothing.
Limitations
what these results do not show
- Legal-BERT has one seed so far; run-to-run variance is about ±0.03 macro F1, so smaller differences are not evidence either way.
- The Longformer comparison (does more context per window help?) has not been run.
- The CNN’s attributions show it flags Cap on Liability on the word “liability” in insurance clauses, not on capping language.
Reproduce
sources last checked 2026-11-08
pip install -e ".[analysis]"
legal-risk-prepare
legal-risk-train-cnn --epochs 12
legal-risk-train-transformer
legal-risk-evaluate --run_dir outputs/cnn --split testThe pipeline
- Six labels: Cap on Liability, Non-Compete, License Grant, Audit Rights, Termination for Convenience, Insurance
- Splits assigned per contract and saved to disk, so overlapping windows never straddle train and test, and every model is scored on the same 76 contracts
- Windows counted in words, not tokens, so all models see identical boundaries and a score gap is a gap between models
- Per-label thresholds (0.22 to 0.83) tuned on validation, never on test
- Noisy-OR pooling from windows to contracts, chosen after measuring seven pooling rules and a learned aggregator
- A demo that flags passages and, for the CNN, the exact phrases that triggered each flag
- 90 tests that run as plain scripts, including one showing accuracy would rate an all-negative model above 95%
How it's built
each layer feeds the one below
- CUAD v1: 510 contracts
- spans by category
- splits per contract, saved
- overlapping word windows
- TextCNN · BERT · Legal-BERT
- one shared training loop
- thresholds from validation
- noisy-OR pooling to the contract
- window and document scores