Clause Risk Review

A neural network course project · multi-label classification of contract clauses on CUAD v1

Try the demo → Paste a contract and see which clauses the model flags, and why. It runs in your browser.

Commercial contracts are long, and a handful of clauses carry most of the risk. This project trains models to read a contract and flag which of six risk-relevant clause types it contains:

A contract can contain several of these at once, so the model makes six independent yes/no decisions rather than choosing one label.

What the model does

  1. Reads the data. CUAD v1 is 510 real contracts in which lawyers marked where each clause appears. 358 contracts are used for training, 76 for tuning, and 76 are held back for the final test.
  2. Cuts each contract into windows. The median contract is 5,039 words, far more than a model can read at once, so each contract is cut into overlapping windows of 300 words (2,600 for Longformer).
  3. Labels each window. A window is marked with a clause type only if it actually overlaps a clause the lawyers marked.
  4. Trains a classifier. For every window, the model outputs six probabilities, one per clause type. 85% of windows contain no clause at all, so mistakes on real clauses are weighted more heavily during training.
  5. Picks a cutoff per clause type. Each clause type gets its own yes/no threshold, chosen on the tuning contracts and never on the test contracts.
  6. Combines windows into one answer per contract. Every window counts as a separate piece of evidence, using a rule called noisy-OR (explained below).

The models

ModelWhat it isSizeReads at once
TextCNNA small convolutional network, trained from scratch. The baseline.~4M300 words
BERTPretrained on general English110M300 words
Legal-BERTSame design as BERT, pretrained on legal text110M300 words
LongformerPretrained, built to read long text149M2,600 words

The models were chosen to answer two questions. BERT and Legal-BERT differ only in what they were pretrained on. BERT and Longformer differ mainly in how much text they read at once.

Results

All scores are on the 76 test contracts, which no model saw during training or tuning. F1 balances missed clauses against false alarms. AP (average precision) measures how well the model ranks real clauses above false ones. Both run from 0 to 1, higher is better.

ModelContract F1Contract APWindow F1Runs
TextCNN0.8340.8980.6583
BERT0.8610.9170.6973
Legal-BERT0.8830.9280.7443
Longformer0.8310.8860.743*2

Each score is the average over several training runs with different random seeds. Contract scores ask whether the clause was found anywhere in the contract. Window scores ask whether it was spotted in a single window. *Longformer's windows are nine times larger, so its window score is not comparable with the others. Training the same model twice changes F1 by up to about ±0.03, which is why every model was trained more than once.

Question 1: does pretraining on legal text help?

Yes. Over three runs each, Legal-BERT beat BERT by 0.047 window F1 (0.744 against 0.697), and its worst run beat BERT's best. It also ranked real clauses better (window AP 0.805 against 0.750) and varied less from run to run. At contract level the gap is smaller (0.883 against 0.861) and the runs overlap.

Clause type (window F1, average of 3 runs)BERTLegal-BERTGain
Termination for Convenience0.5860.693+0.107
Cap on Liability0.7400.817+0.077
License Grant0.7410.797+0.056
Non-Compete0.5470.580+0.033
Audit Rights0.7160.734+0.018
Insurance0.8540.847−0.007

This is the pattern we predicted. The gain is largest on clause types written in legal phrasing, and there is none on Insurance, whose wording ("shall maintain insurance") is ordinary English.

Our first comparison trained each model only once and found no difference. One unusually low Legal-BERT run decided that result, which is why we retrained each model with three different seeds.

Question 2: does reading more of the contract at once help?

No. Longformer reads 2,600 words per window against BERT's 300, yet over two runs it scored 0.831 contract F1: below every BERT and Legal-BERT run, and level with the much smaller CNN.

Two reasons. These clauses can usually be recognised from their own sentence, so extra surrounding text adds little. And bigger windows mean fewer, vaguer training examples: 1,380 windows against BERT's 13,829, each saying only that a clause is somewhere in 2,600 words.

Longformer was given up to ten passes over the data where BERT had three. Both runs stopped improving after pass 7, so the result is not from cutting training short. Its validation F1 after each pass:

Pass123456789
Run 10.3010.4030.5890.5350.6620.6490.7000.6850.661
Run 20.1020.3660.4480.5490.5630.6490.6920.6800.682

Combining windows: noisy-OR

Each window gets its own score, but the question is about the whole contract. The first version of this project kept only the highest-scoring window. That throws evidence away: a contract with two windows at 0.80 and 0.79 got the same score as one with windows at 0.80 and 0.02.

Noisy-OR treats every window as a separate witness. The clause is missing only if every witness is wrong:

P(clause) = 1 − (1 − p1) × (1 − p2) × …

Window scoresHighest windowNoisy-OR
0.80 and 0.790.8000.958
0.80 and 0.020.8000.804

Measured on the CNN's predictions, noisy-OR scored 0.846 F1 and 0.904 AP against 0.840 and 0.899 for the highest-window rule. That difference is within the noise, so it is not claimed as a win. Noisy-OR is used because it counts all the evidence and was never worse. Simply averaging the windows was far worse, at 0.511 F1.

Limits

Code, tests and the full write-up: github.com/ManasDasri/NNDL · detailed results in handbook chapter 12 · CNN model on Hugging Face