A neural network course project · multi-label classification of contract clauses on CUAD v1
Try the demo → Paste a contract and see which clauses the model flags, and why. It runs in your browser.
Commercial contracts are long, and a handful of clauses carry most of the risk. This project trains models to read a contract and flag which of six risk-relevant clause types it contains:
A contract can contain several of these at once, so the model makes six independent yes/no decisions rather than choosing one label.
| Model | What it is | Size | Reads at once |
|---|---|---|---|
| TextCNN | A small convolutional network, trained from scratch. The baseline. | ~4M | 300 words |
| BERT | Pretrained on general English | 110M | 300 words |
| Legal-BERT | Same design as BERT, pretrained on legal text | 110M | 300 words |
| Longformer | Pretrained, built to read long text | 149M | 2,600 words |
The models were chosen to answer two questions. BERT and Legal-BERT differ only in what they were pretrained on. BERT and Longformer differ mainly in how much text they read at once.
All scores are on the 76 test contracts, which no model saw during training or tuning. F1 balances missed clauses against false alarms. AP (average precision) measures how well the model ranks real clauses above false ones. Both run from 0 to 1, higher is better.
| Model | Contract F1 | Contract AP | Window F1 | Runs |
|---|---|---|---|---|
| TextCNN | 0.834 | 0.898 | 0.658 | 3 |
| BERT | 0.861 | 0.917 | 0.697 | 3 |
| Legal-BERT | 0.883 | 0.928 | 0.744 | 3 |
| Longformer | 0.831 | 0.886 | 0.743* | 2 |
Each score is the average over several training runs with different random seeds. Contract scores ask whether the clause was found anywhere in the contract. Window scores ask whether it was spotted in a single window. *Longformer's windows are nine times larger, so its window score is not comparable with the others. Training the same model twice changes F1 by up to about ±0.03, which is why every model was trained more than once.
Yes. Over three runs each, Legal-BERT beat BERT by 0.047 window F1 (0.744 against 0.697), and its worst run beat BERT's best. It also ranked real clauses better (window AP 0.805 against 0.750) and varied less from run to run. At contract level the gap is smaller (0.883 against 0.861) and the runs overlap.
| Clause type (window F1, average of 3 runs) | BERT | Legal-BERT | Gain |
|---|---|---|---|
| Termination for Convenience | 0.586 | 0.693 | +0.107 |
| Cap on Liability | 0.740 | 0.817 | +0.077 |
| License Grant | 0.741 | 0.797 | +0.056 |
| Non-Compete | 0.547 | 0.580 | +0.033 |
| Audit Rights | 0.716 | 0.734 | +0.018 |
| Insurance | 0.854 | 0.847 | −0.007 |
This is the pattern we predicted. The gain is largest on clause types written in legal phrasing, and there is none on Insurance, whose wording ("shall maintain insurance") is ordinary English.
Our first comparison trained each model only once and found no difference. One unusually low Legal-BERT run decided that result, which is why we retrained each model with three different seeds.
No. Longformer reads 2,600 words per window against BERT's 300, yet over two runs it scored 0.831 contract F1: below every BERT and Legal-BERT run, and level with the much smaller CNN.
Two reasons. These clauses can usually be recognised from their own sentence, so extra surrounding text adds little. And bigger windows mean fewer, vaguer training examples: 1,380 windows against BERT's 13,829, each saying only that a clause is somewhere in 2,600 words.
Longformer was given up to ten passes over the data where BERT had three. Both runs stopped improving after pass 7, so the result is not from cutting training short. Its validation F1 after each pass:
| Pass | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Run 1 | 0.301 | 0.403 | 0.589 | 0.535 | 0.662 | 0.649 | 0.700 | 0.685 | 0.661 |
| Run 2 | 0.102 | 0.366 | 0.448 | 0.549 | 0.563 | 0.649 | 0.692 | 0.680 | 0.682 |
Each window gets its own score, but the question is about the whole contract. The first version of this project kept only the highest-scoring window. That throws evidence away: a contract with two windows at 0.80 and 0.79 got the same score as one with windows at 0.80 and 0.02.
Noisy-OR treats every window as a separate witness. The clause is missing only if every witness is wrong:
P(clause) = 1 − (1 − p1) × (1 − p2) × …
| Window scores | Highest window | Noisy-OR |
|---|---|---|
| 0.80 and 0.79 | 0.800 | 0.958 |
| 0.80 and 0.02 | 0.800 | 0.804 |
Measured on the CNN's predictions, noisy-OR scored 0.846 F1 and 0.904 AP against 0.840 and 0.899 for the highest-window rule. That difference is within the noise, so it is not claimed as a win. Noisy-OR is used because it counts all the evidence and was never worse. Simply averaging the windows was far worse, at 0.511 F1.
Code, tests and the full write-up: github.com/ManasDasri/NNDL · detailed results in handbook chapter 12 · CNN model on Hugging Face