supporting the legal reasoning process by classification
TRANSCRIPT
Chair of Software Engineering for Business Information Systems (sebis)
Faculty of Informatics
Technische Universität München
wwwmatthes.in.tum.de
Supporting the Legal Reasoning Process by Classification of
Judgments Applying Active Machine LearningLinus Boehm, 07.05.18 Final Presentation, BA-Thesis
Motivation
Legal Reasoning Process
Approach
Implementation
Data Set
Evaluation
Conclusion
Outline
© sebis171120 Boehm BA KickOff 2
Motivation
Legal Documents
• are growing and changing: 2013-2017 period more than 550 adopted laws [1]
• getting more complex
Document Discovery
• often done manually (knowledge-intensive)
• very time-consuming and costly [2]
Supporting the legal reasoning process by classification of Judgments
• common law relies heavily on precedents
• judgments relevant in (German) civil law?
• further development of the law by judges
• Case law is often applied in dynamic areas of life
• German landlord and tenant law is largely shaped by case law
© sebis 3180507 Boehm BA Final
Legal Reasoning Process – Target
© sebis180507 Boehm BA Final 4
Supporting the legal reasoning process of a lawyer:
• Searching for relevant laws, precedent cases, and facts
• Aim: Seeking for evidence to solve the issue in favor of the client
• Support lawyer by automatic recognition of relevant predicates
• e.g. including predicates about unlawfulness of contracts
Legal Reasoning Process – Target
© sebis180507 Boehm BA Final 5
Characteristics of the legal reasoning process of a judge:
• Striving for consistency
• Further development of the law
• Attempts to regulate social relations by applying the
legislature’s intention
Opportunity for Lexia to support the legal reasoning process
of a judge:
• Automatic recognition of relevant predicates
Approach
© sebis180507 Boehm BA Final 6
Manuel Annotation of
JudgmentsImplementation of a BTC for
judgmentsEvaluation
Adaption of Lexia and LexML Evaluating common
performance indicators like
f1score, ROC, recall and
precision
Sentences about the
unlawfulness of contract clauses
Manual Annotation Process
Example:
[…] entsprechende Formularklausel wegen unangemessener Benachteiligung des Mieters nach § 307 BGB
unwirksam […]
Problems with the Annotation „Rule“:
- Sometimes one Part is mentioned in following or previous sentence -> partly causing the False Positives
© sebis180507 Boehm BA Final 7
Contract clause legal justification unlawfullness+ +
e.g:
Vertragsklausel
Formularklausel
§x des Vertrages
e.g:
Paragraph of Civil Code
Precendant of the BGH
e.g:
Unwirksam
Nichtig
Nichtigkeit
Implementation
© sebis180507 Boehm BA Final 8
Implementation
© sebis180507 Boehm BA Final 9
Rest
Interface
User Interface
Importer Data and Text
Mining Engine
Data Access Layer
ES
Rest
Interface
Data Access Layer
MDCSV
Exporter
Machine Learning Engine
Importer
Lexia LexML
Own illustration based on Muhr [3]
Implementation
© sebis180507 Boehm BA Final 10
Rest
Interface
User Interface
Importer Data and Text
Mining Engine
Data Access Layer
ES
Rest
Interface
Data Access Layer
MDCSV
Exporter
Machine Learning Engine
Importer
Lexia LexML
Adaption of the Sentence Segmenter
Adaption of the UI
- Pipeline Configuration
- Annotator for Binary Labels
Implementation
© sebis180507 Boehm BA Final 11
Rest
Interface
User Interface
Importer Data and Text
Mining Engine
Data Access Layer
ES
Rest
Interface
Data Access Layer
MDCSV
Exporter
Machine Learning Engine
Importer
Lexia LexML
- Adding Binary Classification
- Weighted NB and LR to encounter
unbalanced classes
Exporter for Binary Evaluation Metrics
Implementation
© sebis180507 Boehm BA Final 12
Rest
Interface
User Interface
Importer Data and Text
Mining Engine
Data Access Layer
ES
Rest
Interface
Data Access Layer
MDCSV
Exporter
Statistical
Evaluation
Machine Learning Engine
Importer
R Scripts for semi-automated Evaluation Illustration of
different performance Measurements like f1score, PR-
Curve, ROC, etc.
Lexia LexML
Active Learning – Setting
AML Pipline Stages:
• Preprocessing: Tokenizer, Stopwords Removing
• Vector Representation: Binary Word Representation
• Classifier: Multinominal Naive Bayes (Weighted)
• Query Strategy: Uncertainty Sampling (Maximum Vote
Entropy)
© sebis180507 Boehm BA Final 13
Machine Learning
Model
Labeled
training set
Oracle
(human annotator)
Unlabeled
pool
ℒ 𝑈
Data Set
© sebis180507 Boehm BA Final 14
Data Set
Raw Documents
• Over 800 imported and annotated judgments
• BGH judgments of the VIII. Civil Senate
• specialized in law on the sale of goods, landlord and tenancy law
Training & Test Set
• Using a subset of 82 documents
• Only Tenor and Reasoning part imported
• 3135 Sentences of which are 72 True (2,3%)
• 80/20 partition
© sebis180507 Boehm BA Final 15
Evaluation
© sebis180507 Boehm BA Final 16
Discover weight factor
© sebis180507 Boehm BA Final 17
Discover weight factor
© sebis180507 Boehm BA Final 18
Weight 20/0.05
seems to perform
well
Discover weight factor
© sebis180507 Boehm BA Final 19
Comparison of Seed Set Size and Query Size Influence
- Weight true with factor 200 (w8)
- Seed Set random sampled
© sebis180507 Boehm BA Final 20
Influence of Seed Set
© sebis180507 Boehm BA Final 21
Missed class/cluster effect
Influence of Seed Set
© sebis180507 Boehm BA Final 22
Reasons
- Small Random Sampled Seed Sets are often not representative
- Assumption BC 3% labeled as True:
- P(no True in Seed Set with n=20) ≈ 54%
- Unbalanced classes can lead to missed class/cluster effect
- => sufficient exploration phase during the seed set generation is crucial
© sebis180507 Boehm BA Final 23
ROC with unbalanced classes
© sebis180507 Boehm BA Final 24
ROC with unbalanced classes
© sebis180507 Boehm BA Final 25
All Configurations seem to
perform well
N = 624 Predicted = 1 Predicted = 0
Actual = 1 TP=8 FN=6
Actual = 0 FP=4 TN=606
© sebis180507 Boehm BA Final 26
TPR (recall) = 𝑇𝑃
𝑇𝑃+𝐹𝑁FPR =
𝐹𝑃
𝐹𝑃+𝑇𝑁
Reason: TN is dominant -> FPR even for low thresholds relative stable [4]
Influence of Query Size
© sebis180507 Boehm BA Final 27
Influence of Query Size
© sebis180507 Boehm BA Final 28
Influence of Query Size
© sebis180507 Boehm BA Final 29
Recall Heatmap
© sebis180507 Boehm BA Final 30
Recall= 𝑇𝑃
𝑇𝑃+𝐹𝑁
Precision Heatmap
© sebis180507 Boehm BA Final 31
Low FP cause
high precision
even when
Precision = 𝑇𝑃
𝑇𝑃+𝐹𝑃
F1Score Heatmap
© sebis180507 Boehm BA Final 32
Demo
© sebis180507 Boehm BA Final 33
Conclusion
• Binary Classification is a promising approach for identifying predicates in judgments
• Seed Set Generation and Size is crucial
Limitations
• Evaluation with only one preprocessing configuration
• No Cross-Validation
• Strict Annotation rule misses some sentences
Future Work
• Evaluation of different preprocessing configurations in context of judgments
• Validate results with Cross-Validation
• Try different methods to encounter unbalenced classes
• Use a combination of AML and rule-based learning for a better class balance
• Using different predicates then Sentences like paragraphs or “n-grams” of sentences
© sebis180507 Boehm BA Final 34
Technische Universität München
Faculty of Informatics
Chair of Software Engineering for Business
Information Systems
Boltzmannstraße 3
85748 Garching bei München
Tel +49.89.289.
Fax +49.89.289.17136
wwwmatthes.in.tum.de
Linus Boehm
Advisor: Ingo Glaser
17132
References
(1) https://www.bundestag.de/blob/194870/7c8a01e16c98fc9c32ddb203d7bd88e0/gesetzgebung_wp18-
data.pdf
(2) Gruner, Ronald H. "Anatomy of a lawsuit." (2008).
(3) Muhr, Johannes, Master Thesis, Design, Prototypical Implementation and Evaluation of an Active Machine
Learning Service in the Context of Legal Text Classification (2017)
(4) https://www.kaggle.com/lct14558/imbalanced-data-why-you-should-not-use-roc-curve
© sebis171120 Boehm BA KickOff 36
Research Method
Design Science as most fitting method
- Focus is on the creation of an effective IT solution
- Description of the design problem
- Development an new IT artefact
- Strict evaluation of the IT artefact to measure the utility
© sebis180507 Boehm BA Final 37
Environment IS Research Knowledge Base
Business
Needs
Applicable
KnowledgeAssess Refine
Develop/Build Foundations/
Methodologies
People/Organizations
Application in the Appropriate Environment Additions to the Knowledge Base
Technology
• Lexia
• LexML
• Extended Version of
LexML
• Adaptions to Lexia
• Software Developers
• Legal Experts • (Active) Machine Learning
• Binary Text Classification in
jurisdictions
• Rule-based Approaches
• ML Frameworks
• Document Classification
• Sentence Classification
• Binary Text Classification
Justify/Evaluate
Research Questions
How can ML support the legal reasoning process of lawyers and judges?
What challenges occur in a BTC of judgments?
What is the best combination of Seed Set and Query Size for a BTC in context of jurisdiction?
© sebis180507 Boehm BA Final 38