6th IEEE International Conference on Signal Processing, Computing and Control (ISPCC 2k21), Oct 07-09, 2021, JUIT,

Solan, India

Analysis of Legal Case Document Automated

Summarizer
2021 6th International Conference on Signal Processing, Computing and Control (ISPCC) | 978-1-6654-2554-4/21/$31.00 ©2021 IEEE | DOI: 10.1109/ISPCC53510.2021.9609442

Naimoonisa Begum Ankur Goyal

Department of Computer Science & Engineering AIT-CSE,
K.L University, Vaddeswaram, Vijayawada, Andhra Pradesh, Chandigarh University, Mohali,Punjab,India
India [email protected]
[email protected]
Abstract— The Legal case documents available after  Listing classifications of automated systems of text
judgements are lengthy in size and need to be summarized for summarizing methods and explanation of the
future reference by the legal experts, professionals and clients various legal text summarization techniques and
who need help for reference to similar cases. In the Common
comparison between the methodologies.
Law system of India, the legal case documents summary is
done manually by specially appointed legal attorneys and is  Results are provided after comparison of examples
referred to as Gold Summaries. This process is time consuming of legal automated systems proposed in the
as the legal case documents are very voluminous in size. We
endeavor the survey of different automated legal domain based
literature.
automated summarization techniques using artificial  The architecture of different automated
intelligence. We try to compare some extraction based legal methodologies of legal text and their advantages
case documents summarizers. The paper initiates with general
introduction to legal domain text summarization techniques,
and disadvantages are discussed.
different metrics to measure the performance of automated  An overview of the various techniques and
summaries generated by various legal domain-based text components used for design and implementation of
summarization methodologies. The conclusion finally suggests
some directions for future research. automated systems are specified.

Keywords—Text-based summarization, Evaluation metrics,

Legal-domain text-based summarization.
II. APPLICATIONS FOR LEGAL CASE DOCUMENTS USING
ARTIFICIAL INTELLIGENCE
I. INTRODUCTION
The modern legal systems nationwide generally follow There can be many applications for legal case documents
these basic systems: Civil Law, Statutory Law, Common using Artificial Intelligence like matching similar type of
Law, Religious Law or mixture of these. The countries cases to get relevant information for reference, performing
which practice Common Law system are Australia, Canada, legal prediction from already available legal documents and
Ireland, India, New Zealand, Hong Kong, South Africa, providing answers to legal questions for awareness of legal
Pakistan, and many more places. domain.
Legal experts were employed to create manual Applying AI to legal case documents should target in
summaries of the legal case documents which would result in helping the Legal professionals but not replace them. The
high payments. The legal case documents automated Legal professionals can target on complicated and complex
summarization can be implemented by applying various AI systems and simple cases can be handled by the AI based
techniques. A vital role is played by Artificial Intelligence in models [9].
the legal case document automated summarization [1] .
III. GENERAL METHODS OF TEXT AUTOMATED
Automation of legal case documents helps in easily finding SUMMARIZATION
the relevant legal case summaries to match the legal domain
requirements of different stakeholders like legal experts, Automated text summarization results in producing a clear
professionals and clients. The Automation of Legal case and communicative summary without losing the key
systems will help the users to understand the main points of content and meaning of the original text [3].
original legal document without the necessity to read
The automated summaries should retain the semantics of the
complete document. The different stakeholders will get
text and match human written summaries. Over time various
benefit from these automatically generated summaries and it techniques have been developed for the same.
saves a lot of time and effort for them.
There have been several legal case document automation A. Single-document summarization
techniques available for several courts across different
countries. But it imposes difficulty in applying the similar From a single input document creating a relevant summary.
techniques of automated summarization for all types of B. Multi-document summarization
courts documents. This is because the legal jargon, structure, From a collection of input documents creating a relevant
vocabulary and size of different courts globally may be summary.
different [2] .
The main contributions of this paper analysis include:

978-1-6654-2554-4/21/$31.00 ©2021 IEEE 533

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.
C. Extractive summarization 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
Selection of most relevant sentences is done from input 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 + 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠
documents and relevant summary is created.
𝑒𝑞 − 1
D. Abstractive summarization
Summary is generated automatically using new sentences The recall gives the ratio between the true positives and
from the input document . the true positives + false negatives on a dataset as shown in
eq-2.
The text summarization has various applications like
Media monitoring, Newsletters, Financial research, Legal 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠
contract analysis, Social media marketing, Question 𝑅𝑒𝑐𝑎𝑙𝑙 =
answering and bots. 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 + 𝐹𝑎𝑙𝑠𝑒 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠
IV. THE TASKS IN ARCHITECTURE OF AN AUTOMATED TEXT
SUMMARIZATION SYSTEM 𝑒𝑞 − 2

A. Pre-Processing
The F-score, or F1-score, is used as measure for accuracy on
Structured representation of the original text [10] is a dataset. The F-score combines the model’s precision and
produced using many linguistic techniques like removal of recall and is defined to calculate harmonic mean of
stop-words, sentences segmentation, stemming, part-of- precision & recall.
speech tagging, and words tokenization.

B. Processing technique B. ROUGE

The abbreviation of ROUGE is Recall-Oriented
Applying one among the available text-based Understudy-for-Gisting-Evaluation [1]. The metric makes
summarization techniques for converting input documents to comparison between an automatically or machine generated
automated summary. summary with human-produced or reference summary.

C. Post-Processing ROUGE-N – It measures the unigram overlap, bigram

overlap, trigram overlap and n-gram overlap.
From the generated automated summary, some issues
like anaphora resolution and selected sentences reordering ROUGE-L – using LCS it measures the matching longest
are done before the final summary is generated. in-sequence of words.

ROUGE-S – This metric is used for skip-bigram

V. MOTIVATION cooccurrence. It is used for measuring the number of skip-
bigram overlapping between automated and reference
The domain specific Automated Text Summarization translations.
systems are specific to summarization of documents from
specialized domain like health care documents, legal-
case documents etc.
C. BLEU
The vocabulary used in the judgments of courts, legal The abbreviation for BLEU is Bi-lingual Evaluation
documents, acts and amendments, Bills, is exclusive and Understudy is a metrics that is used to compare human
has different nomenclatures. translation with the machine generated translation [2].
The comparison is made between n-gram of any reference
The syntax used in legal writing is complex, which translation or human translation with the machine generated
makes the legal documents unique. These features make translation. BLEU is considered as the first metrics to have
the automated legal text summarization a challenging the higher correlation of reference translation to human
task. translation.
VI. METRICS FOR EVALUATION OF MACHINE TRANSLATION The output of BLEU is generated between 0 and 1. The
value 1 is considered as human generated and machine
A. Precision and Recall generated translations are much correlated.
The two most frequently used measures from the
information retrieval domain are named as precision and
other is recall [3].
Precision is the ratio of true positives and true positive+
False positives on a dataset as shown in eq-1.

534

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.
VII. ALGORITHMS OF LEGAL CASE DOCUMENTS AUTOMATED Punctuation Removal means removal of all unwanted
SUMMARIZATION punctuations from text document excluding the dot
operator(.), as it acts as separator of sentence.
A. C4.5, Winnow Naive Bayes, and SVM
In Legal-domain Document, extra white spaces are inserted
to format the document, which adds more space, to cut the
The machine-learning methodologies such as C4.5,
Winnow, Naïve Bayes, and SVM worked on information size of document the Extra White spaces removal is
extraction which relies on semantic processing of legal- required.
domain case documents.
A method in which a root word is produced by the removal
of prefixes and suffixes of the words is proposed as Word
These algorithms are based on the characteristics like cue
words, sentences length, location, quotations objects and Stemming.
thematic phrases. Relevant phrases are identified by using relative frequency
technique, called as Identification of Key Phrase.
B. FLEXICON, SALOMON, LEXA Process of identification and separation of paragraph into a
sentence is termed as Sentence Segmentation.
Different systems were developed with the goal of explicitly
summarizing legal-domain based documents. Tokenization is termed as a process that splits Sentence into
individual words.
FLEXICON stands for Fast Legal Expert Consultant and
was one among the earliest systems developed. It is 2) Extraction of Features of Sentence
keyword based which refers terms in large database and
finds important regions of the text. Following Pre-processing technique, sentence feature
extraction methodology is applied where each sentence from
SALOMON which was introduced later uses cosine the document denotes a vector point. The Eleven features
similarity to find texts that are similar to group regions [4]. used in extraction methodology were used which are listed
LEXA is another system which is based on citation analysis as:
to generate automated summaries.
a) Sentence-Position feature
C. Hybrid Systems
In the Legal-based domain Documents, sentences appearing
The collaboration of techniques and methods from artificial at first paragraph and last paragraph are expected to contain
intelligence results in Hybrid system [11]. The following important information, hence highest score is given to them.
techniques were used in hybrid systems to provide the
automated summarization of Legal case documents. b) Proper Noun
• Matching technique using Keyword or Key phrase The sentence, that contains a greater number of Named
• Case-based technique Entity which starts with a Uppercase letter, referred as a
Proper Noun, may be considered as relevant sentence to be
Hybrid system frame work takes input to the system which included in an Automated Document Summary.
is a requested by the legal experts which are legal judgement
documents.
The system is loaded with past cases and roles are decided c) Sentence-Length
by preprocessing and hence structured text is maintained.
When the user wants to search certain case details the key Sentence Length is a process which identifies the best
phrase is matched, and the relevant of summarized sentences from the summary. The Length of sentence is
documents is generated. normalized by counting the number of words in a sentence.

d) Tf * isf
D. Feature-based Techniques of Extraction
Term-Weight calculates the importance of the sentences by
1) Preprocessing method tracing the frequency of re-occurrences of the terms within
Preprocessing method cleans the noisy text covering the document, termed as raw-term frequency. As importance
spelling and grammatical mistakes [12]. The pre-processing may not increase proportionately along the raw-term
methods are applied, named as Case Folding, Word frequency, there may be necessity of inverse-sentence
Stemming, Identification of Key Phrases, Sentence frequency, that will filter the words termed to be important
Segmentation and Tokenization, Removal of Stop Words, re-occurring words from the sentences.
Punctuations and White Spaces.
e) Sentence-Sentence Similarity
The translation of uppercase characters to lower case is
called as Case Folding like “CRIMINAL” TO “criminal”. Neighboring Sentences Similarity from a document is found
Removal of insignificant words which may appear using a measure referred as cosine-similarity.
recurrently but does not provide relevant meaning to text
processing can be termed as Removal of Stop Word. f) Citation

535

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.
Citations are needed in legal documents which are used for Six automated tools out of which four were online chosen
referring someone. The Common Law System is followed namely Auto Summarizer, Text Summarizer, Split Brain,
by India where judgements can be used for other case and SMMRY1 remaining were Apple Inc.’s Summary and
reference. V., VS. Denotes the citations. Galgani et al.’s summarization was used for comparison [4].
The legal domain experts were asked to provide summaries
g) Local and Layout Features for some selected cases.

Every Legal based Document has a layout structure, The ROUGE scores generated for each automated system
which has a head-note with information like court, judge, compared to the summaries legal expert is shown in Table I.
petitioner, respondent names and dates. Local Features and
the Layout Features has details that are related to Legal Case Summarizer could cover most important points of the
Document head note. case and proved to be most effective in generating
reasonable flow of events.
h) Paragraph-Structure feature

Every legal Document will have a distinct internal Table I ROUGE scores of automatic generated text summaries referred
from [4].
paragraph format hence high score can be assigned to first &
last paragraphs.
Auto Text Split SMM Apple Case- Galgani
Metrics
i) Thematic-Word Sum Sum Brain RY Sum Sum
In Legal- domain document final decision called as main
theme is specified by Thematic Word and should be Rouge1 .207 .183 .241 .248 .175 .194 .132
included into summary.

j) Indicators/Cue-Phrases Rouge2 .089 .072 .146 .137 .097 .114 .049

Cue-Phrase represents frequently used key-phrases,

which indicates the fact finders of the case and act as Rouge3 .059 .049 .123 .104 .075 .091 .026
indicators of rhetorical roles for summary.
Rouge4 .048 .043 .117 .090 .068 .085 .019
h) Legal-Thesaurus

It means words/phrases which includes Legal RougeL .017 .015 .056 .062 .033 .061 .017
vocabularies for training data.

3) Fuzzy logic text summarization

F. LetSum
After the Feature Extraction process which includes Letsum is one among the few automated text summarization
eleven features mentioned above the Legal-domain Text techniques developed specifically for legal-based case
Summarization is implemented using Fuzzy Logic documents. LetSum approach was developed for automated
techniques. summaries for Canadian legal documents.
Finally sentence score for each sentence is generated and
sentences are extracted to generate the final summary. The approach was to identify and extract the relevant units
by determining the segment themes in the legal judgement
documents.
E. CaseSummarizer
The LetSum approach for automated summarization does
The Case Summarizer is an automated summarizer preprocessing by splitting the text into paragraphs, sentences
which was developed for Australian legal documents. and tokens. After this step Thematic segmentation is done
by identifying the themes. Then Filtration is done to
The internal pipeline of Case Summarizer consists of eliminate the citations and using selection the relevant units
distinct three steps the preprocessing, sentence relevant are identified. Production phase is to control the size of
scoring, and domain processing method, using user interface summary and the finally the Table of Summary is created.
the summaries were presented externally.
The automated summaries were generated using the
The Case Summarizer could generate summaries which themes of the segments as INTRODUCTION, CONTEXT,
could provide reasonable idea with respect to context of JURIDICAL ANALYSIS and CONCLUSION [6].
case, but missed some important points.
For automatic summary evaluation, comparison is made
Evaluation of automated summarization was performed for between sentences produced by LetSum and the sentences
a set of five exclusively selected legal-case documents of produced by the systems namely Baseline, Microsoft Word
the Federal Court of Australia (FCA). and Pertinence Mining, Mead [7], with the human generated
summaries. The result of evaluation is shown in Table II.

536

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.
The LetSum system shows the best performance as it Case Arguing .824 .787 .805
generated the higher score.
Case History .808 .796 .802

Table II ROUGE scores of statistical evaluations with LetSum showing Arguments .860 .846 .853
best scores reference values from [6].
Ratio
.924 .901 .912
Decidendi
ROUGE ROUGE ROUGE
ID ROUGE 1 ROUGE 2
3 4 L
Final Decision .986 .962 .974
Baseline .47244 .27569 .19391 .14472 .34683
Micro
Word .44473 .21295 .13747 .09727 .29652 .896 .864 .879
Average
Pertinen
ce .32833 .15127 .09798 .07151 .22375
Mining VIII. COMPARATIVE STUDY OF ALGORITHMS OF AUTOMATED
LetSum .57500 .31381 .20708 .15036 .45185 SUMMARIZATION OF LEGAL TEXT
The ROUGE scores are generated by comparison of
Mead .45581 .22314 .14241 .10064 .32089 automated generated summarization with the gold standard
summarization.
Table-4 shows the ROUGE scores generated by
G. Graphical Model (CRF) algorithmic methods of legal text summarization techniques
for the Indian supreme court legal-domain based case
Graphical models were one the of the models suggested for judgments .
automated legal-based text summarization task.
In the automated summarization process preprocessing stage Table IV Results of different Extractive legal summarization
Techniques are referred from [8].
starts with sending a input legal document in which the
division of document is done into sentences segments, and Methods Type Rouge 1 Rouge 2 Rouge L

Exec Time (Sec)

tokens.[5]
Some new feature identification techniques were introduced
to explore alignments of paragraph which includes the
understanding of text abbrevations ,section numbers etc, Re-call

F-Score

Re-call

F-Score

Re-call

F-Score
which are precise to of legal documents structure. In the
preprocessing stage the other NLP tools, like stemming,
filtering stop list word, etc., are also carried out.
Graphic .15 .34
S .386 .351 .171 .297 2.4
The resulting understandable words are used in the term al Model 9 3
distribution model called K-mixture model for the
normalization of terms. The document segmentation is done LetSum S .408 .298 .112
.07 .37
.235
10.
by identifying rhetorical status of each sentence using 3 1 16
graphical model (CRFs). The sentences that are selected at Case
term distribution model present the structured way of .06 .15 7.9
Summar US .198 .139 .094 .094
3 4 5
summary . izer

The graphical model was categorized based on seven

types of rhetorical roles for the subdivision of legal-domain The table displays the execution times for summary
based case document . The roles were case identifying, generation of the different legal text summarizing algorithms
establishment of facts of case, case arguing, case history , for a specific case.
ratio decidendi, arguments and also final decision [4].
Conditional Random Fields (CRFs) were used to perform The legal domain experts attempted to justify the
segmentation of legal-based text into seven categories. automated summaries generated considering the features like
how these summaries can focus on 4 relevant aspects of a
Conditional Random Fields model abbrevated as CRFs legal case judgment document – (1) the court ruling or
model is used as one among the trending graphical-based holding of the Court combined with a reason , (2) the legal
models for the text-based subdivision problems is proved as based facts, (3) the statutes , and (4) precedents [8].
best one among existing models. CRF Model performance Graphical model gives better performance compared with
for text segmentation task is shown in the Table III. LetSum with less execution time, ROUGE scores, and results
in best F-score.
Table III For the specified 7 rhetorical categories the Precision, Recall and
F-measure are listed in table referred from [5]. The legal domain experts analysed the quality of
automated summary generated by Graphical Model and gave
Rhetorical
Precision Recall
F- the opinion that summary had statues quoted and included
Category Measure case arguments but however the other two parts were
Case missing.
.946 .868 .905
Identifying
Establishing
.924 .886 .904 The Graphical Model and LetSum shows comparable
Facts performance with respect to ROUGE scores. But one the

537

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.
drawback noticed with Letsum is that it has highest high level of research is still in process. Therefore more
execution time which can impact online applications. research is still required. The methodologies implemented
for legal documents could not provide the reasoning factor
Legal-domain experts analysed the overall quality of the for holding or ruling of the case. This feature plays an
automated summary by LetSum and gave the opinion that important role of the automated summary, as the legal
summary covered facts of case, statues and precedents. experts can decide whether to include or not the legal case
From the different legal text summarization techniques, for the reference. Therefore more research is still required.
Graphical Model technique and the LetSum technique has
comparable performance, while Case Summarizer technique REFERENCES
performs poor comparatively. [1] Lin, C. Y. (2004). Rouge: A package for automatic evaluation of
summaries. In Text summarization branches out (pp. 74-81).
IX. CONCLUSION [2] Yang, An, et al.(2018), "Adaptations of ROUGE and BLEU to better
evaluate machine reading comprehension task." arXiv preprint
The difficulty is encounterd in manual text summarization arXiv:1806.03578.
of the dense amount of the legal textual content , hence [3] Yang, A., Liu, K., Liu, J., Lyu, Y., & Li, S. (2018). Adaptations of
ROUGE and BLEU to better evaluate machine reading
Automated summarization appears to be the technolgy time comprehension task. arXiv preprint arXiv:1806.03578.
saving and instant referral to legal related issues.There are
[4] Polsley, S., Jhunjhunwala, P., & Huang, R. (2016). Case summarizer:
many automated legal text summarizers from the literature, a system for automated summarization of legal texts. In Proceedings
but they are still less efficient in performance from the of COLING 2016, the 26th international conference on
results compared to manual text summarization. The Computational Linguistics: System Demonstrations (pp. 258-262).
computer still finds to identify and understand the ‘‘most [5] Saravanan, M., Ravindran, B., & Raman, S. (2006). Improving legal
relevant” parts of the legal text. The main goal of applying document summarization using graphical models. Frontiers in
AI is not meant to replace the legal experts but to help Artificial Intelligence and Applications, 152, 51.
their work. Hence automated result, should be regarded as [6] Farzindar, A. (2004). Atefeh Farzindar and Guy Lapalme,'LetSum, an
automatic Legal Text Summarizing system in T. Gordon (ed.), Legal
reference. The legal professionals can deal with complex Knowledge and Information Systems. Jurix 2004: The Seventeenth
cases and the simple cases can be dealt by the Automated Annual Conference. Amsterdam: IOS Press, 2004, pp. 11-18. In Legal
model. However, for the safety purpose , these automated Knowledge and Information Systems: JURIX 2004, the Seventeenth
simple cases are required to be still be reviewed by experts. Annual Conference (Vol. 120, p. 11). IOS Press.
The LegalAI must act as a means of support to help legal [7] Radev, D., Otterbacher, J., Qi, H., & Tam, D. (2003). Mead reducs:
based system. Michigan at duc 2003. In Proceedings of the Document
Understanding Conference.
In this paper, we have to given special emphasis on different [8] Bhattacharya, P., Hiware, K., Rajgaria, S., Pochhi, N., Ghosh, K., &
legal-domain text-based summarization techniques. This Ghosh, S. (2019). A comparative study of summarization algorithms
survey shows that significant work has been done on applied to legal case judgments. In European Conference on
extraction based summarization area, but abstractive based Information Retrieval (pp. 413-428). Springer, Cham.
summaries from legal text documents has to be explored in [9] Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020).
How does NLP benefit legal system: A summary of legal artificial
detail. intelligence. arXiv preprint arXiv:2004.12158.
The legal based text summarization methodologies applied [10] Gupta, V., & Lehal, G. S. (2010). A survey of text summarization
to legal case documents could not give the ruling or holding extractive techniques. Journal of emerging technologies in web
for the case added with a reason factor which is an intelligence, 2(3), 258-268.
important aspect of summary generation as this would help [11] Kavila, S. D., Puli, V., Raju, G. P., & Bandaru, R. (2013). An
automatic legal document summarization and search using hybrid
the legal experts for reference.
system. In Proceedings of the international conference on frontiers of
A successful text summarization technique is based on how intelligent computing: Theory and applications (FICTA) (pp. 229-
easily a user can understand the automated summary of the 236). Springer, Berlin, Heidelberg.
document. [12] Megala, S. S., Kavitha, A., & Marimuthu, A. (2014). Feature
extraction based legal document summarization. International Journal
The abstraction based summarization is almost similar to of Advance Research in Computer Science and Management Studies,
human summarization, but as it is a huge complex area and 2(12),346-352..

538

Authorized licensed use limited to: Zhejiang University. Downloaded on January 28,2024 at 06:59:56 UTC from IEEE Xplore. Restrictions apply.