\section{Introduction}
\label{intro}

In our times, Computer Science has been recasting many different fields of expertise. Many subfields of Computer Science, like Natural Language Processing (also known as NLP), have steadily improved a myriad of professional and scientific activities. Natural language processing (NLP) is a theory-motivated range of computational techniques for the automatic analysis and representation of human language (\cite{cambria2014}). Even simple NLP mechanisms, such as dictionaries and word frequency, with effective processing, can identify interesting underlying facts about written texts. \cite{zhong2020does} article explore in depth how  NLP methods benefit legal systems. 

One of the human areas of knowledge that are the most dependent on text is Law. Millions of papers, legislation, court decisions, and appeals are produced daily, and many different specializations, such as lawyers, judges, defendants, and plaintiffs, have different necessities that could be supplied by intelligent systems. 

Therefore, it is a reasonable thought to consider that AI could also be used to optimize the daily routines of Law professionals. As this research intends to show, recent studies in the field of AI and Law have been growing for the last years, opening new research areas and market applications (\cite{harper2021legal}). Even though Law methods are long-established, AI for Law is still a developing field. Also, when analyzing the whole state of the art, we could see that the overwhelming amount of research covers the English language only. Analyses in this area for Brazilian Portuguese are still infant, presenting an exciting challenge for new applications (\cite{oliveira2020indicators}). 

As a primary goal, this research sought to develop a framework to predict judicial outcomes in the São Paulo Justice Court\footnote{Tribunal de Justiça do Estado de São Paulo, Brasil, TJSP.}. The São Paulo Justice Court is the largest judicial court on the planet, considering the number of legal processes\footnote{https://www.tjsp.jus.br/QuemSomos (in Brazilian Portuguese)}. A computational prediction model that offers a satisfactory result for this large judicial court could be of great use and, maybe after fine-tuning, it could also attend any other court. 
Over the last years, researchers have been dedicated to predict outcomes of judicial cases, using NLP methods and AI (\cite{sun2020legal}, \cite{antos2021practical}). However, to the best of our knowledge, we have no research with this intention in Brazilian Portuguese or for Brazilian courts, as of 2021. 

At first, we built a text crawler to collect data from the judicial outcomes. This data set formed an annotated corpus to train and evaluate the proposed prediction system. A pre-processing phase was required for feature extraction from the data set. Selected features were inserted into machine learning frameworks, such as neural networks and support vector machines, to preview judicial outcomes of the court. 

The remainder of this work is organized as follows. In Section~\ref{backgd}, we present related work in the theoretical background, indicating the previous researches and knowledge. In Section~\ref{meth}, we explain the methodology adopted in our research. In Section~\ref{exp}, the experiments performed on the labeled data set are described. Finally, in Section~\ref{conc} we present our conclusion and future work possibilities.

\section{Theoretical Background}
\label{backgd}

In this section, we present an overview of the existing work concerning Artificial Intelligence and Natural Language Processing applied to legal texts. We also present some of the challenges intrinsically related to the field. 

\subsection{Natural Language Processing, Artificial Intelligence, and Law}
According to \cite{info:doi/10.2196/16816} Natural language processing (NLP) refers to the ability of machines to understand and explain the way humans write and talk.
Earlier, \cite{Manning:1999:FSN:311445} defined NLP as the utilization of quantitative and probabilistic approaches to the automatic processing of texts and speech. Natural Language Processing, unlike image and audio processing, traditionally treat words as discrete atomic symbols, with arbitrary encodings (\cite{Phuoc2017}).
NLP is considered as a subfield of Computer Science (CS) and Artificial Intelligence concerned with computer programs that can learn from experience and thus improve their performance over time (\cite{Russell:2003:AIM:773294}). 

Machine Learning is a promising trend in the field of CS, and it opens new possibilities of research in many areas of knowledge. This technique relies on the hypothesis that large groups of data have, in general, hidden patterns that can be statistically inferred. Therefore, in order to develop a robust machine learning strategy, the most crucial requisite is finding a mathematical model that represents the entry data as trustworthy as possible. 

Those concepts, over the last decades, have intertwined with fields that do not necessarily belong to the natural sciences.
\cite{Branting2017a} cite that automation of legal reasoning and problem-solving has been a goal of Computer Science  research from its earliest days. However, according to the author, broad adoption of legal computer systems never occurred, and Computer Science and law remained a niche research area with little practical impact. 

Firstly, in order to understand how NLP and AI and interact with Law, we must first understand what this field is. Law, as defined by \cite{Le2015}, is the system of rules that guarantees peace, personal freedom, and social justice by regulating human behaviors in all aspects of life. Legal documents, according to the authors, are documents that state some contractual relationship or grant some rights. 

Since NLP works well with documents, the application of NLP in Law may hence seem like a perfect match. \cite{Hyman2015} point out the Zubulake versus UBS Warburg case, a series of trials and decisions dealing with what data a litigant has to preserve and under what circumstances the parties must pay for search and production costs, as a seminal case for AI in Law. According to the authors, this case became a landmark for the practical applications of the research field because of the inability of the defendant to recover hundreds to thousands of emails that were claimed by the plaintiff to be relevant to the main issue in the lawsuit. 

However, regardless of all the recent advancements in technology, \cite{Surden2014} argues that modern AI algorithms have been unable to replicate most human intellectual abilities, falling far short in advanced cognitive processes, such as analogical reasoning, that are basic to legal practices. This phenomenon is due, according to \cite{Branting2017a}, to the difficulties of scaling the logic-based approach to the dimensions of complex, dynamic, real-world legal systems. Two main challenges have been written by the authors: the problem of efficiently and verifiably representing legal texts in the form of local expressions, and the difficulty of evaluating legal predicates from facts expressed in the language of ordinary discourse.  

As reported by \cite{Ashley2009}, two long-time goals in AI and Law research are classifying case texts automatically and predicting case outcomes in a manner that is clear so that attorneys can understand them. \cite{Sulea2017a} argue that law professionals would greatly benefit from the type of automation provided by machine learning. According to the authors, artificial intelligence ``systems could act as a decision support system or at least a sanity check for law professionals.'' \cite[pp. 1]{Sulea2017a}. This finding is of utmost importance to overcome one of the main arguments that opponents to the adoption of AI in the Law field have: that the intelligent systems would eventually replace law professionals. We can infer, by looking at the state-of-the-art in the area, that present researchers seek to help law professionals with the most demanding and repetitive tasks, leaving them free to perform better in other activities where computers can not help. 

This task, however, has a high level of difficulty. As posited by \cite{Surden2014}, many of the tasks performed by attorneys do appear to require the type of higher-order intellectual skills that are beyond the capability of current AI techniques. However, as written by \cite{Branting2017a}, recent advances in both human language technology and techniques for large-scale data analysis have vastly increased capabilities for automated interpretation of a legal text. Possible hope for AI, therefore, becomes the development of Big Data technologies.  

According to \cite{Branting2017a}, there are three main areas of data-centric research in AI and Law:

\begin{itemize}
\item \textbf{Case-oriented research}, in which the researchers focus on the significant characteristics of cases considered as a whole. Examples of practical applications of this area are litigation assistance and tax recommendation systems. 
\item \textbf{Document-oriented research}, in which researchers focus on the analysis of individual documents. Examples of applications are information extraction for Law, automated summarization, and form completion. 
\item \textbf{Corpus-oriented research}, in which researchers focus on the proprieties of entire collections of legal texts, including network structures, temporal and sequential characteristics. Possible applications are argumentation mining and judicial dataset analysis. 
\end{itemize}

\cite{Branting2017a} goes on specifying possible legal tasks amenable to each research area. Those tasks are essential not only on a research level but also to inform potential future research pathways for the field of AI and Law. \\
These possible tasks are:
\begin{enumerate}
\item Legal analysis;
\item Information retrieval;
\item Legal prediction;
\item Argument generation;
\item Dialectical argumentation; 
\item Document drafting;
\item Legal planning; 
\item Legislative drafting;
\item Trend analysis;
\item Adjudication; and
\item Legal document auditing and quality control.
\end{enumerate}

In the effort of this research, we began by adopting document-oriented research. We have found this option as the most viable one. After all, case-oriented researches are difficult to conduct in Brazil, because some of the documents are heard in private, with legal confidentiality\footnote{In Brazilian Portuguese, \emph{segredo de justiça}.}. Therefore, one would not have complete access to the case files. On the other side, we do not have access to a full corpus of labeled documents, so a corpus-oriented research is also not the primary goal. As each of the judicial outcomes goes as a single document in the Brazilian law system, document-oriented research was the leading choice to deal with the present problem, to perform legal prediction.

\subsection{Nature of legal activity and practice}

Two different schools of thinking dictate the outcome of judicial cases: Legal Formalism and Legal Realism. As mentioned by \cite{Liu2017}, Legal Formalism postulate that judicial decision-making is rationally determinate, in which the procedure of judges' decision making can be modeled either deductively, using formal rules, or with more complex reasoning paradigms. On the opposite side, Legal Realism dictates that formal legal rules are rationally indeterminate on many occasions, insisting that judges decide appellate cases primarily by responding to the stimulus of the facts of the case. 

As stated by~\cite{Surden2014}, a lawyer might employ a combination of judgment, experience, and knowledge of the law to make reasoned predictions about the likelihood of outcomes on particular legal issues or overall issue of liability, often in contexts of considerable legal and factual uncertainty. \cite{Zeng2007} write that lawyers frequently make arguments by analyzing, assessing, abstracting, and interpreting the significance of similarities and differences between cases. Successful arguments critically depend on whether the cited precedents can convince the judge or court. 

This finding of \cite{Surden2014} is confirmed by the works of \cite{Barraud2017}. They argue that judges are people that depend strongly on their knowledge of rules, syllogisms, and logic, while judging with their intuitions and sensibility. Judges also have very particular ways of writing, as mentioned by \cite{Alarie2017}, frequently developing very peculiar writing skills in order to individualize the way they present information. \cite{Le2015} also remembers that legal documents, in general, are very formal, and their structure is also very important, maybe as much it's readability. \cite{Branting2017a} write that even if legal rules appear with perfect fidelity, the terms in the rules are typically impossible for a layperson to interpret. 

Those problems of interpretation can be summarized as written by \cite{Boella2016}.
\begin{itemize}
\item \textbf{Terms with different meanings than ordinary:} Some words have acquired meaning from statutory definitions and scholarly of judicial interpretations that differ from their purpose in standard language; 
\item \textbf{Terms can vary in different contexts and jurisdictions:} Also called polysemy, names can have multiple meanings, according to the legal field of expertise;
\item \textbf{Intentional vagueness:} Legislation can also be intentionally vague sometimes in order to allow for social and technological changes; and
\item \textbf{General problems of language:} Imprecise use of language, opening possibilities of interpretation.
\end{itemize}



Those precedents, as written by \cite{Conrad2017a}, have patterns that repeat themselves, benefiting practitioners by seeing such patterns comprised of facts, claims, counter-claims, legal principles applied, analysis and decisions. However, as stated by \cite{Alarie2017}, those patterns are challenging to find, because of the highly contextualized nature of legal writing. 

This task increases in difficulty due to the fact, as reported by \cite{Katz2017general}, that courts have to deal with many different types of juridical analysis, such as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. Because of this broad range of judicial specializations, the amount of data gathered is also growing in exponential level. Therefore, as regarded by \cite{Zeng2007}, the need to manage legal knowledge effectively for lawyers and judges to locate knowledge and information becomes urgent, due to the rapidly growing volume of the landmark cases. 

\subsection{Text-based analysis of judicial texts}

\cite{Aletras2016} have  discovered that formal facts of a case are the most important predictive factor. This factor is of utmost importance since legal texts also have specific characteristics that make them different from other kinds of narratives. For instance, mentions in legal texts have specific structures, which are different from mentions in the public domain, as analysed by~\cite{Tran2014}. \cite{Aletras2016} have written that the textual content and the different parts of a case are essential factors that influence the outcomes reached by judicial courts. Even before the technological era, as \cite{Alarie2017} state, judicial decisions were placed in published volumes, as were legislative acts, regulations, and academic and practice-based commentaries in order to re-utilize legal texts in different cases. 

\cite{Surden2014} writes that the combination of human intelligence and com\-put\-er-ba\-sed analytics will likely prove superior to that of social analysis alone. \cite{Kingston2017} also countersign this idea by saying that AI technologies ought to be able to assist by providing best advice, asking all and only the relevant questions, monitoring activities, and carrying out assessments. 

When studying legal precedents via AI, the task of matching specific case facts may prove to be complicated. As mentioned by \cite{Zeng2007}, many issues may arise from many different perspectives, such as what the relevant law is, how one could interpret the applicable law in the context, how one could apply the law and what the facts of the case are. This problem is also observed by \cite{Sannier2017}, mentioning that when identifying and elaborating legal requirements, analysts need to follow the cross-references in legal texts and consider the additional information in the cited provisions. \cite{Tran2014} also realize this issue, mentioning that, at the discourse level, legal texts contain a lot of reference phenomena. 

\cite{Aletras2016} raise the hypothesis that published judgments could be used to test the possibility of a text-based analysis for ex-ante forecasting of outcomes. This idea is corroborated by \cite{Surden2014}, arguing that entities concerned with legal outcomes could, in principle, leverage data from past client scenarios and other relevant public and private data to build machine learning predictive models about likely future results on particular legal issues that could complement legal counseling. 

\cite{Aletras2016} mention that the judgments of judicial courts have distinctive structures, which makes them particularly suitable for a text-based analysis. \cite{Ashley2009} have also followed this way of thinking. The authors have developed a model to extract information from the textual descriptions of the facts of decided cases and apply that information to predict the outcomes of the issues. However, analysts may not forget that, as cited by \cite{Sannier2017}, a critical complexity that arises in the analysis of legal texts is that legal provisions are typically interrelated and spread over different texts that cannot be considered in isolation of one another. 

\cite{Liu2017} write that an effective way to explain past decisions of judges and to predict future ones is to study the empirical variables that reflect the non-legal facts of cases, rather than the pure legal deductive arguments.  In the same line of thinking, \cite{Zeng2007} write that past legal cases are frequently used to support ideas and judicial opinions, even in classical methods, not using artificial intelligence or any mathematical or statistical models. According to the authors, past cases are called \emph{precedents} in the Common Law system and can be followed, analogized, distinguished or overruled. \cite{Sulea2017} also wrote that general NLP methods have played an essential role in the intersection between artificial intelligence and law. 

It is important to highlight that the main core of Common Law, the use of precedents, is not a common tenet of the Civil Law, the law model used in the Brazilian judicial system. Even though the use of precedents has recently been more frequent after Brazil's Constitution of 1988\footnote{http://www.brazil.gov.br/about-brazil/news/2018/11/civil-law-tradition-guides-rights-in-brazil-but-common-law-is-also-present}, Brazil still focuses on laws and codes instead of analyzing past cases.

\subsection{Potential issues in Artificial Intelligence and Law}

Regarding the technical issues, as stated by \cite{Surden2014}, there are some well-known limitations to the application of AI in Law. The first one is that a model will only be useful to the extent that the class of future cases has pertinent features in common with the prior analyzed topics in the training set. Therefore, the model will not contemplate subtle changes in judicial thinking over time, only if those changes arise to represent a considerable size of the training data. 

The authors also present an example: not every law firm will have a stream of cases that are sufficiently similar to one another such that the past case has elements that are useful to predict future outcomes. Hence, one may infer that only the largest law firms will have the necessary financial and technological assets to develop such models. 

As to the social issues, another possible problem, as stated by \cite{Surden2014}, is an overgeneralization, also known in machine learning as \emph{overfitting}. The model in intrinsically based in the cases provided for the training set. Thus, if the training set has cases that are so finely tuned to the idiosyncrasies of a few judicial matters, it will not be able to have the necessary adaptability to cases of different judicial natures. For that reason, the past case data upon which a machine learning algorithm is trained may be systematically biased in a way that leads to inaccurate results in future legal cases. 

\cite{Katz2017general} suggest that qualitatively-oriented legal experts tend to suggest model improvements based on anecdote or their untested mental model, instead of reliable and factual data. The authors write that, to support a case from a model's future applicability, it should consistently outperform a baseline comparison. This requisite is not necessary only for scientific purposes but also to gain attorneys' trust in the model. 

Another problem, not entirely dependent on Artificial Intelligence, is the problem of accessibility of Law. As written by \cite{Boella2016}, difficulties of accessibility arise because of the following reasons.

\begin{itemize}
\item Law is increasing in scope, volume, and complexity. This problem is also noted by \cite{Francesconi2009}, which points out the size of legal literature as one of the factors of paramount importance for consideration in future researches; 
\item The myriad of areas of expertise of law, frequently not classified intuitively on official legislative portals. Again, \cite{Francesconi2009} indicate this problem to be of fundamental importance, stressing out the importance of delimitating the legal domain;
\item Legal norms are coming from different sources, such as regional, national, or supranational authorities. \cite{Hyman2015} use the Philip Morris litigation case as an example of this item, saying that, at one point on the issue, more than 30 million pages of documents were available, coming from many federal government agencies; and 
\item New legislation modifying or overriding existing norms but not explicitly saying so. 
\end{itemize}

Therefore, expert consultation still proves itself to be fundamental to the making of a practical AI system for Law. Because of this limitation, \cite{Kingston2017} argue that AI commercial systems should rely on the Pareto Principle: an 80-20 rule, where the AI system should cover 80\% of the questions, and leave the remaining 20\% to the decision of the legal experts using the application. That remaining 20\% can also be used to refine the AI model, turning the algorithms gradually more useful to predict new outcomes. 

\subsection{Usefulness of NLP and AI in Law}

As cited by \cite{Aletras2016}, recent advances in Natural Language Processing and Machine Learning have provided us with many different tools to build predictive models that can be used to infer the patterns driving judicial decisions. This possibility is also suggested by \cite{Surden2014}, writing that attorneys could potentially use machine learning to highlight useful unknown information that exists within their current data but disappears due to complexity. 

\cite{Liu2017} also write that NLP  and ML have augmented possibilities based on the exploration of the semantic of law and case texts. And according to \cite{Barraud2017}, NLP and AI have expanded the limits of justice, transforming it into a predictive, quantitative, statistic, and simulative justice. As stated by \cite{Alarie2017}, intelligent judicial systems can not only help lawyers with timely and objective assessments of their claims but also governments, by using legal classifiers to help evaluate claims and manage litigation risks. 

Another practical use of NLP in the legal field is discourse analysis. Discourse analysis is a widespread tool utilized in social research. Moreover, researchers use it in addition to ML methods in Natural Language Processing fields, such as opinion mining and sentiment analysis. 

A very effective way to compare different thoughts on the judiciary process is by studying the language used by other people.  As mentioned by \cite{fairclough2003}, social agents are not entirely free, being socially constrained by the language they use. Comparing different writing styles can be a way to infer what are the social constraints of each region of the world. 

Law texts are usually dealing with different positions between the agents. Those differences are directly expressed by their writing styles. \cite{fairclough2003} argue that differences in the style of writing can be summarized into five different scenarios:

\begin{itemize}
\item an openness to, acceptance of, recognition of difference; an exploration of difference, as in ‘dialogue’ in the richest sense of the term
\item an accentuation of difference, conflict, polemic, a struggle over meaning, norms, power
\item an attempt to resolve or overcome difference
\item a bracketing of difference, a focus on commonality, solidarity
\item consensus, a normalization and acceptance of differences of power which brackets or suppresses differences of meaning and norms
\end{itemize}

The author also mentions that those writing styles are not disconnected, and only a single discourse can share more than one level of writing style. In this work, we presume that those levels of stylistic difference can be more effectively measured by NLP. As \cite{fairclough2003} mentions, one way of capturing those differences ``is through looking at collocations, patterns of co-occurrence of words in texts, simply looking at which other words most frequently precede and follow any word which is in focus, either immediately or two, three and so on words away.''. 

NLP can adequately address this issue by using, for example, the concept of n-grams. In a similar way of thinking, \cite{brown1992} also propose the utilization of n-grams to identify similarities between expressions. For them, we can presume two histories are equivalent if they end in the same $n - 1$ words.

\subsection{Recent Advancements}

Artificial Intelligence methods have been used effectively in diverse areas of the law. \cite{Mcshane2012}, for example, have used AI and NLP to build a Hierarchical Bayesian model to predict fraud settlements in American federal securities class action lawsuits, using data from risk metrics, identifying predictors of settlement incidence and settlement amount. \cite{talley2011measure} used regular expressions and Latent Semantic Analysis to analyze force measures clauses in mergers and acquisitions agreements, by replicating, correcting, and extending the reach of the hand-coded data. 

\cite{Zeng2007} have used case-based reasoning to solve new judicial problems by remembering and adopting previous similar situations, developing a new set of sub-elements for legal case representation. In case-based logic, as mentioned in the previous sections, each judicial case usually consists of three parts: description of the problem or situation, the solution to that problem or situation, and the outcome of that solution. By introducing new contextual features, the authors have managed to help retrieval in the domain of accident compensation.

\cite{Gokhale2017} have developed a co-training algorithm to classify human rights abuses, using SVM and Logistic Regression on a domain ontology created for the domain of human rights as background knowledge, so as to extract the initial terms for generating the labeled data to train the classifier. \cite{Branting2017a} have used Hierarchical Attention Networks, SVMs and Maximum Entropy classifications for decision support in administrative adjudication, such and routine licensing, permitting, immigration, and benefits decisions of the Board of Veterans Appeals (BVA) and World Intellectual Property Organization (WIPO) domain name dispute decisions. SVMs are also used by \cite{Fornaciari2013} to automatically detect deception, such as slander and false testimony, by analyzing the results obtained by using stylometric techniques to identify deceptive statements in a corpus of hearings collected in Italian courts.

\cite{Remmits2017} have used Latent Dirichlet Allocation to discover the main topics of discussion in judicial outcomes of the United States Supreme Court. In this work, the author also compares whether or not legal experts and people with a non-legal background agree in their judgments, discovering that domain experts and non-domain experts might evaluate topics differently. \cite{Mochales2011} have used argumentation mining to structure better legal arguments, capturing main issues and evidence of a given corpus, also stating that the method needs further research to automatically acquire the necessary background knowledge and more specifically common sense and world knowledge.

\cite{Le2015} have used index extraction using structural information of sentences for Japanese legal documents, assigning each token with a weight, which is a statistical score to indicate its importance. \cite{ElJelali2015} also uses information retrieval to support adjudication in Italian court decisions, by adopting machine learning and natural language processing techniques to better match disputant case descriptions (informal and concise) with court decisions (formal and verbose). 

Also, using information retrieval techniques, \cite{Hyman2015} have developed a process model for knowledge discovery, using judicial cases as examples of their model's applicability, finding the constructs of uncertainty, context and relevance while retrieving information. \cite{Boella2016} have developed a system to create ontologies destined to give the relevant law on any given topic, using NLP tools to semi-automate the lower-skill tasks. With a similar goal, \cite{Francesconi2009} have created a system to search and retrieve Italian legal literature, by creating a centralized index of legal resources, using OAI and machine learning approaches. \cite{Aires2017} have used deontic logic to identify potential norm conflicts in contracts.

Some recent research have also been explicitly conducted in Brazilian Portuguese.  An a first example, \cite{Bertalan2020PredictingJO} used a model that uses six algorithms in order to perform legal prediction. Another example is found in \cite{DeAraujo2017}, that have used ontology-based algorithms adopted by using a domain ontology of legal events and a set of linguistic rules integrated through inference mechanism to classify legal documents in Brazilian Supreme Court judicial outcomes. 

In the field of legal prediction, our main research goal, recent advancements have significantly improved the state-of-the-art. Considering a very significant work related to this project, \cite{Aletras2016} have used a dataset of cases from the European Court of Human Rights, containing cases that violate three articles of their Convention. These are:
  \begin{itemize}
      \item Article 3 - \emph{Prohibits torture and inhuman and degrading treatment};
      \item Article 6 - \emph{Protects the right to a fair trial}; and
      \item Article 8 - \emph{Provides a right to respect for one’s private and family life, his home, and his correspondence}.
  \end{itemize}
  The authors then selected an equal number of cases that violate (+1) and do not violate (-1) each of the Articles. After using regular expressions and pre-processing tools to extract the texts, the authors obtained N-gram features for the Procedure, Circumstances, Facts, Relevant Law, Law, and the Full case. After the extraction of N-grams, they formed groups using vector-space models to find the main topics of each article. They used Support Vector Machines (SVM) to achieve a 0.78 accuracy on predicting Topics for Article 3, 0.84 on predicting Topics and Circumstances for Article 6, and 0.78 on predicting Topics and Circumstances for Article 8. 

After the prediction step, the authors also studied the weights of their SVMs so as to find the main words that impacted each of the violations. For Article 3, for example, words such as ``\emph{injury, protection, ordered, damage, civil, caused, failed, claim, course, connection, region, effective, quashed, claimed, suffered, suspended, carry, compensation, pecuniary, ukraine}'' impacted positively towards a violation. On the other hand, ``\emph{sentence, year, life, circumstance, imprisonment, release, set, president, administration, sentenced, term, constitutional, federal, appealed, twenty, convicted, continued, regime, subject, responsible}'' impacted negatively towards a violation. They repeated the same process for the other two Articles.

\cite{Katz2017general} have used Random Forests to predict the behavior of the Supreme Court of the United States. The authors rely on a United States Supreme Court dataset with 240 variables, such as chronological variables, case background variables, justice-specific variables, and outcome variables. The authors then add a disposition coding for each one of the cases, labeling them as 'Reversed', 'Affirmed', or 'Other'. They converted all other categorical variables to binary or indicator variables. For the prediction effort, the authors used different methods, such as Support Vector Machines and Random Forests, which they have found to be the best tool to work in their dataset. With Random Forests, the authors reached as high as 0.77 of recall value for the `Affirmed' class. In the binary labeling, they had a 0.78 of recall value for the `Not Reversed' cases. 

In another influential research, \cite{Sulea2017a} have done a similar investigation, predicting the law area (like Criminal, Social, or Commercial Law) and decisions of the French Supreme Court by using lexical features and SVMs. The authors used a diachronic collection of rulings from the French Supreme Court (\emph{Court de Cassation}, in European French). The complete collection contained 131,830 documents, each consisting of a unique ruling and metadata. Standard metadata available in most documents included law area, timestamp, case ruling (e.g., cassation, rejet, non-lieu, etc.), case description, and cited laws. 

After pre-processing, their dataset contained 126,865 different court rulings, each containing a case description and four different types of labels: a) a law area; b) the date of decision; c) the case ruling itself, and; d) a list of articles and laws cited within the description. Features were then selected using hierarchical clustering, and SVMs were used to classify the dataset. The authors reached as high as 90.2\% of accuracy to classify the law area; 96.9\% of accuracy, using a 6-class SVM to classify the court ruling; and 74.3\% of accuracy, using a 7-class SVM to estimate the date of the case and ruling. 

All those works indicate that the field of AI and NLP applied to Law is a growing field, offering new and promising research opportunities to the future. Law is a field that covers a wide range of applications, and usually produces a significant amount of data, especially text. Consequently, researchers can produce substantial outcomes, with noteworthy applications. 

\section{Research Methodology}
\label{meth}
In this section, we describe the domain characterization and the necessary steps to conclude the research. We also describe the evaluation measures used to assess the effectiveness of the proposed model. 

\subsection{Data Collection and Pre-processing}

For the research, we collected a corpus of judicial outcomes from the eSAJ\footnote{http://esaj.tjsp.jus.br/cjpg/}, the electronic system of the TJSP, Tribunal de Justiça de São Paulo. We selected a few previously defined judicial subjects to restrict the documents captured. We chose only judicial subjects with very well-defined outcomes. Namely, second-degree murder (\emph{Homicídio simples}), from now on called homicide, and active corruption (\emph{Corrupção Ativa}), from now on called corruption. We then selected those judicial outcomes with the condemnation or absolution of the defendant. Many different judicial subjects do not have explicit terms for condemnation or absolution.  Therefore,  it is of utmost importance to find those judicial subjects with clear and well-established results.

\begin{figure}
	\centering
 	  \caption{Methodological phases of the research.}
		\includegraphics{metodologia.jpg}
	\label{fig:preprocessing}
\end{figure}

We have implemented a web text crawler to capture the data from eSAJ, São Paulo Justice Court's judicial electronic system. As the user can select from many different fields to exhibit judicial opinions, such as classes, subjects, judges, and process numbers, the crawler could choose the appropriate texts. The crawler saved the documents retrieved from the queries to files. The complete code for the text crawler is available in the author's GitHub page \footnote{https://github.com/vbertalan}. 

Using the text crawler developed for this research, we have collected 2,467 cases in total, only selecting homicide and corruption subjects, resulting in 1,681 homicide cases and 786 corruption cases. The crawler was used to gather documents from different periods. The total distribution, including absolutions and condemnations, is shown in Table~\ref{totaldocs}. Related crimes were not considered for this research.

\begin{table}[h!]
\centering
\caption{Documents collected for the research.}
\label{totaldocs}
\begin{tabular}{|l|r|l|r|l|}
\hline\noalign{\smallskip}
\textbf{Judicial subject}        & \textbf{Homicide} & \textbf{\%} & \textbf{Corruption} & \textbf{\%} \\ \hline
\textbf{Number of Absolutions}   & 844               & 50.2        & 197                  & 25.0        \\ \hline
\textbf{Number of Condemnations} & 837               & 49.7        & 589                 & 75.0        \\ \hline
\textbf{Total Cases}             & 1,681             & 100         & 786                 & 100         \\ \hline
\noalign{\smallskip}\hline
\end{tabular}
\end{table}

In the next step, we pre-processed the data retrieved to remove unnecessary information. We began this step by tokenizing the text and eliminating stopwords, as suggested by~\cite{Moens2001}. For both tasks, we relied on NTLK developed by~\cite{10.5555/1717171}


As we were to manipulate the data gathered in machine learning and NLP algorithms, it was necessary to label the dataset to have a guide to the algorithms of supervised learning. We have read each of the judicial outcomes, classified them between condemnation (+1) and absolution (-1). We have used the professional guidance of Brazilian lawyers with the purpose of better understanding the texts. The language adopted worldwide in the field of Law is notoriously obscure. Therefore, we decided that professional consulting was necessary to understand the outcome of each of the judicial cases fully.


\subsection{TF-IDF}
TF-IDF, the acronym for `Term Frequency--Inverse Document Frequency'. It is a numerical statistical measure used to quantify the importance of a word in a document or corpus. Let $N$ be the number of documents in a corpus $D$ ($N=|D|$). Let also  $|\{d \in D: t \in d\}|$ be the number of documents where the term $t$ appears.
TF-IDF may be calculated according to Equation~\ref{equation_tfidf}, with $\mbox{TF}_{t,d}$ standing for the frequency of term $t$ in $d$, whereas $\mbox{IDF}_{(t,D)}$ stands for the inverse document frequency which is a measure of how much information the word provides.

\begin{align}
\mbox{TF-IDF}_{(t,d, D)} &= \mbox{TF}_{(t,d)} \cdot \mbox{IDF}_{(t,D)} \nonumber \\
                   ~ &= f_{(t,d)}) \cdot \log \frac{N}{n_t} \label{equation_tfidf}
\end{align}


\subsection{Word Embeddings}

We can use different methods with the intention of transforming data. One of the most popular is word embedding. Word embeddings, as defined by \cite{turian2010}, are vectors composed by real numbers distributed over an inter dimensional space, induced by semi-supervised learning. Word embeddings have become a very effective alternative to transform the pure text into mathematical values, making it easier to manipulate data using machine learning algorithms. The algorithms calculate the similarities between two vectors by using cosine similarity. Each dimension of the vector represents a characteristic, intending to capture semantic, synthetic or morphological proprieties of a word in a distributed way. 

In this research, we have used GloVe, a method created by \cite{pennington2014glove}. GloVe, a reduction for Global Vectors, is a word embedding method developed to learn word vectors such that their dot product equals the logarithm of the words’ probability of co-occurrence. Rather than using a window to define local context, GloVe constructs an explicit word-context or word co-occurrence matrix using statistics across the whole text corpus. It has the ability to combine local and global representations of a term by mixing the features of two model families, namely the global matrix factorization and local context window methods. We have used the pre-trained GloVe corpora developed by \cite{ICMCGlove} for Brazilian Portuguese. 



\subsection{Machine Learning Algorithms}

Over our bibliographic review for this dissertation, we came across different NLP algorithms, used in many purposes. We selected algorithms that had good performances in similar classification works so that we could try their accuracy in our datasets. We show the full list of chosen algorithms and a few examples of their applications in Table \ref{algorithmchoices}. 

\begin{table}[h!]
\centering
\caption{Chosen algorithms and practical NLP applications.}
\label{algorithmchoices}
\resizebox{\textwidth}{!}
{\begin{tabular}{|c|l|}
\hline
\textbf{Algorithm}                  & \multicolumn{1}{c|}{\textbf{Practical NLP Applications}} \\ \hline
Logistic Regression                 & \vtop{\hbox{\strut \cite{Liu2017}, for prediction of circumstances and topics of law cases}\hbox{\strut \cite{Pelle2018}, for offensive text detection}}           \\ \hline
Linear Discriminant Analysis        &   \vtop{\hbox{\strut \cite{Krestel2009}, for tag recommendation in search websites}\hbox{\strut \cite{Pavlinek2017}, for text classification in newsgroups datasets}}              \\ \hline
K Nearest Neighbors                 & \vtop{\hbox{\strut \cite{Chantar2011}, for document categorization in the Arabic language}\hbox{\strut \cite{Desmet2014}, for automatic recognition of suicidal messages in social media}}                \\ \hline
Classification and Regression Trees &  \vtop{\hbox{\strut \cite{7050801}, for measuring sentiment analysis on Twitter}\hbox{\strut \cite{Rios-Figueroa2011}, for predicting judicial independence in Latin American courts}}               \\ \hline
Naive Bayes                         &  \vtop{\hbox{\strut \cite{Harcourt2015}, for feature selection and vectorization in legal documents}\hbox{\strut \cite{Rios-Figueroa2011}, for measuring sentiment analysis on Facebook statuses}}               \\ \hline
Support Vector Machines             &   \vtop{\hbox{\strut \cite{Do2017}, for legal question answering and ranking}\hbox{\strut \cite{Sulea2017}, for predicting law area and decisions of French Supreme Court cases}}              \\ \hline
Multilayer Perceptron               &   \vtop{\hbox{\strut \cite{Rao2016}, for political text classification}\hbox{\strut \cite{Sa2017}, for defining the author reputation of product comments}}              \\ \hline
Recurrent Neural Networks           & \vtop{\hbox{\strut \cite{Alschner2017}, for automated production of legal texts}\hbox{\strut \cite{Kim2017}, for demographic inference on Twitter}}                 \\ \hline
Long Short Term Memory      &  \vtop{\hbox{\strut \cite{Li2017}, for political ideology analysis}\hbox{\strut \cite{Xie2017}, for mining product adverse events in social media}}               \\ \hline
Gated Recurring Unit        &  \vtop{\hbox{\strut \cite{Luo2017}, for predicting charges for criminal cases}\hbox{\strut \cite{zhang2018detecting}, for detecting hate speech on Twitter}}               \\ \hline
Hierarchical Attention Networks     & \vtop{\hbox{\strut \cite{Branting2017a}, for predicting models for decision support in administrative adjudication
}\hbox{\strut \cite{gao2018hierarchical}, for information extraction from cancer pathology reports}}                \\ \hline
\end{tabular}}
\end{table}

\subsubsection{Logistic Regression}

Logistic regression is a model used to predict a categorical variable, usually binary, from a series of explanatory continuous or binary variables.  Like all regression analyses, logistic regression is a predictive analysis. It is used to predict the occurrence of an event directly. The algorithm takes a weighted combination of the input features (in our research, the result of the TFIDF transformation). It passes it through a sigmoid function, which outputs any real number to a number between 0 and 1. 

As cited by \cite{10.5555/1162264}, it works as a statistical method destined to find an equation that predicts an outcome for a binary variable, from one or more response variables. The response variables can be categorical or continuous, as the model does not strictly require continuous data. Logistic regression uses the log odds ratio rather than probabilities to predict group membership, and an iterative maximum likelihood method rather than the least squares to fit the final model.

\subsubsection{Linear Discriminant Analysis}

Linear Discriminant Analysis is a technique from multivariate statistics used to discriminate and classify objects. It ranks each sample in one of many populations, using a \emph{p} number of characteristics, seeking to minimize the probability of a wrong classification. In order to do this, the algorithm uses a combination of linear features that present a higher capability of classification between populations. This combination is called a discriminant function. 

\cite{balakrishnama1998linear} argue that Linear Discriminant Analysis handles the case where the within-class frequencies are unequal, and their performances has been examined on randomly generated test data. The discriminant function seeks to verify which variables from the subset are essential to classify that subset among the populations. Since our research aims to rank the subset among two groups, condemnation and absolution, we have used the Linear Discriminant Analysis technique. This approach uses the discriminant function to classify each value between two populations, by selecting the minimum ratio of the difference between a pair of group multivariate means to the multivariate variance within the two groups.


\subsubsection{K-Nearest Neighbors - KNN}

The algorithm K-Nearest Neighbors (KNN) is a supervised learning algorithm that intends to find the \emph{k} labeled examples closest to a non-classified example, by getting the label of those most comparable examples. The algorithms in the KNN family do not demand a significant computational effort during the training phase. However, the computational cost to label a new instance is considerably high, since, in the worst case, this example will have to be compared will all the other instances present in the training dataset. 

As cited by \cite{10.5555/1162264}, in regions of high data density, models may lead to over-smoothing and washing out of a structure that might otherwise be extracted from the data. However, simplifying the model may lead to noisy estimates elsewhere in data space where the density is smaller. Thus the optimal choice for the model may be dependent on location within the data space. Nearest-neighbor methods for density estimation address this issue.

\subsubsection{Regression Trees}

Classification and regression trees (CART) are machine-learning methods for constructing prediction models from data. As stated by \cite{lohtrees}, the models are obtained by recursively partitioning the data space and fitting a simple prediction model within each partition.  Classification trees work for dependent variables that take a finite number of unordered values, with prediction error measured in terms of misclassification cost. Regression trees are for dependent variables that take continuous or ordered discrete values, with prediction error typically measured by the squared difference between the observed and predicted values. 

In this research, we the CART version available in the NLTK Framework. The method of analysis uses classification rules made by decision trees. The tree begins with a root node with all the text characteristics. The following nodes contain subsets and subdivisions of the data. Each division results in precisely two nodes. This method allows the identification of homogeneous groups of data by systematically comparing their characteristics, intending to establish a relationship between explanatory variables and a single answer variable - in our case, the label of condemnation or absolution. The model is adjusted through successive divisions in the dataset, to make the subsets every time more homogeneous towards the answer variable. The division process is repeated until none of the variables selected show significant influence in the division, or if the subset is too small to be divided again. 

\subsubsection{Naïve Bayes}

Naïve Bayes classifiers are a family of probabilistic classifies that draw on the Bayes's Theorem to generate models with high independence assumptions between the features. As stated by \cite{Manning:2008:IIR:1394399}, Bayesian classifiers are the ones in which an object $x$ is assigned to a class, $C_{k}$, based on the probability that $x$ belongs to $C_{k}$. We show an example of the formula in Equation \ref{equationbayes}.

\begin{equation}
\label{equationbayes}
\mbox{P}(C_{k} | x) = \frac{\mbox{P}(C_{k})\mbox{P}(x | C_{k})}{\mbox{P}(x)}
\end{equation}

Where:

\begin{itemize}
	\item ${P}(C_{k} | x)$ is the probability of hypothesis $C_{k}$ given the data $x$. This value is called the posterior probability.
	\item ${P}(C_{k})$ is the probability of hypothesis $C_{k}$ being true (regardless of the data). This value is called the prior probability of $C_{k}$.
	\item ${P}(x | C_{k})$ is the probability of data $x$ given that the hypothesis $C_{k}$ was true. 
	\item ${P}(x)$ is the probability of data $x$ (regardless of the hypothesis).
\end{itemize}

\subsubsection{Support Vector Machines}

Support Vector Machines are an algorithm destined to binary classification by plotting the elements of a dataset and trying to separate it by defining a separation function. The most effective separation function is the one that shows the best classification by offering the largest margin between the two given classes. 

In this model, the support vectors are the dots from both classes that are closest to the separation function. This separation function is also called a hyperplane. The algorithm plots the new element in the same space to predict new features, and verifies in which group the new element has fallen into.  

As stated by \cite{10.5555/1162264}, if there are multiple solutions, all of which classify the training data set precisely, then we should try to find the one that will give the smallest generalization error. According to the author, the support vector machine approaches this problem through the concept of the margin. This concept is defined to be the smallest distance between the decision boundary and any of the samples. In Support Vector Machines the decision boundary is chosen to be the one that maximizes the margin.

\subsubsection{Multilayer Perceptrons - MLP}

Multilayer Perceptrons are algorithms that extract characteristics from a dataset, composed by units called \emph{Perceptron neurons}, which are interconnected. Those neurons are units responsible for controlling the error that arises from each prediction made by the algorithm. 

The MLP architecture works in layers. The first layer functions as a sensory receptor, receiving an input signal (data). The last layer is known as the exit layer, in which we can see the answer from the MLP to the input signal. Between the first layer and the exit layer, we can have one or more hidden layers. The hidden layers and the exit layer are composed of Perceptron neurons, receiving the input signal, processing the signal through an activation function, and passing it to the next layer. 

The weights (synapses) have to be calibrated after every input so that the network learns the most important characteristics from the dataset. The calibration of those synapses works through an algorithm destined to minimize the error produced by the MLP, known as \emph{backpropagation}. 

\subsubsection{Recurrent Neural Networks - RNN}

MLPs work for many applications. However, if we have an input that behaves like a time series when a value intrinsically depends on the previous output, this signal is going to affect the next input directly. Values tend to get lost inside MLP architectures, since all the inputs receive a singular output, without dependencies between them.

Recurrent Neural Networks are an architecture that handles variable-length sequential input by way of a recurrent, shared hidden state. An RNN is a machine learning model in which $M$ inputs are entirely connected to $N$ units. Since the meaning of a word in a text is entirely dependent on the previous and posterior terms, we can also consider text a time series, being strong candidates for RNNs. 

In an RNN, the output of a unit in step $n+1$ does not only depend on the external exits of the network in the previous step $(u(n-k),k=0,...,M-1)$, but also of the previous outputs of units $y_k(n),k=1,...,N$, having feedback inputs in the recurring layer, allowing the network to keep information in memory through the epochs. 

\subsubsection{Long Short Term Memory Networks -- LSTM}

After running an RNN for many epochs, a known problem that may occur is called the \emph{vanishing gradient problem}, in which the gradient of the loss function decays exponentially with time, effectively preventing the weight from changing its value. In this scenario, the RNN becomes stalled and does not offer any valuable prediction capabilities. 

A Long Short Term Memory Network (LSTM) is a different type of RNN architecture destined to avoid the vanishing gradient problem. LSTM networks use select units in addition to standard units. Those units add \emph{memory cells} that can incorporate data inside memory for more extended periods than an ordinary RNN could. 

A set of \emph{gates}, namely the \emph{input gate}, the \emph{output gate}, and the \emph{forget gate} is used to control whether and when data enters the memory, the emission of an output, and when to forget the previous data from processing in the next epochs, respectively. This RNN architecture allows the network to keep only valuable information to the subsequent epochs, effectively learning dependencies for a longer time (\emph{Long Term}), and discarding information that does not add to the model (\emph{Short Term}). 

\subsubsection{Gated Recurring Unit Networks - GRU}

Gated Recurring Unit networks are based on the LSTM architecture, with a few notable differences, like the absence of memory cells and the absence of an output gate. To compensate those losses, GRUs work by operating a \emph{reset gate} and an \emph{update gate}. The reset gate works by measuring the previous activation with the next candidate activation, in order to discard (forget) the previous state, while the update gate works by deciding whether it will use the candidate activation to update the cell state. 

While LSTMs restricts the cell state through the control of its gates, GRUs expose the memory content to other units in the architecture. Without the restrictions, and with a simpler model, GRUs are usually faster to train than LSTMs. 

\subsubsection{Hierarchical Attention Networks -- HAN}

Hierarchical Attention Networks, as described by \cite{yang-etal-2016-hierarchical}, are a neural network architecture that highlights the importance of individual words or sentences in the construction of the representation of a document. Since not all terms are equally important to the classification of a text, and sentences do not all represent the same meaning, this model stresses the most important sequences that affect the document's class. 

HANs are usually composed by 6 layers:

\begin{itemize}
	\item An \emph{embedding layer}, responsible for creating a matrix with the characteristics (size of the vocabulary, the maximum length of sentences) of the documents being processed 
	\item A \emph{word sequence encoder}, a word-level bi-directional GRU to get a rich representation of words
	\item A \emph{word attention layer}, a layer to get important information in a sentence
	\item A \emph{sentence encoder}, a sentence level bi-directional GRU to get a rich representation of words
	\item A \emph{sentence attention layer}, a layer to get important information in a sentence
	\item A \emph{final layer}, destined to fully connect all the previous outputs and apply a softmax activation function.
\end{itemize}

\begin{figure
	\centering
 	  \caption[Diagrammatic example of a HAN.]{Diagrammatic example of a HAN. Source: \cite{yang-etal-2016-hierarchical}.}
		\includegraphics{han.jpg}
	\label{fig:han}
\end{figure}

HANs work by queuing a substructure called word encoder, followed by another substructure called sentence encoder. The first applies attention to each of the words inputted in order to form sentence representations. The second applied attention to each of the sentences received before to create document representations. An example of a HAN can be seen in Figure \ref{fig:han}.  

For the \textbf{word encoder}, the following structure was built:

\begin{itemize}
	\item An \textbf{input layer}, to receive the output generated by GloVe, the tokens $w_{i_t}$, representing word $i$ per sentence $t$, in a matrix of None per N, with N representing the maximum words in a sentence in the dataset. It is essential to mention that the term \emph{None} is used by Keras, the Python framework that we have used, to represent any scalar number so that we can use this model to infer on an arbitrarily long input. 
	\item To make the model understand sequences of characters, we have an \textbf{embedding layer}, destined to process strings. This layer assigns multidimensional vectors $W_{ew_{i_j}}$ to each token. Therefore, words are represented numerically as $x_{i_t}$, as a projection of the term in a continuous vector space. There are many embedding methods available. For this research, we have used the GloVe framework. This layer will output a matrix with None per N per the number of dimensions of the word embedding training file, in our case, 600. 
 	\item The third layer contains an \textbf{encoding layer}, in our case, a Bidirectional GRU, to encode the data. The bidirectionality works by reading the sentence from the first to the last word, and reversing the order afterward, in order to understand the connections between words in the left and the right. As an example, in the sentence \emph{The black car is beautiful}, the term $black$ relates directly to the word $car$, as it gives a character to the word, and the word $is$ also represents a strong correlation to $car$, indicating that the following word will describe its nature. The context annotations outputted are represented by $h_{i_t}$.
	\item The following \textbf{dense layer} works by applying the activation function (in our case, ReLU, to counter the vanishing gradient problem) to return the output of the neural network. 
	\item Afterward, the result is processed in the \textbf{word attention layer}, which is an MLP destined to learn the importance of the words through training with randomly initialized weights ($W$), biases ($b$), and the outputs of the encoding layer, as in the Equation \ref{eqhan1}:
	\begin{equation}
	\label{eqhan1}
    u_{i_t} = tanh(W_w h_{i_t} + b_w)
    \end{equation}

    After that step, the result $u_{i_t}$ is then multiplied by a trainable context vector $u_w$, and normalized to an importance weight per word $\alpha_{i_t}$ by a softmax function, described in Equation \ref{eqhan2}. The word context vector $u_w$ is randomly initialized and jointly learned during the training process.
    
    \begin{equation}
    \label{eqhan2}
    \alpha_{i_t} = \frac{\exp(u_{i_t}^T u_w)}{\sum_{t}^{}\exp(u_{i_t}^T u_w)}
    \end{equation}

Finally, those importance weights $\alpha_{i_t}$ are multiplied by the context annotations $h_{i_t}$, being called sentence vectors, and are inputted into the sentence encoder. This operation is described in Equation \ref{eqhan3}. 
    
    \begin{equation}
    \label{eqhan3}
    s_i = \sum_{t}^{}\alpha_{i_t} h_{i_t}
    \end{equation}
\end{itemize}

As for the \textbf{sentence encoder}, the following structure was built: 

\begin{itemize}
	\item The \textbf{input layer} receives the result from the last layer of the word attention, with a matrix of None per M per N, with M being the maximum number of sentences in one document, and N being the maximum words in a sentence in the dataset.
	\item The second layer represents the \textbf{time distributed model}, responsible for wrapping every input it receives as a dense layer, applying to all word-level layers on each sentence. In contrast, a regular dense layer would compute all the inputs as single N units.
	\item As in the word encoder, an \textbf{encoding layer} is used, in our case, a Bidirectional GRU. As previously mentioned, the GRU is used to understand the semantic relations between the sentences. 
	\item After, a \textbf{dense layer} is stacked, with ReLu activation, to retrieve the outputs $h_i$. 
	\item Finally, the result is then inputted into the \textbf{sentence attention layer}. It works in a similar way to the word attention layer, but the final output is a document vector $v$, which can be used as features for document classification. Trainable weights and biases are again randomly initialized and jointly learned during the training process. The operation is described in Equations \ref{eqhan4}, \ref{eqhan5}, and \ref{eqhan6}.  
	
		\begin{equation}
		\label{eqhan4}
    u_{i_t} = tanh(W_s h_i + b_s)
    \end{equation}
    
        \begin{equation}
        \label{eqhan5}
    \alpha_{i_t} = \frac{\exp(u_i^T u_s)}{\sum_{t}^{}\exp(u_i^T u_s)}
    \end{equation}
    
        \begin{equation}
        \label{eqhan6}
    v = \sum_{i}^{}\alpha_i h_i
    \end{equation}
\end{itemize}

After the processing of the HAN networks, every word gets an attention coefficient, indicating the importance of that word in its sentence. An example from \cite{yang-etal-2016-hierarchical} is shown in Figure \ref{fig:exampleattentionhan}. The can see that sentence 1 (\emph{"pork belly = delicious"}) and the final words of sentence 3 (\emph{"these were a-m-a-z-i-n-g"}) are marked in pink. This highlight happens because the HAN has implied that those two sentences are among the most important of that text - that is, they are among the highest sentence attention weights. Inside those sentences, two words are marked in blue. That represents that those two words carry the most important terms in those sentences, also having the highest word attention weights.

\begin{figure}[h!
	\centering
 	  \caption[Example of attention generated by the HAN.]{Example of attention generated by the HAN. Source: \cite{yang-etal-2016-hierarchical}.}
		\includegraphics{exampleattentionhan.jpg}
	\label{fig:exampleattentionhan}
\end{figure}

HANs are currently one of the most popular neural network algorithms being adopted in Computer Science. Only in 2021, the year of publication of this research, we have seen their applications in many different fields, such as Psychology, to detect cyberbullying among adolescents, in \cite{cheng2021modeling}; Medicine, to diagnose cognitive impairment and Alzheimer’s disease, in \cite{zhang2021than}; Urban Management, to manage crowd flow predictions, in \cite{{zhou2021spatiotemporal}}; Social networks, in personal tag recommendations, in \cite{sun2021hierarchical}; and Engineering, with with useful life prediction for multi-sensor signals, in \cite{li2021hierarchical}. We believe this flexibility shows the strength of the attention models to handle different texts across many different vocabularies. 

The adoption of HANs in the legal field is a novel approach but has been positively adopted worldwide, in the works such as \cite{chalkidis2019extreme}, who used HANs to classify the field of legal texts, and the previously mentioned \cite{ma-etal-2019-sentence}, who used the method to predict the outcome of claim verifications. Variations of the HAN model can be seen in \cite{liu2019}, who created a variation of the HAN model to determine the charges in criminal cases or types of disputes in civil cases according to the fact descriptions, and in \cite{WANGWenguan505}, who adopted an improved version of the algorithm for crime prediction, legal article recommendation, and sentence prediction from judicial documents. Our research is the first, to the best of our knowledge, to predict judicial outcomes in Brazilian Portuguese, with results that emulate the ones found in different languages the fields of the Law. 

\subsection{Evaluation measures}

As it is known, researches in the fields of NLP, including automatic classification of texts and data recovery, should be consistently evaluated so as to compare the efficacy of the results. It is common to use standard quality evaluation measures to achieve this comparison, such as accuracy, precision, recall, and F-measure. 

\subsubsection{Accuracy}

The most standard measure of the efficiency of any experiment is \emph{accuracy}. The accuracy shows the division between the number of correct predictions over the total number of documents, as shown below in Equation \ref{eqaccuracy}. In this formula, we sum the number of True Positives ($tp$) with the number of True Negatives ($tn$), and divide that sum by the total sum of all possible results (True Positives, True Negatives, False Positives, False Negatives). 

\begin{equation}
\label{eqaccuracy}
\mbox{ Accuracy} = \frac{\mbox{tp + tn}}{\mbox{tp + tn + fp + fn}}  
\end{equation}

\subsubsection{Precision and recall}

Besides accuracy, we can also evaluate an algorithm by selecting a specific class from all the classes being studied. If one class is more critical to the research being conducted than the others, perhaps accuracy will not be a useful measure, since the algorithm may be having a high accuracy because it is classifying all the texts in the wrong class. 

Alternative measures may prove to be more effective to address this problem. Two of the most used are \emph{precision} and \emph{recall}. In order to better explain those concepts, it is useful to use a contingency table, which shows the semantics of possible combinations of text classification, matching actual classes with predicted classes.

\begin{table}[H]
\centering
\caption{Contingency Table}
\label{Contingency Table}
\begin{tabular}{ll|ll|}
\cline{3-4}
                                               &              & \multicolumn{2}{l|}{\textbf{Actual Class}}                     \\ \cline{3-4} 
                                               &              & \multicolumn{1}{l|}{\textbf{Yes}}        & \textbf{No}         \\ \hline
\multicolumn{1}{|c|}{\textbf{Predicted Class}} & \textbf{Yes} & \multicolumn{1}{l|}{True Positive (tp)}  & False Positive (fp) \\ \hline
\multicolumn{1}{c|}{\textbf{}}                 & \textbf{No}  & \multicolumn{1}{l|}{False Negative (fn)} & True Negative (tn)  \\ \cline{2-4} 
\end{tabular}
\end{table}

From the contingency table, we can infer the equations for precision and recall. According to \cite{Manning:1999:FSN:311445}, precision is defined as the relevant fraction of recovered documents, while recall is the recovered fraction of relevant documents. The formula for precision can be seen in Equation \ref{eqprecision}, and the formula for recall can be seen in Equation \ref{eqrecall}. 

\begin{equation}
\label{eqprecision}
\mbox{ Precision} = \frac{\mbox{tp}}{\mbox{tp+fp}}  
\end{equation}

\begin{equation}
\label{eqrecall}
\mbox{ Recall} = \frac{\mbox{tp}}{\mbox{tp + fn}} \end{equation}

\subsubsection{F-measure}

With all the multitude of different evaluation measures, a new problem arises: the existence of a unified measure of comparison. Depending on the nature of the research, a small loss of precision with an improvement in the recall of the results is acceptable. Even though the balance of those measures strongly depends on the context of the research, it is possible to express this balance by using an equation called \emph{F-measure}. 

F-measure is defined by \cite{Manning:2008:IIR:1394399} as the harmonic mean between precision (P) and recall (C), and can be expressed by Equation \ref{eqfmeasure}. 

\begin{equation}
\label{eqfmeasure}
\mbox{ F-measure} = \frac{1}{\alpha \frac{1}{P} + (1 - \alpha) \frac{1}{R}} = \frac{(\beta ^{2}+1)PC}{\beta ^{2}P + C} = \frac{(tp}{tp + \frac{1}{2} (fp+fn)} 
\end{equation}

Where $\alpha$ in this equation represents the weight between precision and recall, which relates to $\beta$ by Equation \ref{eqbeta}.

\begin{equation}
\label{eqbeta}
\beta ^{2} = \frac{1 - \alpha }{\alpha }
\end{equation}

\subsubsection{K-Fold cross-validation}

In K-Fold Cross-Validation, the original dataset is randomly sliced in $k$ disjoint parts, of approximately equal size. Those partitions are called \emph{folds}. The classifier is then trained from a dataset composed of $k-1$ folds, and the remaining fold is used as a validation set. 

In this method, it is essential to split the dataset $d$ in $K$ parts of equal size $m_k$, in which ${\sum_{k=1}^{K} m_k = n}$. The whole process has to have $k$ iterations, and in each iteration, the testing set will be given by $d_k$, with $k=1,2,...,K$, and the training set for the algorithm will be the sum of the other $K-1$ parts. Therefore, at the end of the $k$ iterations, we will have used all the data available in both training and testing steps. 

\section{Experiments}
\label{exp}

In this chapter, we present the experiments and results found after the datasets were processsed for each of the algorithms chosen, both neural networks and non-neural networks. We also list the word attention weights for each of the datasets. Lastly, we show the top-ranked words in all scenarios.

We highlight that several different combinations of hyperparameters were tried before the optimal match was found. In every case that we do not mention the hyperparameters used, the standard set offered by the framework was the best choice.

\subsection{Non-Neural Networks}

We have used K-Fold Cross-Validation, as mentioned in the previous chapter, with 10 Folds.

After we did our testing,  we could infer that Regression Trees are the method which showed highest accuracy in both data sets chosen, even though the other algorithms showed varying results. SVM, as an example, showed a good performance in the homicides dataset but did not match the results Regression Trees showed in the corruption dataset. Regression Trees have always kept good predicting outcomes. Those results match other results found by other researches in the legal area in many different countries, such as the one conducted by \cite{Kastellec2010}, who obtained good outcomes by using Regression Trees in the American legal system. The author mentions that Regression Trees have the capability of studying intrinsic conceptions of Law, revealing patterns that other methods cannot emulate as effectively. 

Other researchers also used the same method, such as \cite{Rios-Figueroa2011}, who used Regression Trees to analyze the concept of judicial independence and corruption among Supreme Courts in Latin America,  \cite{Antonucci2014}, who adopted Regression Trees to measure the efficiency of Italian courts, and \cite{Kufandirimbwa2012}, who used the same algorithm to predict outcomes in Zimbabwe. 

Those researches show that, even though legal systems are exceedingly different around the world and throughout other languages and countries, such as Brazil, the United States, Italy, and Zimbabwe, they do have similar characteristics that can be effectively measured by the correct algorithms. In that way, we can see that legal texts might have intrinsic factors that remain even when languages change. 

\subsection{Homicides Dataset}

The metrics for the homicides dataset, after the first experiment, are shown in Table \ref{metrics_algorithms_homicides}. We can see that Logistic Regression, Linear Discriminant Analysis and Support Vector Machines showed the highest performance, with high values of precision, recall, f-score and accuracy. Support Vector Machines showed the hiuhest performance in 3 of the 4 metrics. Regression Trees  showed the highest value in recall. 

\begin{table}[h!]
\centering
\caption{Metrics for the algorithms, in the homicides dataset, with mean values after 10-Fold cross-validation.}
\label{metrics_algorithms_homicides}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Algorithm}           & \textbf{Precision} & \textbf{Recall} & \textbf{F-Score} & \textbf{Accuracy} \\ \hline
Logistic Regression          & 0.948895        & 0.934847     & 0.939733      & 0.941117       \\ \hline
Linear Discriminant Analysis & 0.921595        & 0.928063      & 0.922120      & 0.923271       \\ \hline
K-Neighbors                  & 0.779389        & 0.820864     & 0.795847      & 0.795330       \\ \hline
Regression Trees             & 0.888953        & \textbf{0.954632}      & 0.888741      & 0.892924       \\ \hline
Naive Bayes                  & 0.651370        & 0.894028     & 0.769831      & 0.723989       \\ \hline
Support Vector Machines      & \textbf{0.951587}        & 0.933694     & \textbf{0.940827}      & 
\textbf{0.952380}       \\ \hline
\end{tabular}
\end{table}




\subsection{Corruption dataset}
\label{corruptiondataset}

The metrics for the corruption dataset, after the first experiment, are shown in Table \ref{metrics_algorithms_corruption}. We can see that Regression Trees were the best algorithm, scoring the highest value in all of the four metrics. 

\begin{table}[h!]
\centering
\caption{Metrics for the algorithms, in the corruption dataset, with mean values after 10-Fold cross-validation.}
\label{metrics_algorithms_corruption}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Algorithm}           & \textbf{Precision}   & \textbf{Recall}      & \textbf{F-Score}     & \textbf{Accuracy}    \\ \hline
Logistic Regression          & 0.814269                  & 0.827583                  & 0.870421                  & 0.824269          \\ \hline
Linear Discriminant Analysis & 0.859783                 & 0.828997          & 0.807658          & 0.839766          \\ \hline
K-Neighbors                  & 0.851981                  & 0.828655          & 0.897243          & 0.824854          \\ \hline
Regression Trees             & \textbf{0.967917} & \textbf{0.968705} & \textbf{0.973648} & \textbf{0.968421} \\ \hline
Naive Bayes                  & 0.876901                  & 0.899474          & 0.925169          & 0.866374          \\ \hline
Support Vector Machines      & 0.876901                  & 0.872200          & 0.930724         & 0.876901          \\ \hline
\end{tabular}
\end{table}




\subsection{Neural Networks}
\label{neural}

For the experiments involving neural networks, we have used K-Fold Cross Validation, as mentioned in the previous chapter, with 10 Folds. The same methodology was applied to the homicides and the corruption datasets. All tests were run using a GloVe file destined for Brazilian Portuguese, with 600 embedding dimensions.

The convergence criteria were defined by modeling the learning rate and loss functions. For the learning rate, it would be lowered by 0.2 every 3 epochs that the loss function would not lower. For the loss function itself, the algorithm stops if, after 5 epochs, it does not decrease at least 0.001. 

The results for the homicides dataset are shown in Table \ref{metrics_homicides}. The results for the corruption dataset are shown in Table \ref{metrics_corruption}. 

\begin{table}[h!]
\centering
\caption{Metrics for neural networks on the homicides dataset, with mean values after 10-Fold validation}
\label{metrics_homicides}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Dataset} & \textbf{Precision} & \textbf{Recall}   & \textbf{F-Score}  & \textbf{Accuracy} \\ \hline
MLP              & 0.981267           & 0.986228          & 0.983620          & 0.984562          \\ \hline
RNN              & 0.864426           & 0.882808          & 0.866621          & 0.853257          \\ \hline
LSTM             & 0.986512           & 0.985387          & 0.985890          & 0.986355          \\ \hline
GRU              & \textbf{0.992026}  & \textbf{0.993282} & \textbf{0.992625} & \textbf{0.992275} \\ \hline
HAN              & 0.966543           & 0.986666          & 0.976097          & 0.986666          \\ \hline
\end{tabular}
\end{table}

\begin{table}[H]
\centering
\caption{Metrics for neural networks on the corruption dataset, with mean values after 10-Fold validation}
\label{metrics_corruption}
\begin{tabular}{|l|l|l|l|l|}
\hline
\textbf{Dataset} & \textbf{Precision} & \textbf{Recall}   & \textbf{F-Score}  & \textbf{Accuracy} \\ \hline
MLP              & 0.749448           & 0.987332          & 0.855894          & 0.749448          \\ \hline
RNN              & 0.797958           & 0.989252          & 0.882732          & 0.804203          \\ \hline
LSTM             & 0.940756           & 0.986466          & 0.962574          & 0.942762          \\ \hline
GRU              & \textbf{0.996551}  & \textbf{0.998245} & \textbf{0.997391} & 0.996551          \\ \hline
HAN              & 0.985882           & 0.993251          & 0.985540          & \textbf{0.997853} \\ \hline
\end{tabular}
\end{table}

We can see that we got the highest metrics using GRU and HANs. As for the former, the accuracy gains shown for the previous models are kept for the GRUs, and the mistakes are decreased, even with a simplified structure, with only two gates(the reset and the update gate). One important point to mention is that, throughout our experiments, the GRU was faster to train than the LSTM. Therefore, we have higher accuracy values combined with faster training times, which would point to the GRUs as good choices to handle judicial texts.

Our results match the ones found by \cite{chung2014empirical}, who have shown thatthe GRU is faster than the LSTM, but with comparable accuracies. The authors alsomention that the choice of the type of network between LSTMs and GRUs may dependheavily on the dataset and corresponding task. Our research shows that, for our datasets, GRU offers higher metrics.

As for the HANs, our results match similar works that compare those networks structures against other neural network models found in the academia, such as the one by \cite{gao2018hierarchical}, who used HANs to extract information from cancer pathology reports. Using F-Scores as the main comparison metrics, the authors found that micro and macro F-scores for the HAN with pretraining were (0.852, 0.708), compared to Naive Bayes (0.518, 0.213), Logistic Regression (0.682, 0.453), Support Vector Machines (0.634, 0.434), Random Forests (0.698, 0.508), Extreme Gradient Boosting (0.696, 0.522), RNNs (0.505, 0.301), and Convolutional Neural Networks (0.714, 0.460). In another research, \cite{ma-etal-2019-sentence} compared HANs to Convolutional Neural Networks, SVMs, LSTMs and DeClarE to predict the outcome of claim verifications. The authors also found the highest values of precision, recall and F-Scores in the HAN networks. \cite{tarnpradab2018toward} also use HANs, to summarize online forum discussions, outperforming SVM and Logistic Regression models. 

Since the HAN showed good results, comparable to the ones found by the GRU, but has a slower training time, it is indispensable to mention that both algorithms have proven to be choices with high accuracy for our datasets. GRUs have high accuracies but do not implement the Attention model, not giving Attention Weights to every word token. If the prediction of the outcomes is the sole interest of a research, GRUs could be the choice with the highest accuracy. If the analysis of the word tokens is necessary, HANs can be adopted without a significant loss of the overall accuracy. 

\section{Attention Weights Analysis}
\label{attention}

We have computed all the attention weights for every word in the datasets. Since words have different semantic meanings in distinct words, that have appeared in our attention weights dataset multiple times. 

As the absolution and condemnation are different document classifications, words also might have different attention weights for both cases. This difference is the reason why we have mapped each of the datasets twice, for each outcome, in order to map all the possible words that affect the importance of each sentence. The total number of word tokens is shown in Table \ref{number_tokens}. 

\begin{table}[H]
\centering
\caption{Number of word tokens for each dataset}
\label{number_tokens}
\begin{tabular}{|l|l|}
\hline
\textbf{Dataset}         & \textbf{Number of Unique Tokens} \\ \hline
Homicide Absolutions     & 248460                           \\ \hline
Homicide Condemnations   & 466461                           \\ \hline
Corruption Absolutions   & 66929                            \\ \hline
Corruption Condemnations & 252620                           \\ \hline
\end{tabular}
\end{table}

After the classification had been performed, we sought to order all the words in each of the datasets by their attention weights. Therefore, each word will have a unique value, ranging from 0 (where the word would have no importance in the classification of the document) to 1 (where the word would have maximum importance in the classification of the document).

It is useful to mention that a word might have different attention weights in distinct sentences. As a short example, the sentence \emph{The defendant robbed a bank} and the sentence \emph{The defendant did not participate in the robbery, because it was going to a blood bank} both have the word \emph{bank}, but in very different contexts. In the first sentence, the word would be a vital contributor to the condemnation, while it would contribute to the absolution in the second sentence. 

Therefore, words have appeared more than once in our final calculations, with different attention weights. The list with the top 50 words for each of the four outcomes are listed below in Tables \ref{waw_corruption} and \ref{waw_homicide}.

\begin{table}[H]
\centering
\caption{Word attention weights for the corruption dataset}
\label{waw_corruption}
\begin{tabular}{|l|l|l|l|l|l|}
\hline
\multicolumn{3}{|c|}{\textbf{Corruption Absolutions}}         & \multicolumn{3}{c|}{\textbf{Corruption Condemnations}}        \\ \hline
\textbf{Position} & \textbf{Word} & \textbf{Att. Weight} & \textbf{Position} & \textbf{Word} & \textbf{Att. Weight} \\ \hline
1                 & real          & 0.98                      & 1                 & assunto       & 0.99                      \\ \hline
2                 & irregular     & 0.97                      & 2                 & supra         & 0.99                      \\ \hline
3                 & nenhuma       & 0.96                      & 3                 & autos         & 0.98                      \\ \hline
4                 & àquela        & 0.95                      & 4                 & apresentou    & 0.98                      \\ \hline
5                 & ofereceu      & 0.94                      & 5                 & decisão       & 0.98                      \\ \hline
6                 & reconheceu    & 0.94                      & 6                 & cpp           & 0.98                      \\ \hline
7                 & apenas        & 0.94                      & 7                 & decisão       & 0.97                      \\ \hline
8                 & pública       & 0.93                      & 8                 & público       & 0.97                      \\ \hline
9                 & levado        & 0.93                      & 9                 & regime        & 0.97                      \\ \hline
10                & parcialmente  & 0.93                      & 10                & valdefran     & 0.97                      \\ \hline
11                & memoriais     & 0.92                      & 11                & contou        & 0.96                      \\ \hline
12                & resta         & 0.92                      & 12                & demonstrada   & 0.96                      \\ \hline
13                & multa         & 0.92                      & 13                & ativa         & 0.96                      \\ \hline
14                & localidade    & 0.91                      & 14                & exame         & 0.96                      \\ \hline
15                & originário    & 0.91                      & 15                & ministério    & 0.96                      \\ \hline
16                & inicial       & 0.90                      & 16                & começou       & 0.96                      \\ \hline
17                & segue         & 0.89                      & 17                & quantia       & 0.96                      \\ \hline
18                & oferecendo    & 0.89                      & 18                & propina       & 0.96                      \\ \hline
19                & polícia       & 0.88                      & 19                & polícia       & 0.96                      \\ \hline
20                & pretensão     & 0.88                      & 20                & peculato      & 0.96                      \\ \hline
\end{tabular}
\end{table}


\begin{table}[H]
\centering
\caption{Word attention weights for the homicide dataset}
\label{waw_homicide}
\begin{tabular}{|l|l|l|l|l|l|}
\hline
\multicolumn{3}{|c|}{\textbf{Homicide Absolutions}}           & \multicolumn{3}{c|}{\textbf{Homicide Condemnations}}          \\ \hline
\textbf{Position} & \textbf{Word} & \textbf{Att. Weight} & \textbf{Position} & \textbf{Word} & \textbf{Att. Weight} \\ \hline
1                 & bo            & 0.52                      & 1                 & bo            & 0.52                      \\ \hline
2                 & mogi          & 0.42                      & 2                 & cristina      & 0.49                      \\ \hline
3                 & santos        & 0.41                      & 3                 & horário       & 0.49                      \\ \hline
4                 & justiça       & 0.41                      & 4                 & infração      & 0.47                      \\ \hline
5                 & sala          & 0.40                      & 5                 & penal         & 0.46                      \\ \hline
6                 & competência   & 0.40                      & 6                 & cf            & 0.46                      \\ \hline
7                 & volta         & 0.40                      & 7                 & regime        & 0.45                      \\ \hline
8                 & origem        & 0.39                      & 8                 & homicídio     & 0.45                      \\ \hline
9                 & infância      & 0.39                      & 9                 & qualificado   & 0.44                      \\ \hline
10                & social        & 0.38                      & 10                & disparos      & 0.44                      \\ \hline
11                & júri          & 0.38                      & 11                & exposto       & 0.44                      \\ \hline
12                & júri          & 0.38                      & 12                & provisório    & 0.44                      \\ \hline
13                & júri          & 0.37                      & 13                & sentença      & 0.44                      \\ \hline
14                & altura        & 0.36                      & 14                & juízo         & 0.44                      \\ \hline
15                & permitido     & 0.36                      & 15                & golpes        & 0.43                      \\ \hline
16                & júri          & 0.36                      & 16                & justiça       & 0.43                      \\ \hline
17                & soubessem     & 0.36                      & 17                & acusação      & 0.43                      \\ \hline
18                & ordinário     & 0.36                      & 18                & socos         & 0.43                      \\ \hline
19                & central       & 0.36                      & 19                & análise       & 0.42                      \\ \hline
20                & júri          & 0.36                      & 20                & lesões        & 0.42                      \\ \hline
\end{tabular}
\end{table}

We can see that, even though some words are repeated in both scenarios, others have a noteworthy significance towards absolving or condemning a defendant. Those words can be used as a key to predict the outcome of a legal document effectively. 


\section{Conclusion}
\label{conc}

As a first achievement, our work has been able to produce a labeled corpus of judicial cases, with examples of homicide and corruption subjects. When our research began, we found that no labeled corpus was available for the Brazilian judicial system, even though some studies on Law have been conducted in the last years. Since the intersection of AI and Law is a novelty in Brazilian science, we think that our corpus may help future researches to build new prediction strategies. Our corpus also has characteristics that we did not explore in this work. Still, it may be of great help to future analysts, such as the gender of the judge that analyzed the case, and whether the county of the matter is located in a capital or country city. 

As a second achievement, we demonstrated that algorithms could predict the outcome of judicial cases, given the text that was written on their court decisions. Whether using non-neural networks, such as SVM and CART, or neural networks, such as LSTM, GRU, and HAN, we had results that exceeded 95\% accuracy for most cases. Besides having a high accuracy rate for some algorithms, our work has also proven that other methods are not as effective, such as K-Neighbors and RNNs. 

For the non-neural network models, as mentioned previously, we have found that Regression Trees are the method that showed the highest accuracy to predict results in both data sets being analyzed, frequently outperforming the other methods studied. Support Vector Machines, as an example, showed a good performance in the homicides dataset but did not match the results Regression Trees showed in the corruption dataset. In both datasets, Regression Trees have always kept good predicting outcomes. 

Our results match other results found by other researches in the legal area in many different countries, such as the one conducted by \cite{Kastellec2010}, who obtained good outcomes by using Regression Trees in the American legal system. The thesis presented by the author, mentioning that Regression Trees have the capability of studying legal conceptions of Law, revealing patterns that other methods cannot emulate as effectively, can also be seen in our research. \cite{Kastellec2010} also writes that classification trees could also work to increase understanding of legal rules and legal doctrine by capturing many aspects of the relationship between case facts and case outcomes, a statement that could open new research possibilities in Law.  

Other researches have also confirmed the efficacy of Regression Trees, such as \cite{Rios-Figueroa2011}, who used the method to analyze the concept of judicial independence and corruption among Supreme Courts in Latin America, \cite{Antonucci2014}, who adopted Regression Trees to measure the efficiency of Italian courts, and \cite{Kufandirimbwa2012}, who used the same algorithm to predict outcomes in Zimbabwe. 

As previously written, those researches show that, even though legal systems are significantly different among distinct languages and countries, such as Brazil, the United States, Italy and Zimbabwe, they do have similar characteristics that can be effectively measured by the correct algorithms. In that way, we can see that legal texts might have intrinsic features that remain even when languages change. 

As for the neural networks, described in Section \ref{neural}, we have seen that the the GRU and HANs offer the best results. The GRU offers the highest metrics for the homicides dataset, while Hierarchical Attention Networks showed the highest accuracy overall for the corruption dataset and the second-best for the homicides dataset. Our results keep pace with similar works that compare the effectiveness of HAN against other methods, such as the one made by \cite{gao2018hierarchical}, who used HANs compared to Naive Bayes, Logistic Regression, Support Vector Machines, Random Forests, Extreme Gradient Boosting, RNNs, and Convolutional Neural Networks. Other researches have also found that HANs outperform traditional methods. For instance \cite{ma-etal-2019-sentence}, which compared HANs to Convolutional Neural Networks, SVMs, LSTMs, and DeClarE, and \cite{tarnpradab2018toward}, who used HANs against SVM and Logistic Regression models. Since those papers did not include GRUs as one of the possible methods, we cannot also conclude that this method would also be a top pick among the algorithms chosen. Furthermore, HANs embrace the Attention method, which offers an interesting analysis on the word and sentence attention weights of each dataset. 

The accuracies shown by the GRUs are also slightly higher than the results shown by the Regression Trees, indicating that neural networks are in fact effective prediction methods. However, it should also be considered that Regression Trees are computationally faster than the GRUs. Therefore, there is not a single method that could be selected as the best choice for every case presented. 

Moreover, our research shows that the best choices for our datasets, among all the methods compared, are Regression Trees, GRUs, and HANs. However, there is not a single choice between those three methods, since they carry particular advantages and flaws. 

As a first future work path, we intend to research why CART works better than other methods for judicial texts, as shown in Section \ref{corruptiondataset}. Languages have many different characteristics, especially languages with distinct family trees. Finding that CART works better than other methods for Portuguese, Italian, and English is a remarkable discovery, one that deserves more profound studies. 

As a second possibility, we plan to increase our dataset of judicial texts, increasing both the number of cases gathered for homicide and corruption, as well as approaching other judicial subjects. Adding judicial matters that are very different than the ones that were adopted, such as Family Law or Arbitration, can considerably improve the prediction methods that were developed in this research. 

As a final future work direction, we aspire to adopt other algorithms, to expand on the research possibilities. Since NLP and AI are growing rapidly, the rate of novelties to further ameliorate our methods is strikingly high. New algorithms and techniques can be employed to achieve even better results. 


\bibliographystyle{spbasic}     
