\section{Introduction}

The legal system is inherently adversarial.
Every case pitches two parties against each other:
The \emph{claimant}, who alleges their rights have been breached, and the \emph{defendant}, who denies breaching those rights.
For each claim of the claimant, their lawyer will produce an argument for which the defendants lawyer will produce a counter argument.
In precedential legal systems, both sides will rely on previous case decisions to support their position \citep{duxbury_2008, sep-legal-reas-prec, garner2009black}.
The claimant will assert that her circumstances are alike those of previous claimants whose rights have been breached.
The defendant, on the other hand, will allege that the circumstances are in fact more alike those of unsuccessful claimants.
The judge decides who is right, and by doing so establishes a new precedent.
If it is the claimant who is successful in a particular claim, the precedent expands the law by including the new facts in its scope.
If it is the defendant who is successful, the law is contracted by rejection of the new facts from its scope.
The expansion or contraction is encoded in the case outcome, we will refer to them as \textbf{positive outcome} and \textbf{negative outcome}, respectively.

\looseness=-1


Positive and negative outcomes are equally binding, which means that the same reasons that motivate the research of the positive outcome apply to the negative outcome. Both are important for computational legal analysis, a fact that has been known at least since \citet{lawlor}.\footnote{He describes them as \emph{pro-precedent} and \emph{con-precedent}.}
However, the \textit{de facto} interpretation of precedent in today's legal NLP landscape, studies only the positive outcome, by predicting the breach of law given the facts of a legal case \cite{aletras}. 
Several researchers have shown that a simple model can achieve a very high performance for the case outcome prediction task \cite{aletras, chalkidis-etal-2019-neural, svm, lexGLUE}, a finding that has been replicated for a number of jurisdictions \cite{zhong-etal-2018-legal, xu2020distinguish}.

\looseness-1


In this work, we reformulate outcome prediction as the task of predicting both the positive and negative outcome given the facts of the case.
Our results demonstrate that while a simple BERT-based classification model can predict positive outcome with a $75.06$ $F_1$, it predicts the negative outcome with only $10.09$ $F_1$, falling short of a random baseline of $11.12$ $F_1$.
This naturally raises the question: What causes such asymmetry?
In \cref{sec:discussion}, we argue that this disparity is caused by a necessity for deeper legal understanding when it comes to negative outcome prediction.
\looseness -1

Searching for a way to better predict negative outcomes, we hypothesise that building a probabilistic model that is more faithful to the legal process will improve both negative and positive outcome prediction.
To test this hypothesis we develop two such models.
Our first model, which we call the \textbf{joint model}, is trained to jointly predict positive and negative outcome.
Our second model, which we call the \textbf{claim--outcome model}, enforces the relationship between the claims and outcomes.
While the joint model significantly outperforms state-of-the-art models on positive outcome prediction with $77.15$ $F_1$,
the claim--outcome model{} doubles the state-of-the-art models on negative outcome prediction at $24.01$ $F_1$.
We take this as strong evidence that building neural models of the legal process should incorporate domain-specific knowledge of how the legal process works.
\looseness=-1


\section{Judicial Process}\label{sec:precedent}

In order to motivate our two models of outcome, it is necessary to first understand the formation of law as a global phenomenon.
Broadly speaking, the legal process can be understood as a task of narrowing down the legal space where the breach of law might have taken place. 
Initially, before the legal process begins, the space includes all the law there is, i.e., every legal Article.
It is the job of the lawyer to narrow it down to only a small number of Articles, a subset of all law.
Finally, the judge determines which of the claimed Articles, if any, has been violated. 
We can therefore observe two distinct interactions between the real world and the law: first, when a lawyer connects them via a claim, second when a judge connects them via an outcome.

In practice, this means that judges are constrained in their decision.
They cannot decide that a law has been breached unless a lawyer has claimed it was breached.
This also means that lawyers actively shape the outcome by forcing a judge to consider a particular subset of law.
In doing so a lawyer defines the set of laws from which the judge decides on the outcome.
The power of a lawyer is also constrained. On one hand, lawyers will want to claim as many legal Articles as possible, on the other there are only so many legal Articles which are relevant to their clients needs.
There are two rules that arise from the interaction of a lawyer and a judge.
First, positive outcome is a subset of claims.
Second, negative outcome consists of the unsuccessful claims, i.e. the claims the judge rejected.\looseness-1

There is a close relationship between claims and negative outcomes: If we knew the claims the lawyer had made, we could define negative outcome as exactly those Articles that have been claimed but were not found to be violated. 
However, much like outcomes are a product of judges, claims are a product of lawyers and, unlike facts, they are not known before human legal experts interact with the case.
Therefore, to study the relationship of outcomes and facts, one can not simply rely on claims as an additional input to the model.
The only input available are the facts.
\looseness=-1

\paragraph{What's wrong with current work.}
Existing models of outcome ignore the role of a legal claim.
Under the current formulation of the task, 
models determine whether a case corresponds to a violation of each Article,  i.e., they predict a vector in $\{0,1\}^K$ where $1$ indicates a positive outcome and $K$ is the number of legal Articles under consideration. 
However, this definition does not help us define when the result is a negative outcome because $0$ is ambiguous. In this setting $0$ can indicate either ``not claimed'' or ``negative outcome.''
Without claims, the existing models implicitly assume that all Articles have been claimed, which is never the case.
It is this assumption that allows the existing models to neglect negative outcomes.

\paragraph{Reformulating the task.} 
How should negative outcome then be modelled?
Given the domain specific knowledge about the interaction of a judge and a lawyer, our position is that models that predict outcomes should model the claims and outcomes \emph{together}.
Towards this, we first need information about which laws have been {\em claimed}.
In \cref{sec:corpora}, we discuss the creation of a new corpus which equips us for this task.
In the next section we develop two models that jointly predict outcomes and claims using two basic assumptions about how the law operates.
We believe that our reformulation of the task has two advantages. One, considering positive and negative outcomes is a step towards better evaluation of legal outcome prediction models. Two, incorporating the roles of a judge and a lawyer within the models of outcome is a step towards better models of law. 






\begin{figure*}[ht]
\subfloat[\textbf{Simple baseline model}\label{fig:mod1}]
  {\includegraphics[width=.23\linewidth]{images/simple.png}}\hfill
\subfloat[\textbf{MTL baseline model} \label{fig:mod3}]
  {\includegraphics[width=.23\linewidth]{images/mtl.png}}\hfill
\subfloat[\textbf{Joint Model}\label{fig:mod2}]
  {\includegraphics[width=.23\linewidth]{images/joint.png}}\hfill
\subfloat[\textbf{Claim--Outcome Model}\label{fig:mod2}]
  {\includegraphics[width=.23\linewidth]{images/precedent.png}}

\caption{The models under consideration. Green and red boxes represent positive and negative outcomes respectively. Blue boxes represent claims. All our models are trained with facts as the input.\looseness-1}
\end{figure*}


\newcommand{C}{C}
\newcommand{c}{c}
\newcommand{\mathbf{\claim}}{\mathbf{c}}

\newcommand{\positive}{+} 
\newcommand{-}{-}
\newcommand{\Plus}{\Plus}
\newcommand{\Minus}{\Minus}

\newcommand{F}{F}
\newcommand{f}{f}
\newcommand{\mathbf{\fact}}{\mathbf{f}}
\newcommand{\mathbf{W}}{\mathbf{W}}
\newcommand{\mathbf{U}}{\mathbf{U}}
\newcommand{\mathbf{V}}{\mathbf{V}}

\newcommand{=}{=}

\newcommand{\mathrm{H}}{\mathrm{H}}

\newcommand{\boldsymbol{f}}{\boldsymbol{f}}
\newcommand{\mathbf{o}}{\mathbf{o}}
\newcommand{\mathbf{c}}{\mathbf{c}}
\newcommand{\emptyset}{\emptyset}
\newcommand{\textsc{y}}{\textsc{y}}
\newcommand{\textsc{n}}{\textsc{n}}
\newcommand{\textbf{O}}{\textbf{O}}
\newcommand{\textbf{C}}{\textbf{C}}
\newcommand{\mathrm{O}}{\mathrm{O}}
\newcommand{\mathrm{C}}{\mathrm{C}}
\newcommand{\mathrm{O}_{k}}{\mathrm{O}_{k}}


\section{Law-Abiding Models}\label{sec:experiments}
In this section we formulate our two probabilistic models of law.
Our law-abiding models are build on top of the two assumptions described below.
\paragraph{Notation.}
We define a probability distribution over three random variables.
\begin{itemize}
    \item $\mathrm{O}$ is a random variable ranging over $\mathcal{O} = \{+, -, \emptyset\}$, which correspond to positive, negative and null outcome, respectively. The null outcome refers to all the unclaimed Articles, i.e., all Articles which are not positive or negative.
    The values of $\mathrm{O}$ are denoted $o \in \mathcal{O}$.
    We write $o_k$ for the outcome of the $k^{\text{th}}$ Article.
    $\textbf{O}$ is a random variable ranging over $\mathcal{O}^K$ where $K$ is the number of Articles we consider.
    The values of $\textbf{O}$ are denoted as $\mathbf{o} \in \mathcal{O}^K$.\looseness-1
   
   
    \item The random variable $\mathrm{C}$ ranging over $\mathcal{C} = \{\textsc{y}, \textsc{n}\}$, which corresponds to an Article being claimed or not.
    The values of $\mathrm{C}$ are denoted $c \in \mathcal{C}$.
    We write $c_k$ for the claim status of the $k^{\text{th}}$ Article.
    The bolded $\textbf{C}$ is the same as above, i.e., a random variable ranging over $\mathcal{C}^K$.
    The values of $\textbf{C}$ are denoted as $\mathbf{c} \in \mathcal{C}^K$. Note that claims are the inverse of the null outcome
   
    \item The random variable $\boldsymbol{F}$ which ranges over textual descriptions of facts, i.e., $\Sigma^*$ for a vocabulary $\Sigma$. Elements of $\Sigma^*$ are denoted as $\boldsymbol{f}$.\looseness-1
\end{itemize}

\subsection{Joint Model}
We begin with a simple assumption that given the facts of a case legal Articles are independent:
\begin{assumption}[Conditional Independence]\label{assumption1}
Conditioned on the facts $\boldsymbol{f}$, the outcome--claim pairs $(o_k, c_k)$ for Article $k$ are conditionally independent of the outcome--claim pairs for other Articles $k' \neq k$.
\looseness=-1
\end{assumption}
\noindent This assumption is based in the origin of each Article as an independent Human Right, related by the spirit of ECHR, but otherwise orthogonal in nature.
This is indeed how the law operates in general. 
A law, whether codified in an Article or a product of precedent, encodes a unique right or obligation.
In practice this means that a breach of one law does not determine breach of another.
For example, a breach of Article 3 of ECHR (the prohibition of torture) does not entail breach of Article 6 (right to a fair trial).
Even breaches of law that are closely related, for example libel and slander, do not entail each other and allegation of each must be considered independently.

By \cref{assumption1}, the joint distribution over outcomes and claims decomposes over Articles as
\begin{align}
    p(\mathbf{O} = &\mathbf{o}, \mathbf{C} = \mathbf{c} \mid \boldsymbol{F} = \boldsymbol{f})  \\
    &= \prod_{k=1}^K p(\mathrm{O}_k = o_k, \mathrm{C}_k = c_k \mid \boldsymbol{F} =  \boldsymbol{f}) \nonumber
\end{align} 
In the remainder of the text we write $\boldsymbol{F} = \boldsymbol{f}$ as just $\boldsymbol{f}$.
We also write $\mathrm{O}_k = o_k$ and $\mathrm{C}_k = c_k$ as $o_k$ and $c_k$, respectively, when clear from context.\looseness=-1

\subsection{Claim--Outcome Model} 
Our second model builds on the first assumption with a second simple assumption:
\begin{assumption}[Claims and Outcomes]\label{assumption2}
Outcomes are only determined for claims, i.e.
\begin{align}
    p(\mathrm{O}_k = \emptyset \mid \mathrm{C}_k = \textsc{y}, \boldsymbol{f})  &= 0 \\
    p(\mathrm{O}_k = \emptyset \mid \mathrm{C}_k =\textsc{n}, \boldsymbol{f}) &= 1
\end{align}
In English, this assumption says that a judge provides an outcome if and only if a claim is made.
\end{assumption}
\noindent By \cref{assumption2}, we have that each distribution over outcome--claim pairs simplifies into the following:\looseness=-1
\begin{align}
    &p(\mathrm{O}_k = o_k, \mathrm{C}_k = c_k \mid \boldsymbol{f}) = \\
    \,\,\,&\begin{cases}
        p(o_k \mid \textsc{y}, \boldsymbol{f})\,   p(\textsc{y} \mid \boldsymbol{f}) & \textbf{if } c_k = \textsc{y} \wedge	 o_k \neq \emptyset \\
        p(\textsc{n} \mid \boldsymbol{f}) & \textbf{if } c_k = \textsc{n} \wedge	 o_k = \emptyset \\
        0 & \textbf{otherwise}
    \end{cases} \nonumber
\end{align}
Crucially, \cref{assumption2} allows us to reduce the problem to two independent binary classification problems.
First, we train a claim predictor $p(c_k \mid \boldsymbol{f})$ that predicts whether a lawyer would claim that the $k^{\text{th}}$ Article is relevant to the facts $\boldsymbol{f}$. 
Second, we train an outcome predictor that predicts whether the outcome is $+$ or $-$ \emph{given} that the lawyer has claimed a violation of Article $k$.
\looseness-1

\subsection{Neural Parameterization}\label{sec:models}
We consider neural parametrizations for all the distributions discussed above.
At heart of all of our models, is a high-dimensional representation ${\footnotesize \textsf{enc}} \in \mathbb{R}^{d_{1}}$ of the facts $\boldsymbol{f}$ which is the output of a pre-trained language model fine-tuned for our task (see \cref{para:plm}). All our language models rely on $\boldsymbol{f}$ as their sole input. Except where we indicate otherwise, both the language model weights and classifier weights are learned separately for every model presented below.\footnote{For the language models, the weights are fine-tuned from a pre-trained model we initialise from the Hugging Face library.\looseness=-1}




\paragraph{Joint Model.}
First we parameterise the joint model which gives us a joint distribution over
all configurations of $o_k$ and $c_k$ for a specific Article $k$.
In principle, there are six such configurations $\{+, -, \emptyset\} \times \{\textsc{y}, \textsc{n}\}$.
However, after we enforce \Cref{assumption2}, we are left with three configurations $\Big\{\langle +, \textsc{y}\rangle$, $\langle -, \textsc{y}\rangle$, $\langle \emptyset, \textsc{n}\rangle \Big\}$, to which we can assign non-zero probability: 
This reduces the problem to a 3-way classification, which we parameterise as follows:\looseness=-1
\begin{align} 
    \label{eq:MTL_pos}
  
    p(\mathrm{O}_k = o_{k}, &\mathrm{C}_k = c_{k} \mid \boldsymbol{f}) = \\
    &{\footnotesize \textsf{softmax}}(\mathbf{U}_k\, {\footnotesize \textsf{ReLU}}(\mathbf{V}_k\,{\footnotesize \textsf{enc}}(\boldsymbol{f})))_{\langle o_k, c_k\rangle} \nonumber
   
   
   
\end{align}
where ${\footnotesize \textsf{softmax}}(\mathbf{x})_i = \frac{\exp x_i}{\sum_{i'}^I \exp x_{i'}}$, ${\footnotesize \textsf{ReLU}}$ is a $\mathrm{ReLU}$ activation function defined as ${\footnotesize \textsf{ReLU}}(x) = \max(0, x)$, $\mathbf{U}_k \in \mathbb{R}^{3 \times d_2}$ and $\mathbf{V}_k \in \mathbb{R}^{d_2 \times d_1}$ are per-article learnable parameters.
Thus, in total, we have $K(d_2 + d_2d_1 + d_3 + d_3d_1)$ parameters for both classifiers excluding those from the encoder ${\footnotesize \textsf{enc}}$.

\paragraph{Claim--Outcome Model.}





We parametrise the claim--outcome model{} as two binary classification tasks.
One predicting the claims, the other predicting positive outcomes. For the latter binary classification task, one class corresponds to $+$, while the other to both $-$ and $\emptyset$.
This leads to the following pair of binary classifiers:
\begin{align} 
    \label{eq:MTL_pos}
  
 
    &p(\mathrm{C}_k = \textsc{y} \mid \boldsymbol{f}) = \sigma(\mathbf{u}_{k} \cdot  \,{\footnotesize \textsf{ReLU}}(\mathbf{V}_{k}\,{\footnotesize \textsf{enc}}(\boldsymbol{f}))) \\ 
    & p(\mathrm{O}_k = + \mid \mathrm{C}_k= \textsc{y},\boldsymbol{f}) = \sigma(\mathbf{u}'_{k} \cdot \,{\footnotesize \textsf{ReLU}}(\mathbf{V}'_{k}\,{\footnotesize \textsf{enc}}'(\boldsymbol{f}))) \nonumber
   
   
   
   
   
\end{align}
where $\mathbf{u}_k \in \mathbb{R}^{d_2}$, $\mathbf{V}_k \in \mathbb{R}^{d_2 \times d_1}$, $\mathbf{u}'_k \in \mathbb{R}^{d_3}$, and $\mathbf{V}'_k \in \mathbb{R}^{d_3 \times d_1}$ are learnable parameters,
$\sigma(x)= \frac{1}{1 + \exp(-x)}$ is the sigmoid function, and ${\footnotesize \textsf{enc}}$ and ${\footnotesize \textsf{enc}}'$ are two \emph{separate} encoders.
In total, we have $K(d_2 + d_1d_2 + d_3 + d_1d_3)$ parameters excluding those from the encoder ${\footnotesize \textsf{enc}}$.
We use primed symbols to denote separately learned parameters. 
Given these probabilities, we can marginalise out the claims to obtain the probability of a positive outcome:\looseness=-1
\begin{align}  
    p(&\mathrm{O}_k = + \mid \boldsymbol{f})  \\
    &=p(\mathrm{O}_k = + \mid \mathrm{C}_k = \textsc{y},\boldsymbol{f})\, p(\mathrm{C}_k = \textsc{y} \mid \boldsymbol{f}) \nonumber\\ 
    &\quad + p(\mathrm{O}_k = +  \mid \mathrm{C}_k = \textsc{n}, \boldsymbol{f})\, p(\mathrm{C}_k = \textsc{n} \mid \boldsymbol{f}) \nonumber \\
    &\stackrel{(1)}{=} p(\mathrm{O}_k = +  \mid \mathrm{C}_k = \textsc{y},\boldsymbol{f})\, p(\mathrm{C}_k = \textsc{y} \mid \boldsymbol{f}) \nonumber
\end{align}
where (1) is true because $p(\mathrm{O}_k = +  \mid \mathrm{C}_k = \textsc{n}, \boldsymbol{f})$ is always zero (since by \cref{assumption2} no positive outcome can be set on an unclaimed case).

We then predict the probability of negative outcome as the inverse of the probability of a positive outcome multiplied by the probability of a claim: 
\begin{align} 
    p(\mathrm{O}_k &= - \mid \boldsymbol{f}) = \\
    &\Big(1-p(\mathrm{O}_k = +  \mid \textsc{y},\boldsymbol{f}) \Big)\, p(\mathrm{C}_k = \textsc{y} \mid \boldsymbol{f}) \nonumber
\end{align}
This step enforces that the negative outcome probability is always both lower than that of claims and sums up to $1$ with positive outcome.
Finally, we have the relation that:
\begin{equation}
    p(\mathrm{O}_k = \emptyset \mid \boldsymbol{f}) = p(\mathrm{C}_k = \textsc{n} \mid \boldsymbol{f})
\end{equation}
To make a decision, we compute the following 
argmax that marginalises over claims:
\begin{align}
    o_k^\star &= \argmax_{o_k \in \mathcal{O}} p(\mathrm{O}_k = o_k \mid \boldsymbol{f}) \\
    &= \argmax_{o_k \in \mathcal{O}} \sum_{c_k \in \mathcal{C}} p(\mathrm{O}_k = o_k, \mathrm{C}_k = c_k \mid \boldsymbol{f}) \nonumber
\end{align}

\paragraph{Training and Fine-tuning.}
All models in \cref{sec:models} are trained by maximizing the log of the joint distribution $p(\mathbf{o}, \mathbf{c} \mid \mathbf{f})$.
We are given a dataset of triples $\mathcal{D} = \Big\{\left(\mathbf{o}^{(n)}, \mathbf{c}^{(n)}, \boldsymbol{f}^{(n)}\right)\Big\}_{n=1}^N$
Due to independence assumption made, this additively factorises over articles
$\sum_{n=1}^N \log p(\mathbf{o}^{(n)}, \mathbf{c}^{(n)} \mid \boldsymbol{f}^{(n)})  = \sum_{n=1}^N \sum_{k=1}^K \log p(o^{(n)}_k, c^{(n)}_k \mid \boldsymbol{f}^{(n)})$.
We fine-tune ${\footnotesize \textsf{enc}}$ jointly for all $p(o_k, c_k \mid \boldsymbol{f})$.

\section{Baselines}

We contextualise the performance of the joint and claim--outcome model{} with a number of baselines.
As a starting point we build a simple classification model trained to predict only positive or negative outcome separately, see \cref{fig:mod1}.
We further want to test whether the advantage of our joint model stems from encoding the relationship between positive and negative outcome, or whether it is down to simply training on more data.
We test this by formulating the task as a multi-task learning objective, see \cref{fig:mod3}. While this model is trained on the same amount of data as our joint model, it does not explicitly encode the relationship between positive and negative outcomes.

\paragraph{A Simple Baseline.}
For our simple baseline model we formulate the positive and negative outcome prediction as a multi-label classification task.
Despite its being conceptually simple, this model achieves state-of-the-art on the task of case outcome prediction \cite{lexGLUE}, which is an analogous task to our positive outcome prediction task.
Given the facts of a case $\boldsymbol{f}$ we directly model the probability that the outcome is positive as a binary classification problem where the first class is the positive $+$ and the second is the union of negative and unclaimed $\{-, \emptyset\}$.
Likewise, we \emph{separately} model the probability that the outcome is negative as a binary classification problem where the first class is the negative $-$ and the second is the union of positive and unclaimed $\{+, \emptyset\}$.
To this end, we define a pair of binary classifiers:\looseness=-1
\begin{align} 
    \label{eq:longformer}
   
    p(\mathrm{O}_k = + \mid \boldsymbol{f}) &= \sigma(\mathbf{u}_{k}^{1} \cdot {\footnotesize \textsf{ReLU}}(\mathbf{V}_{k}^{1}\,{\footnotesize \textsf{enc}}_{1}(\boldsymbol{f}))) \\ 
    p(\mathrm{O}_k = - \mid \boldsymbol{f}) &= \sigma(\mathbf{u}_{k}^{2} \cdot {\footnotesize \textsf{ReLU}}(\mathbf{V}_{k}^{2}\,{\footnotesize \textsf{enc}}_{2}(\boldsymbol{f}))) 
\end{align}
where $\mathbf{u}_k^{1} \in \mathbb{R}^{d_2}$, $\mathbf{V}_k^{1} \in \mathbb{R}^{d_2 \times d_1}$, $\mathbf{u}_k^{2} \in \mathbb{R}^{d_3}$, and $\mathbf{V}_k^{2} \in \mathbb{R}^{d_3 \times d_1}$ are the per-article learnable parameters.
Thus, in total,we have $K(d_3 + d_3 d_1 + d_2 + d_2d_1)$ parameters excluding those from the fine-tuned encoders ${\footnotesize \textsf{enc}}_{1}$ and ${\footnotesize \textsf{enc}}_{2}$. 
The encoders ${\footnotesize \textsf{enc}}_{1}$ and ${\footnotesize \textsf{enc}}_{2}$ represent two \emph{different} fine-tunings of the encoder.
Note that this approach does \emph{not} model whether or not an Article is claimed, which stands in contrast to the main models propsed by this work.
\looseness-1



\paragraph{MTL Baseline.}
We also consider a version of the simple baseline where we jointly 
fine-tune a single encoder.
Symbolically, this is written as:
\begin{align} 
    \label{eq:MTL_pos}
   
    p(\mathrm{O}_k = + \mid \boldsymbol{f}) &= \sigma(\mathbf{u}_{k}^{1} \cdot {\footnotesize \textsf{ReLU}}(\mathbf{V}_{k}^{1}\,{\footnotesize \textsf{enc}}(\boldsymbol{f}))) \\ 
    p(\mathrm{O}_k = - \mid \boldsymbol{f}) &= \sigma(\mathbf{u}_{k}^{2} \cdot {\footnotesize \textsf{ReLU}}(\mathbf{V}_{k}^{2}\,{\footnotesize \textsf{enc}}(\boldsymbol{f})))
 

\end{align}
where ${\footnotesize \textsf{enc}}$ is \emph{shared} between the perceptron heads.
Other than the sharing, the the MTL baseline is the same as the simple baseline.

\paragraph{Random Baseline.}
Finally, we provide a simple random baseline by sampling the outcome vectors from discrete uniform distribution. The random baseline is an average performance over 100 instantiations of this baseline.
\looseness -1

\section{Experimental Setup}
\paragraph{Pre-trained Language Models.}\label{para:plm} 
We obtain the high-dimensional representation ${\footnotesize \textsf{enc}}(\boldsymbol{f})$ by fine-tuning one of the following pre-trained language models with $\boldsymbol{f}$ as an input. 
First, we use BERT because it's a widely used model in legal AI.
Second, we use LEGAL-BERT, because it is purpose trained on legal text which should give it an advantage in our setting.
Finally we use the $\mathrm{Longformer}$ model. $\mathrm{Longformer}$ is built on the same $\mathrm{Transformer}$ \cite{vaswani2017attention} architecture as $\mathrm{BERT}$ \cite{devlin-etal-2019-bert} and $\operatorname{LEGAL-BERT}$ \cite{chalkidis-etal-2020-legal}, but it can process up to $4{,}096$ tokens. 
We select this architecture because the facts of legal documents often exceed $512$ tokens and a model that can process longer documents could therefore be better suited to our needs.
\looseness-1

\paragraph{Training Details.} All our models are trained with a batch size of $16$. 
We conduct hyperparameter optimisation across learning rate $[3\mathrm{e}{-4}, 3\mathrm{e}{-5}, 3\mathrm{e}{-6}]$, dropout $[0.2, 0.3, 0.4]$ and hidden size of $[50, 100, 200, 300]$.
We truncate individual case facts to $512$ tokens for $\mathrm{BERT}$ and $\operatorname{LEGAL-BERT}$ or $4{,}096$ tokens for the $\mathrm{Longformer}$. 
Our models are implemented using the Pytorch \cite{paszke2019pytorch} and Hugginface \cite{Wolf2019HuggingFacesTS} libraries. 
We use Adam for optimization \cite{DBLP:journals/corr/KingmaB14} and train all our models on $1$ Tesla V$100$ $32$GiB GPU's for a maximum of $1$ hour. 
We train for a maximum of $10$ epochs.\footnote{All our code is available at: \emph{redacted for anonymity}}
The models are trained on the training set, see \cref{tab:splits}. 
We report the results on the test set for the models that have achieved the lowest loss on the validation set.
\looseness-1

\begin{figure*}[h]
\subfloat[Negative outcome results on the outcome corpus. \label{fig:res_precedent_neg}]
  {\includegraphics[width=.5\linewidth]{images/micro_neg_precedent.png}}\hfill
\subfloat[Negative outcome results on the \citeauthor{chalkidis-etal-2021-paragraph} corpus. \label{fig:res_alleged_neg}]
  {\includegraphics[width=.5\linewidth]{images/micro_neg_alleged.png}}\hfill
\caption{Results for  \textbf{\textcolor{myblue}{Simple}},  \textbf{\textcolor{myorange}{MTL}}, \textbf{\textcolor{mygreen}{Joint}} and  \textbf{\textcolor{myred}{Claim-Outcome}} models. Dashed \textcolor{red}{line} is our random baseline.}\label{fig:results}
\end{figure*}

\begin{table}[t]
\centering
\begin{tabular}{lcccc}
\toprule
\multicolumn{4}{c}{\textbf{\citeauthor{chalkidis-etal-2021-paragraph} Corpus}} \\
\midrule
\textbf{Outcome} & \textbf{Train} & \textbf{Validation} & \textbf{Test} \\
\midrule
Positive & 8046  & 835  & 851  \\
Negative & 2259  & 279 & 289  \\
Claims & 8836  & 985 & 991  \\
\midrule
\multicolumn{4}{c}{\textbf{Outcome Corpus}} \\
\midrule
\textbf{Outcome} & \textbf{Train} & \textbf{Validation} & \textbf{Test} \\
\midrule
Positive & 7542   & 844  & 925  \\
Negative & 4413   & 498 & 560  \\
Claims & 8372  & 931 & 1034  \\
\bottomrule
\end{tabular}
 \caption{Number of cases with at least one positive or negative outcome label in the dataset.}
  \label{tab:splits}
\end{table}

\section{Legal Corpora}\label{sec:corpora}

We work with the ECtHR corpus, which contains thousands of instances of case law pertaining to the European Convention of Human rights (ECHR). ECtHR cases contain a written description of case facts, which is our $\boldsymbol{f}$, and information about claims and outcomes. Since positive outcomes are a subset of all claims, the exclusion set of claims and positive outcomes constitutes the set of negative outcomes.

\paragraph{\citeauthor{chalkidis-etal-2021-paragraph} Corpus .} 
To obtain the golden labels for outcomes and claims we first rely on the \citet{chalkidis-etal-2021-paragraph} scrape of the ECHR corpus that contains labeled \emph{alleged violations} and \emph{violations}. The \emph{violations} are case outcomes, while the \emph{alleged violations} are claims.

\paragraph{Outcome Corpus .} However, inspecting \citeauthor{chalkidis-etal-2021-paragraph} Corpus , we find that the \emph{alleged violations} do not contain all of the claims. Alleged violations are only the main claims of the case, a subset of all claims that have been raised. To investigate the full set of claims (and negative outcomes) we process the \citet{chalkidis-etal-2019-neural} scrape of the HUDOC\footnote{See \cref{appendix:glossary} for examples from our dataset or \href{https://hudoc.echr.coe.int/}{HUDOC} for all the ECHR caselaw.} database and extract the full set of claims using regular expressions.\footnote{The Outcome Corpus is available at \textit{redacted for anonymity}.}

We conduct all our experiments on both corpora.
However, not all of the Articles of ECHR are interesting from the perspective of a legal outcome since not all Articles of ECHR can be claimed by a lawyer.
First, ECHR comprises of the convention itself and the protocols to the convention. 
Our of the $51$ Articles of the convention, only Articles $2$ to $18$ contain the rights and freedoms, whereas the remaining Articles pertain to the court and its operation. 
We therefore restrict our study to predicting the outcome of the core rights.
Furthermore, we remove any Articles that do not appear in the validation and test sets.
This leaves us with $K = 17$ and $K = 14$ for the \citeauthor{chalkidis-etal-2021-paragraph} Corpus and Outcome Corpus corpus respectively.
\cref{tab:splits} shows the number of cases containing negative outcome vs positive outcome across the training/validation/test splits.
The full distribution of Articles over cases in both corpora can be found in \cref{appendix:corpora}.
\looseness -1

\begin{table*}[]
\centering
\begin{tabular}{llrrrrrrrr}
                                                                                     & \textbf{}    & \multicolumn{4}{c}{\textbf{Outcome Corpus}}                                                                                              & \multicolumn{4}{c}{\textbf{\citeauthor{chalkidis-etal-2021-paragraph} Corpus}}                                                                                               \\
\toprule
\textbf{Model}                                                                       & \textbf{LLM} & \multicolumn{1}{l}{\textbf{Pos}} & \multicolumn{1}{l}{\textbf{Neg}} & \multicolumn{1}{l}{\textbf{Null}} & \multicolumn{1}{l}{\textbf{All}} & \multicolumn{1}{l}{\textbf{Pos}} & \multicolumn{1}{l}{\textbf{Neg}} & \multicolumn{1}{l}{\textbf{Null}} & \multicolumn{1}{l}{\textbf{All}} \\
\midrule
\multirow{3}{*}{\textbf{Claim-Outcome}}                                                  & BERT         & 74.80                            & \textbf{24.01}                   & 95.53                             & 64.78                            & 63.85                            & 14.65                            & 97.15                             & \textbf{58.55}                   \\
                                                                                     & L-BERT        & 74.90                            & 21.83                            & 95.49                             & 64.07                            & 64.47                            & 13.05                            & 97.14                             & 58.22                            \\
                                                                                     & Longformer   & 74.23                            & 20.55                            & 95.17                             & 63.32                            & 63.53                            & \textbf{14.84}                   & 97.21                             & 58.53                            \\
\midrule
\multirow{3}{*}{\textbf{Joint}}                                                      & BERT         & 76.24                            & 17.43                            & 95.46                             & 63.04                            & 65.15                            & 1.87                             & 97.07                             & 54.70                            \\
                                                                                     & L-BERT        & 76.96                            & 21.93                            & \textbf{95.71}                    & \textbf{64.87}                   & \textbf{67.08}                   & 0.94                             & \textbf{97.19}                    & 55.07                            \\
                                                                                     & Longformer   & \textbf{77.15}                   & 16.24                            & 95.49                             & 62.96                            & 65.94                            & 0.95                             & 97.11                             & 54.67                            \\
\midrule
\multirow{3}{*}{\textbf{\begin{tabular}[c]{@{}l@{}}MTL \\ Baseline\end{tabular}}}    & BERT         & 75.75                            & 12.90                            & -                               & -                              & 63.21                            & 0.95                             & -                               & -                              \\
                                                                                     & L-BERT        & 76.73                            & 9.44                             & -                               & -                              & 65.00                            & 0.95                             & -                               & -                              \\
                                                                                     & Longformer   & 75.83                            & 12.34                            & -                               & -                              & 63.36                            & 0.47                             & -                               & -                              \\
\midrule
\multirow{3}{*}{\textbf{\begin{tabular}[c]{@{}l@{}}Simple \\ Baseline\end{tabular}}} & BERT         & 75.06                            & 6.62                             & -                               & -                              & 65.04                            & 0.00                             & -                               & -                              \\
                                                                                     & L-BERT        & 74.85                            & 10.09                            & -                               & -                              & 65.51                            & 0.00                             & -                               & -                              \\
                                                                                     & Longformer   & 74.12                            & 6.72                             & -                               & -                              & 63.92                            & 1.81                             & -                               & -                         \\
\bottomrule
\end{tabular}
\caption{$F_1$-micro averaged scores for all the models considered over the two datasets.}
\label{table:results}
\end{table*}

\section{Results}
Following \citet{chalkidis-etal-2019-neural}, we report all results as $F_1$ micro-averaged.\footnote{We measure significance using the two tailed paired permutation tests with $p < 0.05$.} The bulk of our results is contained in \cref{table:results}. 
First, we compare the positive and negative outcome prediction performance on our outcome corpus and find that while the best simple baseline model achieves $75.06$ $F_1$ on positive outcomes, the same model achieves only $10.09$ $F_1$ on negative outcomes.
In fact, the model fails to beat our random baseline of $11.12$ $F_1$ on negative outcomes.
The same trend holds over all our model architectures, all the underlying language models and both datasets under consideration. Every time, the negative outcome performance is significantly lower than that of positive outcomes.
Therefore, our first conclusion is that negative outcome is simply harder to predict than its positive counterpart.
\looseness-1

Now we turn to the comparison of different model architectures.
First, we consider the task of negative outcome prediction.
Here, we observe a large and significant improvements using our claim--outcome model{}, see \cref{fig:res_precedent_neg} and \cref{fig:res_alleged_neg}. 
Our claim--outcome model{} is significantly better than every baseline model under consideration, a finding that holds over three underlying language models and both datasets.
A single exception to this rule is the joint model, which narrowly beats our claim--outcome model{} (by $0.1$) on the outcome corpus{} using the $\operatorname{LEGAL-BERT}$ LLM.
Overall, where the best claim--outcome model{} achieves $24.01$ $F_1$ on the outcome corpus{} and  $14.84$ $F_1$ on the \citeauthor{chalkidis-etal-2021-paragraph} corpus, the best simple baseline model only achieves $10.09$ and $1.81$ $F_1$, respectively.
Therefore, our second conclusion is that enforcing the relationship between claims and outcomes significantly improves negative outcome prediction. We expand on this in \cref{app:baselines}.
\looseness-1

Turning to the positive outcome prediction task, the $F_1$ is comparable between the simple baseline and our claim--outcome model{}.
The joint model on the other hand improves significantly over either baseline and achieves the best $F_1$ on both the outcome corpus{} and the \citeauthor{chalkidis-etal-2021-paragraph} corpus ($77.15$ and $67.08$ $F_1$ respectively).
Since the simple baseline model using pre-trained BERT is the state-of-the-art model for positive outcome prediction \cite{lexGLUE}, our third conclusion is that jointly training on positive and negative outcomes is a significantly better way of learning how to predict a positive outcome of a case.
\looseness-1

Comparing $\mathrm{BERT}$, $\operatorname{LEGAL-BERT}$ and $\mathrm{Longformer}$ as the underlying language models, we find that neither $\operatorname{LEGAL-BERT}$ nor $\mathrm{Longformer}$ consistently outperform $\mathrm{BERT}$. This is surprising at first sight, since $\operatorname{LEGAL-BERT}$ nor $\mathrm{Longformer}$ should in theory excel on long legal text.
However, these results are consistent with LexGLUE task A \cite{lexGLUE}, which is most similar to our experiments.
Moreover, in our setting it seems that $\operatorname{LEGAL-BERT}$ has significant positive effect on negative outcome prediction for the joint model and is the underlying language model for our best outcome corpus model.
$\mathrm{Longformer}$ sets the best positive outcome performance on the same corpus.
We therefore find both longer document encoding and legal language pre-training useful in certain settings, although it seems that the choice of model architecture has a larger effect on the performance than the choice of the language model.\looseness-1

The effect varies between corpora and classification model, but for negative precedent prediction, $\operatorname{LEGAL-BERT}$ and $\mathrm{Longformer}$ are both slightly worse for the precedent corpus (\cref{fig:res_precedent_neg}), but significantly better for the \citeauthor{chalkidis-etal-2021-paragraph} corpus (\cref{fig:res_alleged_neg}) when compared to $\mathrm{BERT}$.

Finally, we turn to the question of what is the best model of outcome; is the the joint model or claim--outcome model{}?
Towards answering this question we take an average $F_1$ over all three random variables under consideration; the best model of outcome should do well at distinguishing between positive, negative and null outcome.
We find that while the joint model has a slight edge over the claim--outcome model{} on the outcome corpus{} (by $0.1$), the claim--outcome model{} substantially improves over the joint model on the \citeauthor{chalkidis-etal-2021-paragraph} corpus by $3.48$ $F_1$.
This leads us to believe that claim--outcome model{} is overall better model of legal outcome prediction.
However, both models are valuable in their own right.
Where the joint model improves over the state-of-the-art positive outcome prediction models, the claim--outcome model{} doubles their performance on the negative outcome task.

\section{Discussion}\label{sec:discussion}

The results reported above raise the question of why models severely underperform on negative outcome prediction.
The simplest answer could be the amount of training data that is available for each task.
We test this hypothesis by comparing the performance on Articles $8$ ($796$ negative vs. $654$ positive examples) and $13$ ($1197$ negative vs. $1031$ positive examples) of ECHR where in our outcome corpus{} there is more training data available for the negative outcome than positive outcome.
The results, see \cref{fig:subset}, prove that even when the model has more training data for negative outcome than the positive outcome, predicting negative outcome is still harder.
In particular, for Article $13$ the amount of training data is higher than for Article $8$, yet the drop in performance between positive and negative outcome prediction is still dramatic.
In fact, while the scores achieved by the claim--outcome model{} are still low, the other models (except our joint model) fail to predict a single negative outcome correctly for Article $13$.
We therefore believe that the performance drop is likely to be more related with the complexity of the underlying task, rather than the imbalance of the underlying datasets.
\looseness-1

To find a better explanation of the performance asymmetry, we now turn in our discussion to the legal perspective.
In precedential jurisdictions, of which ECtHR is one \cite{zupancic, lupu2010role, valvoda-etal-2021-precedent}, the decisions of a case are binding on future decisions of the court. 
Two cases with the same facts should therefore arrive at the same outcome. 
Of course, in reality, the facts are never the same. 
Rather, cases with similar circumstances will, broadly speaking, lead to similar outcomes. 
This is achieved by \emph{applying} the precedent. 
In such cases, the judge will in effect say that the new case is not substantially different from an already existing case and therefore the same outcome will be propagated. \looseness-1

On the other hand, if the previous precedent is not to be followed, the judge needs to \emph{distinguish} the case at hand from the precedent.
Distinguishing the case from the precedent, is a more involved task than applying the precedent.
It requires identifying what about the new facts sets the new case apart from the previous one. This can of course be done for both cases with positive and negative outcome. 
Both can be applied or distinguished.\footnote{Overruling is another option, though it is exceptionally rare at the ECtHR \cite{dzehtsiarou2017law}.} 
However, since judges deal with claims, each of which comes with an argument built around the precedent that favours the claimant's viewpoint, we believe that negative outcomes do overwhelmingly rely on distinguishing the case from the precedent. 
This is evidenced in the yearly reports of the \citet{ecthr_report}, which list cases where the judges decided to distinguish the facts of the case at hand. 
Distinguishing almost always leads to a negative outcome. 
We observe the same trend in our ECtHR corpus.
\looseness-1 

It might therefore be the case that while there is such a thing as a prototypical positive precedent, there is no prototypical negative precedent.
This could explain why the simpler architectures struggle to learn to predict it.
While a simpler model is ill-suited for the task since it is trained to find a similarity between the negative outcome cases, our claim-outcome model does not assume that negative outcome cases are similar in the first place.
Instead, our model assumes similarity between claims.
Since claim prediction can be modelled with a high accuracy \cite{chalkidis-etal-2021-paragraph}, we can reveal the negative outcome as a disagreement between a judge and a lawyer (i.e. claims and the outcomes).\looseness-1

We can further identify a possible explanation for the baseline model performance by investigating individual cases in \citeauthor{chalkidis-etal-2021-paragraph} Corpus.
Consider the case of \href{https://hudoc.echr.coe.int/fre?i=001-181583}{Wetjen and Others v. Germany} (Wetjen).
This is a case concerned with Article 8: Right to respect for private and family life.
On a superficial level, the case is similar to that of \href{https://hudoc.echr.coe.int/eng?i=001-99817}{Shuruk v. Switzerland} (Shuruk), and \href{https://hudoc.echr.coe.int/eng?i=001-70957}{Suss v. Germany} (Suss), both also Article 8 cases and both cited in Wetjen.
However, while in Wetjen and Suss, the judges have decided a violation of Article 8 has not occurred, they have ruled the opposite in Shuruk.\looseness-1

Let us briefly consider the facts of these cases.
In Shuruk, religious parents fight for an extradition of a child. The mother of the child argues that it would be an interference with her right to a family life if the child was to be extradited to the husband, who has joined an ultra-orthodox Jewish movement.
A component of the case is an allegation of domestic violence the husband was supposed to perpetrate against his wife.
In Wetjen, religious parents used caning (among other methods) as a punishment for their children. The German State has intervened and placed the children in foster care. 
The parents claim interference with their right to family life.
In Suss, German State has denied a divorced father access to his daughter due to the frequent quarrels between the parents during the visits. The father alleges breach of Article 8.\looseness-1

On the surface, the facts are alike, especially between Shuruk and Wetjen -- both cases contain elements of abuse, religion and state intervention.
However, to a human lawyer, the distinction between the cases is fairly trivial -- the facts are more alike between Wetjen and Suss.
In Wetjen, the State is allowed to intervene to protect a child from an abusive ultra-religious parents, which is very similar situation to Suss, where a State is allowed to intervene to protect the child from quarrels between divorced parents.\looseness-1

All our models are exposed to both Shuruk and Suss in the training set.
However, for the positive outcome baseline, the information about Suss being related to Article 8 is lost.
Conversely, for the negative outcome baseline, the information about Shuruk being related to Article 8 is lost.
It is therefore not surprising that the best performing negative\footnote{Simple Baseline $\mathrm{Longformer}$.} and positive\footnote{Simple Baseline $\operatorname{LEGAL-BERT}$.} outcome baseline models both get the Wetjen outcome prediction wrong.
On the other hand, the best claim--outcome model, which is trained to learn that both Shuruk and Suss are related to Article 8 via the claim prediction objective, gets Wetjen outcome prediction right.

In conclusion, while a better way of modelling negative outcome, our claim-outcome model is not actually superior in learning anything about the law itself.
It simply leverages the fact that outcomes and claims are easy to predict and enforces the relationship between them.
To identify the negative outcome with high $F_1$ will require deeper understanding of law than our models are currently capable of.
\looseness-1

\begin{figure}[t]
\centering
\includegraphics[width=7.9cm]{images/subset.png}
\caption{Article 8 and 13 results for \textbf{\textcolor{myblue}{Simple}},  \textbf{\textcolor{myorange}{MTL}}, \textbf{\textcolor{mygreen}{Joint}} and  \textbf{\textcolor{myred}{Precedent}} models.}
\label{fig:subset}
\end{figure}

\section{Related Work}
Juris-informatics can trace its origins all the way to the late 1950s \citep{kort_1957, nagel1963applying}.
The pioneers have used rule based systems to successfully capture aspects of legal reasoning in thousands of hand crafted rules \citep{hypo}. Yet these systems have been too brittle to be employed in practice due to the ever-changing rules of law. Especially in common law countries, the majority of law is contained in case law, where cases are transcripts of the judicial decisions, making the law change constantly with every new decision. With the advances of natural language processing (NLP) in the past two decades, the interest in developing NLP applications for the legal domain have been rejuvenated by the research aiming towards more robust models of law. Areas explored include question answering \citep{monroy09}, legal entity recognition \citep{cardellino-etal-2017-legal}, text summarisation \citep{Hachey}, judgement prediction \citep{xu2020distinguish} and majority opinion prediction \cite{Valvoda2018}.

Our work is similar to the recent study of Chinese law judgement prediction by \citet{zhong-etal-2018-legal} and  \citet{xu2020distinguish}, who break down the court judgement into the applicable law, charges and terms of penalty. Operating in the civil law system (Germany, China, France etc.), they argue that predicting applicable law is one of the fundamental subtasks, which will guide the prediction for other subtasks. In the context of ECHR law, we argue that legal claims are one such guiding element for outcome prediction. While similar, applicable law and claims are different. In the work above, the judge selects the applicable law from the facts as part of reaching the outcome. This is not the case for ECHR law, or any other precedential legal system known to the authors, where the breach of law is claimed by a lawyer, not a judge.

Another reason that legal claims are not part of the research above is that claims are not available in the Chinese Legal Judgment Prediction task or the large
Chinese AI and Law challenge dataset \citep{Xiao2018CAIL2018AL}. Yet that does not make claims unique for the ECHR law. They are as much part of the ECtHR system as any other precedential legal system. Good candidates for future exploration of the interaction of precedent and claims include the Caselaw Access Project and US Supreme Court caselaw.\footnote{See \href{https://www.oyez.org/cases/2018}{US Supreme Court corpus} and \href{https://case.law/}{Caselaw Access Project}.} While ECtHR cases do conveniently provide claim information as part of the judgement, in most legal systems, the information will be submitted directly to the court and to the other party as part of a legal bundle. ECtHR dataset has been collected by \citet{chalkidis-etal-2019-neural}, who have predicted outcomes of the ECHR law and the corresponding Articles using neural architectures. Our work builds on their research by reinstantiating the outcome prediction task on this dataset to include negative precedent.
\looseness-1

\section{Conclusion}
While positive and negative outcomes are equally important from the legal perspective, the current legal AI research has neglected the latter.
Our findings suggest that, at least for current deep learning models, negative outcome is much harder to predict than positive outcome.
This has severe implications on how well can the current legal models learn law from legal text.
The same models that predict positive precedent with $75.06$ $F_1$ fall short of a random baseline of $10.09$ $F_1$ on the negative outcome prediction task.
\looseness -1

We have formulated a theory of why negative outcome prediction is so much harder to learn.
Specifically, we suspect that negative outcomes are mostly caused by a judge distinguishing the case from its precedent. 
This lead us to believe that learning to predict negative outcomes requires more legal understanding than the current models are capable of, which makes it is a particularly attractive task for evaluating progress in legal AI.

While our work significantly improves over the existing models by inducing the relationship between the judge and the lawyer in our claim--outcome model{} architecture, the best negative outcome prediction model achieves only a third of the performance of the positive outcome one.
In future work we hope to study the precedent more closely to build models capable of narrowing this performance gap.
One possible avenue would be to relax our \cref{assumption1} to study the potential relationships between the individual legal Articles.

\section{Ethical Considerations}\label{sec:ethics}

Legal models similar to the ones we study above have been deployed in the real world.
Known applications include risk assessment of detainees in the US \cite{Kehl2017AlgorithmsIT} and sentencing of criminals in China \cite{Xiao2018CAIL2018AL}. 
We believe these are not sensible usecases for legal AI. 
One must not be tempted to think of outcome prediction as equivalent to some medical task, such as cancer detection, with a breach of law seen as a `tumour' that is either there or not. 
This is a naive viewpoint that ignores the fact that the legal system is a human construct in which the decision makers, the judges, play a role that can shift the truth, something that is impossible when it comes to natural laws. 
Put bluntly, if a doctor makes the wrong diagnosis, the patient dies, but if the judge makes the wrong decision, this decision still becomes law. \looseness-1

Herein lies the ethical dubiousness of any attempt at modelling judges using AI. 
Unlike in the domain of medicine, where identifying the underlying truth is essential for treatment, and thus a successful machine diagnostician is in theory a competition for the human one, in the domain of law the validity of the decision is poised
solely on the best intentions of the judge. 
For some judges this pursuit of the 'right' outcome can go as far as defiance of legal precedent; Lord Denning, whom Margaret Thatcher famously called \emph{``probably the greatest English judge of modern times''} \cite{denning} is one such example. 
We therefore argue a judge should not be replaced by a machine and caution against the use of our, or any other legal AI model currently available, towards automating the judicial system.
