\section{Introduction}
\label{Introduction}

Relation extraction aims to identify the relationship between two nominals, typically in the context of a sentence, making it essential to natural language understanding. Consequently, it is a key component of many natural language processing applications, such as information extraction~\cite{fader_openie_2011}, knowledge base population~\cite{tac_kbp_2010}, and question answering~\cite{yu_improvednr_2017}. Table~\ref{tab:re_examples} lists exemplary relation mentions.

\begin{table}[ht!]
    \centering
    \begin{tabular}{p{6cm} l l l}
        \hline
        Sentence & Subject & Object & Relation \\
        \hline
        Mr. \underline{Scheider} played the \underline{police chief} of a resort town menaced by a shark. & Scheider & police chief & per:title \\[20pt]
        The measure included \underline{Aerolineas}'s domestic subsidiary, \underline{Austral}. & Aerolineas & Austral & org:subsidiaries \\[20pt]
        \underline{Yolanda King} died last May of an apparent \underline{heart attack}. & Yolanda King & heart attack & per:cause\_of\_death \\[20pt]
        The \underline{key} was in a \underline{chest}. & key & chest & Content-Container \\[5pt]
        The \underline{car} left the \underline{plant}. & car & plant & Entity-Origin \\[5pt]
        Branches overhang the \underline{roof} of this \underline{house}. & roof & house & Component-Whole\\
        \hline
    \end{tabular}
    \caption{Relation extraction examples, taken from TACRED (1-3) and SemEval 2010 Task 8 (4-6). TACRED relation types mostly focus on named entities, whereas SemEval contains semantic relations between concepts.}
    \label{tab:re_examples}
\end{table}

State-of-the-art relation extraction models, traditional feature-based and current neural methods, typically rely on explicit linguistic features: prefix and morphological features~\cite{mintz_distantsf_2009}, syntactic features such as part-of-speech tags~\cite{zeng_relationcv_2014}, and semantic features like named entity tags and WordNet hypernyms~\cite{xu_improvedrc_2016}. Most recently, \citet{zhang_graphco_2018} combined dependency parse features with graph convolutional neural networks to considerably increase the performance of relation extraction systems.

However, relying on explicit linguistic features severely restricts the applicability and portability of relation extraction to novel languages. Explicitly computing such features requires large amounts of annotated, language-specific resources for training; many unavailable in non-English languages. Moreover, each feature extraction step introduces an additional source of error, possibly cascading over multiple steps.
Deep language representations, on the other hand, have shown to be a very effective form of unsupervised pre-training, yielding contextualized features that capture linguistic properties and benefit downstream natural language understanding tasks, such as semantic role labeling, coreference resolution, and sentiment analysis~\cite{peters_deepcw_2018}. Similarly, fine-tuning pre-trained language representations on a target task has shown to yield state-of-the-art performance on a variety of tasks, such as semantic textual similarity, textual entailment, and question answering~\cite{radford_improvinglu_2018}.

In addition, classifying complex relations requires a considerable amount of annotated training examples, which are time-consuming and costly to acquire. \citet{ruder_universallm_2018} showed language model fine-tuning to be a sample efficient method that requires fewer labeled examples. 

Besides recurrent (RNN) and convolutional neural networks (CNN), the Transformer ~\cite{vaswani_attention_2017} is becoming a popular approach to learn deep language representations. Its self-attentive structure allows it to capture long-range dependencies efficiently; demonstrated by the recent success in machine translation~\cite{vaswani_attention_2017}, text generation~\cite{liu_generatingwb_2018}, and question answering~\cite{radford_improvinglu_2018}.

In this paper, we propose TRE: a \textbf{T}ransformer based \textbf{R}elation \textbf{E}xtraction model. Unlike previous methods, TRE uses deep language representations instead of explicit linguistic features to inform the relation classifier. Since language representations are learned by unsupervised language modeling, pre-training TRE only requires a plain text corpus instead of annotated language-specific resources. Fine-tuning TRE, and its representations, directly to the task minimizes explicit feature extraction, reducing the risk of error accumulation. Furthermore, an increased sample efficiency reduces the need for distant supervision methods~\cite{mintz_distantsf_2009, riedel_modeling_2010}, allowing for simpler model architectures without task-specific modifications.

The contributions of our paper are as follows:
\begin{itemize}
    \item We describe TRE, a Transformer based relation extraction model that, unlike previous methods, relies on deep language representations instead of explicit linguistic features.
    
    \item We are the first to demonstrate the importance of pre-trained language representations in relation extraction, by outperforming state-of-the-art methods on two supervised datasets, TACRED and SemEval 2010 Task 8.
    
    \item We report detailed ablations, demonstrating that pre-trained language representations prevent overfitting and achieve better generalization in the presence of complex entity mentions. Similarly, we demonstrate a considerable increase in sample efficiency over baseline methods.
    
    \item We make our trained models, experiments, and source code available to facilitate wider adoption and further research.
\end{itemize}





\section{TRE}
This section introduces TRE and its implementation. First, we cover the model architecture (Section~\ref{sec:model_architecture}) and input representation (Section~\ref{sec:input_representation}), followed by the introduction of unsupervised pre-training of deep language model representations (Section~\ref{sec:unsuperv_pretraining}). Finally, we present supervised fine-tuning on the relation extraction task (Section~\ref{sec:superv_re}).

\subsection{Model Architecture}
\label{sec:model_architecture}
TRE is a multi-layer Transformer-Decoder~\cite{liu_generatingwb_2018}, a decoder-only variant of the original Transformer~\cite{vaswani_attention_2017}. As shown in Figure~\ref{fig:model_architecture}, the model repeatedly encodes the given input representations over multiple layers (i.e., Transformer blocks), consisting of masked multi-headed self-attention followed by a position-wise feedforward operation:
\begin{equation}
    \begin{tabular}{l}
        $h_0 = T W_e + W_p$ \\
        $h_l = transformer\_block(h_{l-1})\ \forall\ l\ \in\ [1, L]$ \\
    \end{tabular}
\end{equation}
Where $T = (t_{1}, \ldots, t_{k})$ is a sequence of token indices in a sentence\footnote{In this work a ``sentence'' denotes an arbitrary span of contiguous text, rather than an actual linguistic sentence. For pre-training the input consists of multiple linguistic sentences, whereas relation extraction is applied to a single one. A ``sequence'' refers to the input token sequence.}. $W_e$ is the token embedding matrix, $W_p$ is the positional embedding matrix, $L$ is the number of Transformer blocks, and $h_l$ is the state at layer $l$. Since the Transformer has no implicit notion of token positions, the first layer adds a learned positional embedding $e_p \in \mathbb{R}^d$ to each token embedding $e^p_t \in \mathbb{R}^d$ at position $p$ in the input sequence. The self-attentive architecture allows an output state $h^p_l$ of a block to be informed by all input states $h_{l-1}$, which is key to efficiently model long-range dependencies. For language modeling, however, self-attention must be constrained (masked) not to attend to positions ahead of the current token.
For a more exhaustive description of the architecture, we refer readers to~\citet{vaswani_attention_2017} and the excellent guide ``The Annotated Transformer''\footnote{\url{http://nlp.seas.harvard.edu/2018/04/03/attention.html}}.

\begin{figure}[ht!]
\centering
\includegraphics[width=\textwidth]{TRE_architecture.pdf}
\caption{\textbf{(left)} Transformer-Block architecture and training objectives. A Transformer-Block is applied at each of the $L$ layers to produce states $h_1$ to $h_L$. \textbf{(right)}~Relation extraction requires a structured input for fine-tuning, with special delimiters to assign different meanings to parts of the input.}\label{fig:model_architecture}
\end{figure}


\subsection{Input Representation}
\label{sec:input_representation}
Our input representation (see also Figure~\ref{fig:model_architecture}) encodes each sentence as a sequence of tokens. To make use of sub-word information, we tokenize the input text using byte pair encoding (BPE)~\cite{sennrich_subword_2016}. The BPE algorithm creates a vocabulary of sub-word tokens, starting with single characters. Then, the algorithm iteratively merges the most frequently co-occurring tokens into a new token until a predefined vocabulary size is reached. For each token, we obtain its input representation by summing over the corresponding token embedding and positional embedding.

While the model is pre-trained on plain text sentences, the relation extraction task requires a structured input, namely a sentence and relation arguments.
To avoid task-specific changes to the architecture, we adopt a traversal-style approach similar to~\citet{radford_improvinglu_2018}. The structured, task-specific input is converted to an ordered sequence to be directly fed into the model without architectural changes. Figure~\ref{fig:model_architecture} provides a visual illustration of the input format. It starts with the tokens of both relation arguments $a^1$ and $a^2$, separated by delimiters, followed by the token sequence of the sentence containing the relation mention, and ends with a special classification token. The classification token signals the model to generate a sentence representation for relation classification. Since our model processes the input left-to-right, we add the relation arguments to the beginning, to bias the attention mechanism towards their token representation while processing the sentence's token sequence.

\subsection{Unsupervised Pre-training of Language Representations}
\label{sec:unsuperv_pretraining}
Relation extraction models benefit from efficient representations of long-term dependencies \cite{zhang_graphco_2018} and hierarchical relation types \cite{re_relation_att}. Generative pre-training via a language model objective can be seen as an ideal task to learn deep language representations that capture important lexical, syntactic, and semantic features without supervision~\cite{linzen_assessingta_2016, radford_improvinglu_2018, ruder_universallm_2018}, before fine-tuning to the supervised task -- in our case relation extraction.

Given a corpus $\mathcal{C} = \{c_1, \ldots, c_n\}$ of tokens $c_i$, the language modeling objective maximizes the likelihood
\begin{equation}
    \label{eq:lm_objective}
    L_1(\mathcal{C}) = \sum_i \log P(c_i | c_{i-1}, \ldots, c_{i-k}; \theta)
\end{equation}
where k is the context window considered for predicting the next token $c_i$ via the conditional probability $P$. TRE models the conditional probability by an output distribution over target tokens:
\begin{equation}
    P(c) = softmax(h_L W^T_e),
\end{equation}
where $h_L$ is the sequence of states after the final layer $L$, $W_e$ is the embedding matrix, and $\theta$ are the model parameters that are optimized by stochastic gradient descent.

\subsection{Supervised Fine-tuning on Relation Extraction}
\label{sec:superv_re}
After pre-training with the objective in Eq.~\ref{eq:lm_objective}, the language model is fine-tuned on the relation extraction task. We assume a labeled dataset $\mathcal{D} = ([x_i^1, \ldots, x_i^m], a^1_i, a^2_i, r_i)$, where each instance consists of an input sequence of tokens $x^1, \ldots, x^m$, the positions of both relation arguments $a^1$ and $a^2$ in the sequence of tokens, and a corresponding relation label $r$. The input sequence is fed to the pre-trained model to obtain the final state representation $h_L$. To compute the output distribution $P(r)$ over relation labels, a linear layer followed by a softmax is applied to the last state $h^m_L$, which represents a summary of the input sequence:
\begin{equation}
    P(r | x^1, \ldots, x^m) = softmax(h^m_L W_r)
\end{equation}
During fine-tuning we optimize the following objective:
\begin{equation}
    L_2(\mathcal{D}) = \sum^{|\mathcal{D}|}_{i=1} \log P(r_i | x_i^1, \ldots, x_i^m)
\end{equation}
According to \citet{radford_improvinglu_2018}, introducing language modeling as an auxiliary objective during fine-tuning improves generalization and leads to faster convergence. Therefore, we adopt a similar objective:
\begin{equation}
    L(\mathcal{D}) = \lambda * L_1(\mathcal{D}) + L_2(\mathcal{D}),
\end{equation}
where $\lambda$ is the language model weight, a scalar, weighting the contribution of the language model objective during fine-tuning.

\section{Experiment Setup}
\label{experiment_setup}
We run experiments on two supervised relation extraction datasets: The recently published TACRED dataset~\cite{zhang_position_aware_2017} and the SemEval dataset~\cite{hendrickx_semeval2010t8_2010}. We evaluate the PCNN implementation of \citet{zeng_distant_2015}\footnote{\url{https://github.com/thunlp/OpenNRE}} on the two datasets and use it as a state-of-the-art baseline in our analysis section. In the following we describe our experimental setup. 


\subsection{Datasets}
\label{subsec:exp_datasets}

Table \ref{tab:dataset_stats} shows key statistics of the TACRED and SemEval datasets. While TACRED is approximately 10x the size of SemEval 2010 Task 8, it contains a much higher fraction of negative training examples, which makes classification more challenging.
\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c}
        \hline
        Dataset & relation types & examples & negative examples \\
        \hline
        TACRED & 42 & 106,264 & 79.5\% \\
        SemEval 2010 Task 8 & 19 & 10,717 & 17.4\% \\
        \hline
    \end{tabular}
    \caption{Comparison of the datasets used for evaluation}
    \label{tab:dataset_stats}
\end{table}

\noindent \textbf{TACRED}: The \textit{TAC} \textit{R}elation \textit{E}xtraction \textit{D}ataset~\cite{zhang_position_aware_2017} contains 106k sentences with entity mention pairs collected from the TAC KBP evaluations\footnote{\url{https://tac.nist.gov/2017/KBP/index.html}} 2009--2014, with the years 2009 to 2012 used for training, 2013 for evaluation, and 2014 for testing. Sentences are annotated with person- and organization-oriented relation types, e.g.\ \emph{per:title}, \emph{org:founded}, and \emph{no\_relation} for negative examples. In contrast to the SemEval dataset the entity mentions are typed, with subjects classified into person and organization, and objects categorized into 16 fine-grained classes (e.g., date, location, title). As per convention, we report our results as micro-averaged F1 scores. Following the evaluation strategy of \citet{zhang_position_aware_2017}, we select our best model based on the median validation F1 score over 5 independent runs and report its performance on the test set.

\noindent \textbf{SemEval 2010 Task 8}: The SemEval 2010 Task 8 dataset~\cite{hendrickx_semeval2010t8_2010} is a standard benchmark for binary relation classification, and contains 8,000 sentences for training, and 2,717 for testing. Sentences are annotated with a pair of untyped nominals and one of 9 directed semantic relation types, such as\ \emph{Cause-Effect}, \emph{Entity-Origin}, as well as the undirected \emph{Other} type to indicate no relation, resulting in 19 distinct types in total.  We follow the official convention and report macro-averaged F1 scores with directionality taken into account by averaging over 5 independent runs. The dataset is publicly available\footnote{\scriptsize\url{https://drive.google.com/file/d/0B_jQiLugGTAkMDQ5ZjZiMTUtMzQ1Yy00YWNmLWJlZDYtOWY1ZDMwY2U4YjFk}} and we use the official scorer to compute our results.

\subsection{Pre-training}
\label{subsec:exp_pretraining}
Since pre-training is computationally expensive, and our main goal is to show its effectiveness by fine-tuning on the relation extraction task, we reuse the language model\footnote{\url{https://github.com/openai/finetune-transformer-lm}} published by~\citet{radford_improvinglu_2018} for our experiments. The model was trained on the BooksCorpus~\cite{zhu_aligningba_2015}, which contains around 7,000 unpublished books with a total of more than 800M words of different genres. It consists of $L=12$ layers (blocks) with 12 attention heads and 768 dimensional states, and a feed-forward layer of 3072 dimensional states. We reuse the model's byte pair encoding vocabulary, containing 40,000 tokens, but extend it with task-specific ones (i.e., start, end, and delimiter tokens). Also, we use the learned positional embeddings with supported sequence lengths of up to 512 tokens.

\subsection{Entity Masking}
\label{subsec:exp_masking}
We employ four different entity masking strategies. Entity masking allows us to investigate the model performance while providing limited information about entities, in order to prevent overfitting and allow for better generalization to unseen entities. It also enables us to analyze the impact of entity type and role features on the model's performance. For the simplest masking strategy \textit{UNK}, we replace all entity mentions with a special unknown token. For the \textit{NE} strategy, we replace each entity mention with its named entity type. Similarly, \textit{GR} substitutes a mention with its grammatical role (subject or object). \textit{NE + GR} combines both strategies.

\subsection{Hyperparameter Settings and Optimization}
\label{subsec:exp_hyperparams}
During our experiments we found the hyperparameters for fine-tuning, reported in~\cite{radford_improvinglu_2018}, to be very effective. Therefore, unless specified otherwise, we used the Adam optimization scheme~\cite{adam_2015} with $\beta_1=0.9$, $\beta_2=0.999$, a batch size of 8, and a linear learning rate decay schedule with warm-up over 0.2\% of training updates. We apply residual, and classifier dropout with a rate of 0.1. Also, we experimented with token dropout, but did not find that it improved performance. Table~\ref{tab:best_hyperparameters} shows the best performing hyperparameter configuration for each dataset. On SemEval 2010 Task 8, we first split 800 examples of the training set for hyperparameter selection and retrained on the entire training set with the best parameter configuration.

\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c c c} \\\hline
        & Epochs & Learning Rate & Warmup Learning Rate & $\lambda$ & Attn. Dropout  \\\hline
        TACRED & 3 & 5.25e-5 & 2e-3 & 0.5 & 0.1 \\
        SemEval & 3 & 6.25e-5 & 1e-3 & 0.7 & 0.15 \\
        \hline
    \end{tabular}
    \caption{Best hyperparameter configuration for TACRED and SemEval}
    \label{tab:best_hyperparameters}
\end{table}

\section{Results}
\label{sec:results}
This section presents our experimental results. We compare our TRE model to other works on the two benchmark datasets, demonstrating that it achieves state-of-the-art performance even without sophisticated linguistic features. We also provide results on model ablations and the effect of the proposed entity masking schemes.


\subsection{TACRED}
On the TACRED dataset, TRE outperforms state-of-the-art single-model systems and achieves an F1 score of 67.4 (Table~\ref{tab:results_tacred}). Compared to SemEval, we observe methods to perform better that are able to model complex syntactic and long-range dependencies, such as PA-LSTM~\cite{zhang_position_aware_2017} and C-GCN~\cite{zhang_graphco_2018}. Outperforming these methods highlights our model's ability to implicitly capture patterns similar to complex syntactic features, and also capture long-range dependencies.

We would like to point out that the result was produced by the same ``entity masking'' strategy used in previous work~\cite{zhang_position_aware_2017, zhang_graphco_2018}. Similar to our \textit{NE + GR} masking strategy, described in Section~\ref{subsec:exp_masking}, we replace each entity mention by a special token; a combination of its named entity type and grammatical role. While we achieve state-of-the-art results by providing only named entity information, unmasked entity mentions decrease the score to 62.8, indicating overfitting and, consequently, difficulties to generalize to specific entity types. In Section \ref{subsec:effect_entity_masking}, we analyze the effect of entity masking on task performance in more detail.

\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c c} \hline
        System & P & R & F1 \\\hline
        LR$^\mathrm{\dagger}$ \citet{zhang_position_aware_2017} & 72.0 & 47.8 & 57.5 \\
       
        CNN$^\mathrm{\dagger}$ \citet{zhang_position_aware_2017} & \textbf{72.1} & 50.3 & 59.2 \\
        Tree-LSTM$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 66.0 & 59.2 & 62.4 \\
       
        PA-LSTM$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 65.7 & 64.5 & 65.1 \\
        C-GCN$^\mathrm{\dagger}$ \citet{zhang_graphco_2018} & 69.9 & 63.3 & 66.4 \\
        TRE (ours) & 70.1 & \textbf{65.0} & \textbf{67.4} \\\hline
    \end{tabular}
    \caption{TACRED single-model test set performance. We selected the hyperparameters using the validation set, and report the test score of the run with the median validation score among 5 randomly initialized runs. $\mathrm{\dagger}$ marks results reported in the corresponding papers.}
    \label{tab:results_tacred}
\end{table}


\subsection{SemEval}
\label{subsec:res_semeval}
On the SemEval 2010 Task 8 dataset, the TRE model outperforms the best previously reported models, establishing a new state-of-the-art score of 87.1 F1 (Table~\ref{tab:results_semeval2}).
The result indicates that pre-training via a language modeling objective allows the model to implicitly capture useful linguistic properties for relation extraction, outperforming methods that rely on explicit lexical features (SVM~\cite{rink_semrel_2010}, RNN~\cite{zhang_relationcv_2015}). Similarly, our model outperforms approaches that rely on explicit syntactic features such as the shortest dependency path and learned distributed representations of part-of-speech and named entity tags (e.g., BCRNN~\cite{cai_bidirectionalrc_2016}, DRNN~\cite{xu_improvedrc_2016}, CGCN~\cite{zhang_graphco_2018}).
\begin{table}[ht!]
\centering
\begin{tabular}{l l l l}
\hline
System & P & R & F1 \\\hline
SVM{$^\mathrm{\dagger}$}~\citet{rink_semrel_2010} & -- & -- & 82.2\\
PA-LSTM{$^\mathrm{\dagger}$}~\citet{zhang_graphco_2018} & -- & -- & 82.7 \\
C-GCN{$^\mathrm{\dagger}$}~\citet{zhang_graphco_2018} & -- & -- & 84.8 \\
DRNN{$^\mathrm{\dagger}$}~\citet{xu_improvedrc_2016} & -- & -- & 86.1 \\
BRCNN{$^\mathrm{\dagger}$}~\citet{cai_bidirectionalrc_2016} & -- & -- & 86.3 \\
PCNN~\citet{zeng_distant_2015} & 86.7 & 86.7 & 86.6\\
TRE (ours) & 88.0 & 86.2 & $\mathbf{87.1}$ ($\pm 0.16$)\\\hline
\end{tabular}
\caption{SemEval single-model test set performance. $\mathrm{\dagger}$ marks results reported in the corresponding papers. We report the mean and standard deviation across 5 randomly initialized runs.}
    \label{tab:results_semeval2}
\end{table}

Similar to~\citet{zhang_graphco_2018}, we observe a high correlation between entity mentions and relation labels. According to the authors, simplifying SemEval sentences in the training and validation set to just \textit{``subject and object''}, where ``subject'' and ``object'' are the actual entities, already achieves an F1 score of 65.1. To better evaluate our model's ability to generalize beyond entity mentions, we substitute the entity mentions in the training set with a special unknown \textit{(UNK)} token. The token simulates the presence of unseen entities and prevents overfitting to entity mentions that strongly correlate with specific relations.
\begin{table}[ht!]
    \centering
    \begin{tabular}{l c c l}
        \hline
        System & P & R & F1 \\\hline
        PA-LSTM{$^\mathrm{\dagger}$} \citet{zhang_graphco_2018} & -- & -- & 75.3 \\
        C-GCN{$^\mathrm{\dagger}$} \citet{zhang_graphco_2018} & -- & -- & 76.5 \\
        TRE (ours) & 80.3 & 78.0 & \textbf{79.1} ( $\pm$ 0.37) \\\hline
    \end{tabular}
    \caption{SemEval single-model test set performance with all entity mentions masked by an unknown \textit{(UNK)} token. $\mathrm{\dagger}$ marks results reported in the corresponding papers. Due to the small test set size, we report the mean and standard deviation across 5 randomly initialized runs.}
    \label{tab:results_semeval_masked}
\end{table}
Our model achieves an F1 score of 79.1 (Table \ref{tab:results_semeval_masked}), an improvement of 2.6 points F1 score over the previous state-of-the-art. The result suggests that pre-trained language representations improve our model's ability to generalize beyond the mention level when predicting the relation between two previously unseen entities.

\section{Analysis \& Ablation Studies}
\label{analysis_discussion}
Although we demonstrated strong empirical results, we have not yet isolated the contributions of specific parts of TRE. In this section, we perform ablation experiments to understand the relative importance of each model component, followed by experiments to validate our claim that pre-trained language representations capture linguistic properties useful to relation extraction and also improve sample efficiency. We report our results on the predefined TACRED validation set and randomly select 800 examples of the SemEval training set as a validation set.

\subsection{Effect of Pre-training}
Pre-training affects two major parts of our model: language representations and byte pair embeddings. In Table \ref{tab:ablation_masking}, we first compare a model that was fine-tuned using pre-trained representations to one that used randomly initialized language representations. On both datasets we observe fine-tuning to considerably benefit from pre-trained language representations. For the SemEval dataset, the validation F1 score increases to $85.6$ when using a pre-trained language model and no entity masking, compared to $75.6$ without pre-training. We observe even more pronounced performance gains for the TACRED dataset, where using a pre-trained language model increases the validation F1 score by $20$ to $63.3$. With entity masking, performance gains are slightly lower, at $+8$ on the SemEval dataset and $+9.4$ (UNK) respectively $+3.8$ (NE$+$GR) on the TACRED dataset. The larger effect of pre-training when entity mentions are not masked suggests that pre-training has a regularizing effect, preventing overfitting to specific mentions. In addition, the contextualized features allow the model to better adapt to complex entities. Our observations are consistent with the results of~\citet{ruder_universallm_2018}, who observed that language model pre-training considerably improves text classification performance on small and medium-sized datasets, similar to ours.

\begin{table}[ht!]
    \begin{center}
        \begin{tabular}{l@{\hskip .5in} c c c@{\hskip .3in} c c c}
            \hline
             & \multicolumn{2}{c}{\textbf{SemEval}} & & \multicolumn{3}{c}{\textbf{TACRED}} \\
           
            & None & UNK & & None & UNK & NE + GR \\
            \hline
            Best model & \textbf{85.6} & 76.9 & & 63.3 & 51.0 & \textbf{68.0} \\
            -- w/o pre-trained LM & \textbf{75.6} & 68.2 & & 43.3 & 41.6 & \textbf{64.2} \\
            -- w/o pre-trained LM and BPE & 55.3 & \textbf{60.9} & & 38.5 & 38.4 & \textbf{60.8} \\
            \hline
        \end{tabular}
        \caption{Ablation with and without masked entities for SemEval \textbf{(left)} and TACRED validation set \textbf{(right)}. We report F1 scores over 5 independent runs.}
        \label{tab:ablation_masking}
    \end{center}
\end{table}

In addition, we train a model from scratch without pre-trained byte pair embeddings. We keep the vocabulary of sub-word tokens fixed and randomly initialize the embeddings. Again, we observe both datasets to benefit from pre-trained byte-pair embeddings. Because of its small size, SemEval benefits more from pre-trained embeddings, as these can not be learned reliably from the small corpus. This increases the risk of overfitting to entity mentions, which can be seen in the lower performance compared to UNK masking, where entity mentions are not available. For the TACRED dataset, model performance drops by approximately $3-5\%$ with and without entity masking when not using pre-trained byte pair embeddings. 


\subsection{Which Information is captured by Language Representations?}
Undoubtedly, entity type information is crucial to relation extraction. This is confirmed by the superior performance on TACRED (Table~\ref{tab:ablation_masking}) when entity and grammatical role information is provided (NE$+$GR). The model achieves a validation F1 score of $68.0$, compared to $63.3$ without entity masking. Without pre-trained language representations, the model with NE$+$GR masking still manages to achieve a F1 score of $64.2$. This suggests that pre-trained language representations capture features that are as informative as providing entity type and grammatical role information.
This is also suggested by the work of~\citet{peters_deepcw_2018}, who show that a language model captures syntactic and semantic information useful for a variety of natural language processing tasks such as part-of-speech tagging and word sense disambiguation.


\subsection{Effect of Entity Masking}
\label{subsec:effect_entity_masking}
Entity masking, as described in Section~\ref{subsec:exp_masking}, can be used to limit the information about entity mentions available to our model and it is valuable in multiple ways. It can be used to simulate different scenarios, such as the presence of unseen entities, to prevent overfitting to specific entity mentions, and to focus more on context. Table~\ref{tab:robustness} shows F1 scores on the TACRED validation dataset for different entity masking strategies. As we saw previously, masking with entity and grammatical role information yields the best overall performance, yielding a F1 score of $68.0$. We find that using different masking strategies mostly impacts the recall, while precision tends to remain high, with the exception of the UNK masking strategy. 

When applying the UNK masking strategy, which does not provide any information about the entity mention, the F1 score drops to $51.0$. Using grammatical role information considerably increases performance to an F1 score of $56.1$. This suggests that either the semantic role type is a very helpful feature, or its importance lies in the fact that it provides robust information on where each argument entity is positioned in the input sentence. When using NE masking, we observe a significant increase in recall, which intuitively suggests a better generalization ability of the model. Combining NE masking with grammatical role information yields only a minor gain in recall, which increases from $65.3\%$ to $67.2\%$, while precision stays at $68.8\%$.

\begin{table}[ht!]
    \begin{center}
        \begin{tabular}{p{3cm} c c c}
            \hline
            Entity Masking & Precision & Recall & F1 \\
            \hline
            None & \textbf{69.5} & 58.1 & 63.3 \\
            UNK & 56.9 & 46.3 & 51.0 \\
            GR & 63.8 & 50.1 & 56.1 \\
            NE & 68.8 & 65.3 & 67.0 \\
            NE + GR & 68.8 & \textbf{67.2} & \textbf{68.0} \\
            \hline
        \end{tabular}
        \caption{TACRED validation F1 scores for TACRED with different entity masking strategies.}
        \label{tab:robustness}
    \end{center}
\end{table}

\subsection{Sample Efficiency}
We expect a pre-trained language model to allow for a more sample efficient fine-tuning on the relation extraction task.
To assess our model's sample efficiency, we used stratified subsampling splits of the TACRED training set with sampling ratios from 10\% to 100\%. We trained the previously presented model variants on each split, and evaluated them on the complete validation set using micro-averaged F1 scores, averaging the scores over 5 runs.

\begin{figure}[ht!]
\centering
\includegraphics[width=\textwidth]{sample_efficiency.png}
\caption{Micro-averaged F1 score on the validation set over increasing sampling ratios of the training set}\label{fig:sample_efficiency}
\end{figure}

The results are shown in Figure~\ref{fig:sample_efficiency}. The best performing model uses a pre-trained language model combined with NE$+$GR masking, and performs consistently better than the other models. There is a steep performance increase in the first part of the curve, when only a small subset of the training examples is used. The model reaches an F1 score of more than $60$ with only $20\%$ of the training data, and continues to improve with more training data. 

The next best models are the TRE model without a pre-trained language model, and the TRE model without NE$+$GR masking. They perform very similar, which aligns well with our previous observations. The PCNN baseline performs well when masking is applied, but slightly drops in performance compared to the TRE models after $30\%$ of training data, slowly approaching a performance plateau of around $61$ F1 score. The PCNN baseline without masking performs worse, but improves steadily due to its low base score. The TRE model without a language model seems to overfit early and diminishes in performance with more than $70\%$ training data. Interestingly, the performance of several models drops or stagnates after about 80\% of the training data, which might indicate that these examples do not increase the models' regularization capabilities.


\section{Related Work}
\label{related_work}
\noindent  \textbf{Relation Extraction} \quad Providing explicit linguistic features to inform the relation classification is a common approach in relation extraction. Initial work used statistical classifiers or kernel based methods in combination with discrete syntactic features~\cite{zelenko_kernel_2003, mintz_distantsf_2009, hendrickx_semeval2010t8_2010}, such as part-of-speech and named entities tags, morphological features, and WordNet hypernyms. More recently, these methods have been superseded by neural networks, applied on a sequence of input tokens to classify the relation between two entities. Sequence based methods include recurrent~\cite{socher_semanticct_2012, zhang_relationcv_2015} and convolutional~\cite{zeng_relationcv_2014, zeng_distant_2015} neural networks. With neural models, discrete features have been superseded by distributed representations of words and syntactic features~\cite{turian_wordra_2010, pennington_glove_2014}. \citet{xu_semanticrc_2015, xu_classifyingrv_2015} integrated shortest dependency path (SDP) information into a LSTM-based relation classification model. Considering the SDP is useful for relation classification, because it focuses on the action and agents in a sentence~\cite{bunescu_shortest_2005,socher_groundedcs_2014}. \citet{zhang_graphco_2018} established a new state-of-the-art for relation extraction on the TACRED dataset by applying a combination of pruning and graph convolutions to the dependency tree.
Recently, \citet{verga_transformer_2018} extended the Transformer architecture~\cite{vaswani_attention_2017} by a custom architecture  for biomedical named entity and relation extraction. In comparison, our approach uses pre-trained language representations and no architectural modifications between pre-training and task fine-tuning.

\noindent \textbf{Language Representations and Transfer Learning} \quad
Deep language representations have shown to be a very effective form of unsupervised pre-training. \cite{peters_deepcw_2018} introduced embeddings from language models (ELMo), an approach to learn contextualized word representations by training a bidirectional LSTM to optimize a language model objective. \citet{peters_deepcw_2018} results show that replacing static pre-trained word vectors~\cite{mikolov_efficienteo_2013,pennington_glove_2014} with contextualized word representations significantly improves performance on various natural language processing tasks, such as  semantic similarity, coreference resolution, and semantic role labeling. \citet{ruder_universallm_2018} showed language representations learned by unsupervised language modeling to significantly improve text classification performance, to prevent overfitting, and to also increase sample efficiency. \cite{radford_improvinglu_2018} demonstrated that general-domain pre-training and task-specific fine-tuning, which our model is based on, achieves state-of-the-art results on several question answering, text classification, textual entailment, and semantic similarity tasks.


\section{Conclusion}
We proposed TRE, a Transformer based relation extraction method that replaces explicit linguistic features, required by previous methods, with implicit features captured in pre-trained language representations. We showed that our model outperformes the state-of-the-art on two popular relation extraction datasets, TACRED and SemEval 2010 Task 8. We also found that pre-trained language representations drastically improve the sample efficiency of our approach. In our experiments we observed language representations to capture features very informative to the relation extraction task.

While our results are strong, important future work is to further investigate the linguistic features that are captured by TRE. One question of interest is the extent of syntactic structure that is captured in language representations, compared to information provided by dependency parsing. Furthermore, our generic architecture enables us to integrate additional contextual information and background knowledge about entities, which could be used to further improve performance.

\acks{This research was partially supported by the German Federal Ministry of Education and Research through the projects DEEPLEE (01IW17001) and BBDC2 (01IS18025E), and by the German Federal Ministry of Transport and Digital Infrastructure through the project DAYSTREAM (19F2031A).
}

